在EMR中,aws-sdk及其相关jar的默认版本均为1.11.297。即使我指定了** --packages com.amazonaws:aws-java-sdk-s3:1.11.434 **。所有与aws-sdk相关的jar都被引用到1.11.297
我尝试通过下载jar并在args中使用--jars传递jar来手动将jar指定为运行时
spark-shell --packages com.amazonaws:aws-java-sdk-s3:1.11.434
import com.amazonaws.services.s3.model.InputSerialization
new InputSerialization().getClass.getResource("InputSerialization.class")
res0: java.net.URL = jar:file:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.297.jar!/com/amazonaws/services/s3/model/InputSerialization.class`