Spark依赖关系问题,即使指定了--packages with version,也要考虑Jar的默认版本

时间:2019-04-21 05:21:41

标签: apache-spark dependency-management spark-shell

在EMR中,aws-sdk及其相关jar的默认版本均为1.11.297。即使我指定了** --packages com.amazonaws:aws-java-sdk-s3:1.11.434 **。所有与aws-sdk相关的jar都被引用到1.11.297

我尝试通过下载jar并在args中使用--jars传递jar来手动将jar指定为运行时

spark-shell --packages com.amazonaws:aws-java-sdk-s3:1.11.434
import com.amazonaws.services.s3.model.InputSerialization
new InputSerialization().getClass.getResource("InputSerialization.class")
res0: java.net.URL = jar:file:/usr/share/aws/aws-java-sdk/aws-java-sdk-s3-1.11.297.jar!/com/amazonaws/services/s3/model/InputSerialization.class`

0 个答案:

没有答案