EMR ClassNotFoundException java中的Spark提交

时间：2019-04-18 14:13:59

标签： java apache-spark amazon-emr

我从spring boot应用程序中捆绑创建了jar，在其中我创建了spark会话来执行一些操作。

然后我将此罐子运送到AWS EMR并以spark-submit或java -jar可执行文件的形式运行，而该文件无法正常工作。

这是我jar -tf <jar name>时的罐子结构

META-INF/
META-INF/MANIFEST.MF
org/
org/springframework/
org/springframework/boot/
org/springframework/boot/loader/
org/springframework/boot/loader/data/
org/springframework/boot/loader/data/RandomAccessDataFile$DataInputStream.class
org/springframework/boot/loader/jar/
org/springframework/boot/loader/jar/CentralDirectoryEndRecord.class
org/springframework/boot/loader/data/RandomAccessDataFile$FileAccess.class
org/springframework/boot/loader/jar/CentralDirectoryParser.class
org/springframework/boot/loader/data/RandomAccessDataFile$1.class
org/springframework/boot/loader/jar/ZipInflaterInputStream.class
org/springframework/boot/loader/data/RandomAccessDataFile.class
org/springframework/boot/loader/jar/Handler.class
org/springframework/boot/loader/data/RandomAccessData.class
org/springframework/boot/loader/archive/
org/springframework/boot/loader/archive/Archive$EntryFilter.class
org/springframework/boot/loader/jar/Bytes.class
org/springframework/boot/loader/archive/Archive.class
org/springframework/boot/loader/archive/JarFileArchive.class
org/springframework/boot/loader/PropertiesLauncher$PrefixMatchingArchiveFilter.class
org/springframework/boot/loader/PropertiesLauncher$ArchiveEntryFilter.class
org/springframework/boot/loader/PropertiesLauncher$1.class
org/springframework/boot/loader/PropertiesLauncher.class
org/springframework/boot/loader/Launcher.class
org/springframework/boot/loader/MainMethodRunner.class
org/springframework/boot/loader/ExecutableArchiveLauncher.class
org/springframework/boot/loader/WarLauncher.class
org/springframework/boot/loader/JarLauncher.class
org/springframework/boot/loader/util/
org/springframework/boot/loader/util/SystemPropertyUtils.class
org/springframework/boot/loader/LaunchedURLClassLoader$UseFastConnectionExceptionsEnumeration.class
org/springframework/boot/loader/LaunchedURLClassLoader.class
org/springframework/boot/loader/jar/JarURLConnection$1.class
org/springframework/boot/loader/archive/JarFileArchive$EntryIterator.class
org/springframework/boot/loader/jar/JarURLConnection$JarEntryName.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntryIterator$EntryComparator.class
org/springframework/boot/loader/jar/JarURLConnection.class
org/springframework/boot/loader/jar/JarFile$1.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntryIterator.class
org/springframework/boot/loader/archive/ExplodedArchive$FileEntry.class
org/springframework/boot/loader/archive/ExplodedArchive$1.class
org/springframework/boot/loader/jar/JarFile$2.class
org/springframework/boot/loader/jar/JarFile$JarFileType.class
org/springframework/boot/loader/archive/ExplodedArchive.class
org/springframework/boot/loader/archive/JarFileArchive$JarFileEntry.class
org/springframework/boot/loader/jar/JarFile.class
org/springframework/boot/loader/jar/JarEntry.class
org/springframework/boot/loader/jar/FileHeader.class
org/springframework/boot/loader/jar/StringSequence.class
org/springframework/boot/loader/jar/AsciiBytes.class
org/springframework/boot/loader/jar/JarFileEntries$1.class
org/springframework/boot/loader/jar/JarFileEntries$EntryIterator.class
org/springframework/boot/loader/jar/JarFileEntries.class
org/springframework/boot/loader/jar/CentralDirectoryVisitor.class
org/springframework/boot/loader/jar/JarEntryFilter.class
org/springframework/boot/loader/jar/CentralDirectoryFileHeader.class
org/springframework/boot/loader/archive/Archive$Entry.class
BOOT-INF/
BOOT-INF/classes/
BOOT-INF/classes/com/
BOOT-INF/classes/com/ati/
BOOT-INF/classes/com/ati/media/
BOOT-INF/classes/com/ati/media/ref/
BOOT-INF/classes/com/ati/media/ref/scd/
BOOT-INF/classes/com/ati/media/ref/scd/config/
BOOT-INF/classes/com/ati/media/ref/scd/controllers/
BOOT-INF/classes/com/ati/media/ref/scd/error/
BOOT-INF/classes/com/ati/media/ref/scd/repositories/
BOOT-INF/classes/com/ati/media/ref/scd/services/
BOOT-INF/classes/com/ati/media/ref/scd/services/validator/
META-INF/maven/
META-INF/maven/com.ati.media.ref/
META-INF/maven/com.ati.media.ref/ref-svc-scd/
BOOT-INF/classes/application-local.yml
BOOT-INF/classes/bootstrap.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/SwaggerConfig.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleExceptionHandler.class
BOOT-INF/classes/com/ati/media/ref/scd/ScheduleApplication.class
BOOT-INF/classes/com/ati/media/ref/scd/services/ScheduleServiceImpl.class
META-INF/maven/com.ati.media.ref/ref-svc-scd/pom.xml
BOOT-INF/classes/bootstrap-docker.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/ScheduleConstants.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleCreationException.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/ScheduleMyBatisRepository.xml
BOOT-INF/classes/com/ati/media/ref/scd/services/ConfigurationService.class
BOOT-INF/classes/git.properties
META-INF/maven/com.ati.media.ref/ref-svc-scd/pom.properties
BOOT-INF/classes/application.yml
BOOT-INF/classes/com/ati/media/ref/scd/config/SparkConfig.class
BOOT-INF/classes/com/ati/media/ref/scd/error/ScheduleEditException.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/SparkAthenaRepository.class
BOOT-INF/classes/com/ati/media/ref/scd/services/ScheduleService.class
META-INF/build-info.properties
BOOT-INF/classes/bootstrap-local.yml
BOOT-INF/classes/com/ati/media/ref/scd/controllers/ScheduleController.class
BOOT-INF/classes/com/ati/media/ref/scd/repositories/ScheduleMyBatisRepository.class
BOOT-INF/classes/com/ati/media/ref/scd/services/AthenaQueryBuilder.class
BOOT-INF/classes/com/ati/media/ref/scd/services/validator/ScheduleValidator.class
META-INF/build.properties
BOOT-INF/lib/
BOOT-INF/lib/core-tech-starter-web-base-1.0.0.jar
BOOT-INF/lib/spring-boot-starter-web-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-json-2.0.3.RELEASE.jar
BOOT-INF/lib/jackson-datatype-jdk8-2.9.6.jar
BOOT-INF/lib/jackson-datatype-jsr310-2.9.6.jar
BOOT-INF/lib/jackson-module-parameter-names-2.9.6.jar
BOOT-INF/lib/spring-boot-starter-tomcat-2.0.3.RELEASE.jar
BOOT-INF/lib/tomcat-embed-core-8.5.31.jar
BOOT-INF/lib/tomcat-embed-el-8.5.31.jar
BOOT-INF/lib/tomcat-embed-websocket-8.5.31.jar
BOOT-INF/lib/hibernate-validator-6.0.10.Final.jar
BOOT-INF/lib/spring-web-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-webmvc-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-expression-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-webflux-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-reactor-netty-2.0.3.RELEASE.jar
BOOT-INF/lib/reactor-netty-0.7.8.RELEASE.jar
BOOT-INF/lib/netty-codec-http-4.1.25.Final.jar
BOOT-INF/lib/netty-codec-4.1.25.Final.jar
BOOT-INF/lib/netty-handler-4.1.25.Final.jar
BOOT-INF/lib/netty-buffer-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-4.1.25.Final.jar
BOOT-INF/lib/netty-resolver-4.1.25.Final.jar
BOOT-INF/lib/netty-handler-proxy-4.1.25.Final.jar
BOOT-INF/lib/netty-codec-socks-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-native-epoll-4.1.25.Final-linux-x86_64.jar
BOOT-INF/lib/netty-common-4.1.25.Final.jar
BOOT-INF/lib/netty-transport-native-unix-common-4.1.25.Final.jar
BOOT-INF/lib/spring-webflux-5.0.7.RELEASE.jar
BOOT-INF/lib/reactor-core-3.1.8.RELEASE.jar
BOOT-INF/lib/nio-multipart-parser-1.1.0.jar
BOOT-INF/lib/nio-stream-storage-1.1.3.jar
BOOT-INF/lib/spring-boot-starter-actuator-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-actuator-autoconfigure-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-actuator-2.0.3.RELEASE.jar
BOOT-INF/lib/micrometer-core-1.0.5.jar
BOOT-INF/lib/HdrHistogram-2.1.10.jar
BOOT-INF/lib/LatencyUtils-2.0.3.jar
BOOT-INF/lib/spring-boot-starter-log4j2-2.0.3.RELEASE.jar
BOOT-INF/lib/log4j-slf4j-impl-2.10.0.jar
BOOT-INF/lib/log4j-core-2.10.0.jar
BOOT-INF/lib/log4j-jul-2.10.0.jar
BOOT-INF/lib/log4j-1.2-api-2.7.jar
BOOT-INF/lib/log4j-api-2.10.0.jar
BOOT-INF/lib/lombok-1.18.0.jar
BOOT-INF/lib/micrometer-registry-prometheus-1.0.3.jar
BOOT-INF/lib/simpleclient_common-0.3.0.jar
BOOT-INF/lib/simpleclient-0.3.0.jar
BOOT-INF/lib/spring-cloud-starter-zipkin-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-sleuth-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-sleuth-core-2.0.0.RELEASE.jar
BOOT-INF/lib/aspectjrt-1.8.13.jar
BOOT-INF/lib/brave-5.1.0.jar
BOOT-INF/lib/brave-context-log4j2-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-web-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-http-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-rabbit-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-kafka-clients-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-httpclient-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-httpasyncclient-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-spring-webmvc-5.1.0.jar
BOOT-INF/lib/brave-instrumentation-servlet-5.1.0.jar
BOOT-INF/lib/spring-cloud-sleuth-zipkin-2.0.0.RELEASE.jar
BOOT-INF/lib/zipkin-2.9.3.jar
BOOT-INF/lib/zipkin-reporter-2.7.3.jar
BOOT-INF/lib/zipkin-sender-kafka11-2.7.3.jar
BOOT-INF/lib/zipkin-sender-amqp-client-2.7.3.jar
BOOT-INF/lib/core-tech-starter-amqp-1.0.0.jar
BOOT-INF/lib/spring-cloud-stream-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-validation-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-messaging-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-integration-core-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-integration-jmx-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-tuple-1.0.0.RELEASE.jar
BOOT-INF/lib/spring-integration-tuple-1.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-amqp-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-rabbit-2.0.4.RELEASE.jar
BOOT-INF/lib/spring-amqp-2.0.4.RELEASE.jar
BOOT-INF/lib/amqp-client-5.1.2.jar
BOOT-INF/lib/spring-cloud-stream-binder-rabbit-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-stream-binder-rabbit-core-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-integration-amqp-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-config-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-context-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-security-rsa-1.0.5.RELEASE.jar
BOOT-INF/lib/bcpkix-jdk15on-1.56.jar
BOOT-INF/lib/spring-cloud-config-client-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-aop-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-aop-5.0.7.RELEASE.jar
BOOT-INF/lib/aspectjweaver-1.8.13.jar
BOOT-INF/lib/core-tech-lib-core-1.0.0.jar
BOOT-INF/lib/spring-retry-1.2.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-openfeign-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-openfeign-core-2.0.0.RELEASE.jar
BOOT-INF/lib/feign-core-9.5.1.jar
BOOT-INF/lib/feign-slf4j-9.5.1.jar
BOOT-INF/lib/feign-hystrix-9.5.1.jar
BOOT-INF/lib/feign-java8-9.5.1.jar
BOOT-INF/lib/spring-cloud-starter-netflix-ribbon-2.0.0.RELEASE.jar
BOOT-INF/lib/ribbon-2.2.5.jar
BOOT-INF/lib/ribbon-transport-2.2.5.jar
BOOT-INF/lib/rxnetty-contexts-0.4.9.jar
BOOT-INF/lib/rxnetty-servo-0.4.9.jar
BOOT-INF/lib/rxnetty-0.4.9.jar
BOOT-INF/lib/netty-transport-native-epoll-4.1.25.Final.jar
BOOT-INF/lib/ribbon-core-2.2.5.jar
BOOT-INF/lib/ribbon-httpclient-2.2.5.jar
BOOT-INF/lib/jersey-client-1.19.1.jar
BOOT-INF/lib/jersey-apache-client4-1.19.1.jar
BOOT-INF/lib/servo-core-0.12.21.jar
BOOT-INF/lib/netflix-commons-util-0.3.0.jar
BOOT-INF/lib/ribbon-loadbalancer-2.2.5.jar
BOOT-INF/lib/netflix-statistics-0.1.1.jar
BOOT-INF/lib/commons-text-1.3.jar
BOOT-INF/lib/commons-collections4-4.1.jar
BOOT-INF/lib/core-tech-starter-web-api-1.0.0.jar
BOOT-INF/lib/springfox-swagger2-2.9.2.jar
BOOT-INF/lib/swagger-annotations-1.5.20.jar
BOOT-INF/lib/swagger-models-1.5.20.jar
BOOT-INF/lib/springfox-spi-2.9.2.jar
BOOT-INF/lib/springfox-core-2.9.2.jar
BOOT-INF/lib/springfox-schema-2.9.2.jar
BOOT-INF/lib/springfox-swagger-common-2.9.2.jar
BOOT-INF/lib/springfox-spring-web-2.9.2.jar
BOOT-INF/lib/guava-20.0.jar
BOOT-INF/lib/classmate-1.3.4.jar
BOOT-INF/lib/spring-plugin-core-1.2.0.RELEASE.jar
BOOT-INF/lib/spring-plugin-metadata-1.2.0.RELEASE.jar
BOOT-INF/lib/mapstruct-1.2.0.Final.jar
BOOT-INF/lib/springfox-swagger-ui-2.9.2.jar
BOOT-INF/lib/spring-cloud-starter-netflix-hystrix-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-core-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-ribbon-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-netflix-archaius-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-netflix-archaius-2.0.0.RELEASE.jar
BOOT-INF/lib/archaius-core-0.7.6.jar
BOOT-INF/lib/hystrix-core-1.5.12.jar
BOOT-INF/lib/rxjava-1.3.8.jar
BOOT-INF/lib/hystrix-serialization-1.5.12.jar
BOOT-INF/lib/jackson-module-afterburner-2.9.6.jar
BOOT-INF/lib/hystrix-metrics-event-stream-1.5.12.jar
BOOT-INF/lib/hystrix-javanica-1.5.12.jar
BOOT-INF/lib/asm-5.0.4.jar
BOOT-INF/lib/rxjava-reactive-streams-1.2.1.jar
BOOT-INF/lib/reactive-streams-1.0.2.jar
BOOT-INF/lib/spring-cloud-netflix-hystrix-stream-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-cloud-commons-2.0.0.RELEASE.jar
BOOT-INF/lib/spring-security-crypto-5.0.6.RELEASE.jar
BOOT-INF/lib/spring-cloud-starter-stream-rabbit-2.0.0.RELEASE.jar
BOOT-INF/lib/postgresql-42.2.2.jar
BOOT-INF/lib/athenaJDBC-1.1.0.jar
BOOT-INF/lib/lib-client-scd-0.0.0.jar
BOOT-INF/lib/core-tech-starter-web-client-1.0.0.jar
BOOT-INF/lib/hibernate-jpa-2.1-api-1.0.2.Final.jar
BOOT-INF/lib/gson-2.8.5.jar
BOOT-INF/lib/spring-boot-starter-data-jpa-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-starter-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-boot-autoconfigure-2.0.3.RELEASE.jar
BOOT-INF/lib/javax.annotation-api-1.3.2.jar
BOOT-INF/lib/snakeyaml-1.19.jar
BOOT-INF/lib/spring-boot-starter-jdbc-2.0.3.RELEASE.jar
BOOT-INF/lib/HikariCP-2.7.9.jar
BOOT-INF/lib/spring-jdbc-5.0.7.RELEASE.jar
BOOT-INF/lib/hibernate-core-5.2.17.Final.jar
BOOT-INF/lib/jboss-logging-3.3.2.Final.jar
BOOT-INF/lib/javassist-3.22.0-GA.jar
BOOT-INF/lib/antlr-2.7.7.jar
BOOT-INF/lib/jandex-2.0.3.Final.jar
BOOT-INF/lib/dom4j-1.6.1.jar
BOOT-INF/lib/hibernate-commons-annotations-5.0.1.Final.jar
BOOT-INF/lib/javax.transaction-api-1.2.jar
BOOT-INF/lib/spring-data-jpa-2.0.8.RELEASE.jar
BOOT-INF/lib/spring-data-commons-2.0.8.RELEASE.jar
BOOT-INF/lib/spring-orm-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-context-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-beans-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-aspects-5.0.7.RELEASE.jar
BOOT-INF/lib/mybatis-spring-boot-starter-1.3.1.jar
BOOT-INF/lib/mybatis-spring-boot-autoconfigure-1.3.1.jar
BOOT-INF/lib/mybatis-3.4.5.jar
BOOT-INF/lib/mybatis-spring-1.3.1.jar
BOOT-INF/lib/spring-boot-starter-jooq-2.0.3.RELEASE.jar
BOOT-INF/lib/spring-tx-5.0.7.RELEASE.jar
BOOT-INF/lib/jooq-3.10.7.jar
BOOT-INF/lib/byte-buddy-1.7.11.jar
BOOT-INF/lib/objenesis-2.6.jar
BOOT-INF/lib/spring-core-5.0.7.RELEASE.jar
BOOT-INF/lib/spring-jcl-5.0.7.RELEASE.jar
BOOT-INF/lib/core-tech-starter-web-request-details-1.0.0.jar
BOOT-INF/lib/spark-core_2.11-2.3.0.jar
BOOT-INF/lib/avro-1.7.7.jar
BOOT-INF/lib/jackson-core-asl-1.9.13.jar
BOOT-INF/lib/paranamer-2.3.jar
BOOT-INF/lib/commons-compress-1.4.1.jar
BOOT-INF/lib/xz-1.0.jar
BOOT-INF/lib/avro-mapred-1.7.7-hadoop2.jar
BOOT-INF/lib/avro-ipc-1.7.7.jar
BOOT-INF/lib/avro-ipc-1.7.7-tests.jar
BOOT-INF/lib/chill_2.11-0.8.4.jar
BOOT-INF/lib/kryo-shaded-3.0.3.jar
BOOT-INF/lib/minlog-1.3.0.jar
BOOT-INF/lib/chill-java-0.8.4.jar
BOOT-INF/lib/xbean-asm5-shaded-4.4.jar
BOOT-INF/lib/hadoop-client-2.6.5.jar
BOOT-INF/lib/hadoop-hdfs-2.6.5.jar
BOOT-INF/lib/xercesImpl-2.9.1.jar
BOOT-INF/lib/xml-apis-1.4.01.jar
BOOT-INF/lib/htrace-core-3.0.4.jar
BOOT-INF/lib/hadoop-mapreduce-client-app-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-common-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-shuffle-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-core-2.6.5.jar
BOOT-INF/lib/hadoop-mapreduce-client-jobclient-2.6.5.jar
BOOT-INF/lib/hadoop-annotations-2.6.5.jar
BOOT-INF/lib/spark-launcher_2.11-2.3.0.jar
BOOT-INF/lib/spark-kvstore_2.11-2.3.0.jar
BOOT-INF/lib/leveldbjni-all-1.8.jar
BOOT-INF/lib/jackson-core-2.9.6.jar
BOOT-INF/lib/spark-network-common_2.11-2.3.0.jar
BOOT-INF/lib/spark-network-shuffle_2.11-2.3.0.jar
BOOT-INF/lib/spark-unsafe_2.11-2.3.0.jar
BOOT-INF/lib/jets3t-0.9.4.jar
BOOT-INF/lib/httpcore-4.4.9.jar
BOOT-INF/lib/activation-1.1.1.jar
BOOT-INF/lib/bcprov-jdk15on-1.52.jar
BOOT-INF/lib/java-xmlbuilder-1.1.jar
BOOT-INF/lib/base64-2.3.8.jar
BOOT-INF/lib/curator-recipes-4.0.1.jar
BOOT-INF/lib/curator-framework-4.0.1.jar
BOOT-INF/lib/javax.servlet-api-3.1.0.jar
BOOT-INF/lib/commons-lang3-3.7.jar
BOOT-INF/lib/commons-math3-3.4.1.jar
BOOT-INF/lib/jsr305-1.3.9.jar
BOOT-INF/lib/slf4j-api-1.7.25.jar
BOOT-INF/lib/jul-to-slf4j-1.7.25.jar
BOOT-INF/lib/jcl-over-slf4j-1.7.25.jar
BOOT-INF/lib/compress-lzf-1.0.3.jar
BOOT-INF/lib/snappy-java-1.1.2.6.jar
BOOT-INF/lib/lz4-java-1.4.0.jar
BOOT-INF/lib/zstd-jni-1.3.2-2.jar
BOOT-INF/lib/RoaringBitmap-0.5.11.jar
BOOT-INF/lib/commons-net-2.2.jar
BOOT-INF/lib/scala-library-2.11.8.jar
BOOT-INF/lib/json4s-jackson_2.11-3.2.11.jar
BOOT-INF/lib/json4s-core_2.11-3.2.11.jar
BOOT-INF/lib/json4s-ast_2.11-3.2.11.jar
BOOT-INF/lib/scalap-2.11.0.jar
BOOT-INF/lib/scala-compiler-2.11.0.jar
BOOT-INF/lib/scala-xml_2.11-1.0.1.jar
BOOT-INF/lib/jersey-client-2.26.jar
BOOT-INF/lib/javax.ws.rs-api-2.1.jar
BOOT-INF/lib/javax.inject-2.5.0-b42.jar
BOOT-INF/lib/jersey-common-2.26.jar
BOOT-INF/lib/osgi-resource-locator-1.0.1.jar
BOOT-INF/lib/jersey-server-2.26.jar
BOOT-INF/lib/jersey-media-jaxb-2.26.jar
BOOT-INF/lib/validation-api-2.0.1.Final.jar
BOOT-INF/lib/jersey-container-servlet-2.26.jar
BOOT-INF/lib/jersey-container-servlet-core-2.26.jar
BOOT-INF/lib/netty-all-4.1.25.Final.jar
BOOT-INF/lib/netty-3.9.9.Final.jar
BOOT-INF/lib/stream-2.7.0.jar
BOOT-INF/lib/metrics-core-3.2.6.jar
BOOT-INF/lib/metrics-jvm-3.2.6.jar
BOOT-INF/lib/metrics-json-3.2.6.jar
BOOT-INF/lib/metrics-graphite-3.2.6.jar
BOOT-INF/lib/jackson-databind-2.9.6.jar
BOOT-INF/lib/jackson-module-scala_2.11-2.9.6.jar
BOOT-INF/lib/scala-reflect-2.11.11.jar
BOOT-INF/lib/jackson-module-paranamer-2.9.6.jar
BOOT-INF/lib/ivy-2.4.0.jar
BOOT-INF/lib/oro-2.0.8.jar
BOOT-INF/lib/pyrolite-4.13.jar
BOOT-INF/lib/py4j-0.10.6.jar
BOOT-INF/lib/spark-tags_2.11-2.3.0.jar
BOOT-INF/lib/commons-crypto-1.0.0.jar
BOOT-INF/lib/unused-1.0.0.jar
BOOT-INF/lib/simba-athena-jdbc-driver-2.0.2.jar
BOOT-INF/lib/spark-sql_2.11-2.3.0.jar
BOOT-INF/lib/univocity-parsers-2.5.9.jar
BOOT-INF/lib/spark-sketch_2.11-2.3.0.jar
BOOT-INF/lib/spark-catalyst_2.11-2.3.0.jar
BOOT-INF/lib/scala-parser-combinators_2.11-1.0.4.jar
BOOT-INF/lib/janino-3.0.8.jar
BOOT-INF/lib/commons-compiler-3.0.8.jar
BOOT-INF/lib/antlr4-runtime-4.7.jar
BOOT-INF/lib/orc-core-1.4.1-nohive.jar
BOOT-INF/lib/protobuf-java-2.5.0.jar
BOOT-INF/lib/commons-lang-2.6.jar
BOOT-INF/lib/aircompressor-0.8.jar
BOOT-INF/lib/orc-mapreduce-1.4.1-nohive.jar
BOOT-INF/lib/parquet-column-1.8.2.jar
BOOT-INF/lib/parquet-common-1.8.2.jar
BOOT-INF/lib/parquet-encoding-1.8.2.jar
BOOT-INF/lib/parquet-hadoop-1.8.2.jar
BOOT-INF/lib/parquet-format-2.3.1.jar
BOOT-INF/lib/parquet-jackson-1.8.2.jar
BOOT-INF/lib/arrow-vector-0.8.0.jar
BOOT-INF/lib/arrow-format-0.8.0.jar
BOOT-INF/lib/arrow-memory-0.8.0.jar
BOOT-INF/lib/hppc-0.7.2.jar
BOOT-INF/lib/flatbuffers-1.2.0-3f79e055.jar
BOOT-INF/lib/spark-yarn_2.11-2.3.0.jar
BOOT-INF/lib/hadoop-yarn-api-2.6.5.jar
BOOT-INF/lib/hadoop-yarn-common-2.6.5.jar
BOOT-INF/lib/jaxb-api-2.3.0.jar
BOOT-INF/lib/jetty-util-6.1.26.jar
BOOT-INF/lib/jackson-jaxrs-1.9.13.jar
BOOT-INF/lib/jackson-xc-1.9.13.jar
BOOT-INF/lib/commons-cli-1.2.jar
BOOT-INF/lib/guice-servlet-3.0.jar
BOOT-INF/lib/commons-io-2.4.jar
BOOT-INF/lib/guice-3.0.jar
BOOT-INF/lib/javax.inject-1.jar
BOOT-INF/lib/aopalliance-1.0.jar
BOOT-INF/lib/log4j-1.2.17.jar
BOOT-INF/lib/hadoop-yarn-server-web-proxy-2.6.5.jar
BOOT-INF/lib/hadoop-yarn-server-common-2.6.5.jar
BOOT-INF/lib/jetty-6.1.26.jar
BOOT-INF/lib/hadoop-yarn-client-2.6.5.jar
BOOT-INF/lib/hadoop-aws-2.7.3.jar
BOOT-INF/lib/hadoop-common-2.7.3.jar
BOOT-INF/lib/xmlenc-0.52.jar
BOOT-INF/lib/commons-collections-3.2.2.jar
BOOT-INF/lib/servlet-api-2.5.jar
BOOT-INF/lib/jsp-api-2.1.jar
BOOT-INF/lib/jersey-core-1.19.1.jar
BOOT-INF/lib/jsr311-api-1.1.1.jar
BOOT-INF/lib/jersey-json-1.9.jar
BOOT-INF/lib/jettison-1.1.jar
BOOT-INF/lib/jaxb-impl-2.2.3-1.jar
BOOT-INF/lib/jersey-server-1.19.1.jar
BOOT-INF/lib/commons-logging-1.1.3.jar
BOOT-INF/lib/commons-configuration-1.6.jar
BOOT-INF/lib/commons-digester-1.8.jar
BOOT-INF/lib/commons-beanutils-1.7.0.jar
BOOT-INF/lib/commons-beanutils-core-1.8.0.jar
BOOT-INF/lib/hadoop-auth-2.7.3.jar
BOOT-INF/lib/apacheds-kerberos-codec-2.0.0-M15.jar
BOOT-INF/lib/apacheds-i18n-2.0.0-M15.jar
BOOT-INF/lib/api-asn1-api-1.0.0-M20.jar
BOOT-INF/lib/api-util-1.0.0-M20.jar
BOOT-INF/lib/jsch-0.1.54.jar
BOOT-INF/lib/curator-client-2.7.1.jar
BOOT-INF/lib/htrace-core-3.1.0-incubating.jar
BOOT-INF/lib/zookeeper-3.4.6.jar
BOOT-INF/lib/jackson-annotations-2.9.0.jar
BOOT-INF/lib/aws-java-sdk-1.7.4.jar
BOOT-INF/lib/spark-hive_2.11-2.3.0.jar
BOOT-INF/lib/parquet-hadoop-bundle-1.6.0.jar
BOOT-INF/lib/hive-exec-1.2.1.spark2.jar
BOOT-INF/lib/javolution-5.5.1.jar
BOOT-INF/lib/apache-log4j-extras-1.2.17.jar
BOOT-INF/lib/antlr-runtime-3.4.jar
BOOT-INF/lib/stringtemplate-3.2.1.jar
BOOT-INF/lib/ST4-4.0.4.jar
BOOT-INF/lib/JavaEWAH-0.3.2.jar
BOOT-INF/lib/snappy-0.2.jar
BOOT-INF/lib/stax-api-1.0.1.jar
BOOT-INF/lib/opencsv-2.3.jar
BOOT-INF/lib/hive-metastore-1.2.1.spark2.jar
BOOT-INF/lib/bonecp-0.8.0.RELEASE.jar
BOOT-INF/lib/datanucleus-api-jdo-3.2.6.jar
BOOT-INF/lib/datanucleus-rdbms-3.2.9.jar
BOOT-INF/lib/commons-pool-1.6.jar
BOOT-INF/lib/commons-dbcp-1.4.jar
BOOT-INF/lib/jdo-api-3.0.1.jar
BOOT-INF/lib/jta-1.1.jar
BOOT-INF/lib/commons-httpclient-3.1.jar
BOOT-INF/lib/calcite-avatica-1.2.0-incubating.jar
BOOT-INF/lib/calcite-core-1.2.0-incubating.jar
BOOT-INF/lib/calcite-linq4j-1.2.0-incubating.jar
BOOT-INF/lib/eigenbase-properties-1.1.5.jar
BOOT-INF/lib/httpclient-4.5.5.jar
BOOT-INF/lib/jackson-mapper-asl-1.9.13.jar
BOOT-INF/lib/commons-codec-1.11.jar
BOOT-INF/lib/joda-time-2.9.9.jar
BOOT-INF/lib/jodd-core-3.5.2.jar
BOOT-INF/lib/datanucleus-core-3.2.10.jar
BOOT-INF/lib/libthrift-0.9.3.jar
BOOT-INF/lib/libfb303-0.9.3.jar
BOOT-INF/lib/derby-10.14.1.0.jar
BOOT-INF/lib/core-tech-starter-docker-spring-boot-1.0.0.jar

尝试运行

spark-submit --master yarn --deploy-mode client --driver-memory 10G --executor-memory 15G --class com.ati.media.ref.scd.ScheduleApplication

我看到错误为

java.lang.ClassNotFoundException: com.ati.media.ref.scd.ScheduleApplication
        at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:348)
        at org.apache.spark.util.Utils$.classForName(Utils.scala:235)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:836)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:197)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:227)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

当我尝试使用

java -jar <jar>应用启动后没有任何问题。

java -cp的错误结果与未提交主校准的火花提交相同。

这是怎么了。我在ScheduleApplication中具有主要功能。

1 个答案:

答案 0 :(得分：0)

您需要指定--jars <path to your jar>或在集群级别配置Extrac jars的位置

来自AWS documentation：

  

引导操作或自定义配置将覆盖该类   路径。发生这种情况时，类加载器仅拾取JAR文件   存在于您在配置中指定的位置。

解决此问题的方法很少：

1）指定--jars和jar的路径：

spark-submit --deploy-mode client --class org.apache.spark.examples.SparkPi --master yarn spark-examples.jar 100 --jars /home/hadoop/extrajars/*

2）将自定义JAR的路径（包含缺少的类）添加到Spark类路径。

在 /etc/spark/conf/spark-defaults.conf 中，将自定义JAR的路径附加到错误堆栈跟踪中指定的类名称。在以下示例中， / home / hadoop / extrajars / * 是自定义JAR路径。

例如（如何修改正在运行的群集）：

sudo vim /etc/spark/conf/spark-defaults.conf

spark.driver.extraClassPath :/home/hadoop/extrajars/*
spark.executor.extraClassPath :/home/hadoop/extrajars/*

有关更多信息，或如何在新群集上执行此操作，请参考documentation

相关问题
最新问题