Question

我一直在尝试使用etsy的jvm profiler [https://github.com/etsy/statsd-jvm-profiler来运行我的火花应用来配置我的火花应用程序，但是我遇到了执行问题。

使用给定命令运行时：

spark/bin/spark-submit --conf spark.local.dir=/local/spark-scratch-space --conf spark.yarn.queue=streaming --conf 'spark.driver.extraJavaOptions=-javaagent:statsd-jvm-profiler-2.1.0-jar-with-dependencies.jar=server=x.x.x.x,port=8086,reporter=InfluxDBReporter,database=profiler,username=profiler,password=profiler,prefix=MySparkAPP,tagMapping=US_Offline' --conf 'spark.executor.extraJavaOptions=-javaagent:statsd-jvm-profiler-2.1.0-jar-with-dependencies.jar=server=x.x.x.x,port=8086,reporter=InfluxDBReporter,database=profiler,username=profiler,password=profiler,prefix=MySparkAPP,tagMapping=US_Offline' --class someClass --master yarn-client --num-executors 5 --executor-cores 3 --executor-memory 2G --jars /local/shprin/statsd-jvm-profiler-2.1.0-jar-with-dependencies.jar MyApp

失败并出现错误：

Error opening zip file or JAR manifest missing : statsd-jvm-profiler-2.1.0-jar-with-dependencies.jar Error occurred during initialization of VM agent library failed to init: instrument

我已从此链接下载了预编译的jar：

wget -qc https://github.com/etsy/statsd-jvm-profiler/releases/download/2.1.0/statsd-jvm-profiler-2.1.0-jar-with-dependencies.jar

我对此都很陌生，我正在寻求帮助来理解出了什么问题，因为如果没有探查器运行火花应用，运行就好了。

我需要指针来查看和探索以解决这个问题。任何帮助，欢迎提出建议。

我正在使用java8，而spark应用程序是用scala编写的。我已经检查了statsd-jvm-profiler-2.1.0-jar-with-dependencies.jar中存在的清单文件

资源管理器日志：

＆＃13;

Exception from resource manager : 
18/06/12 17:05:04 INFO yarn.YarnAllocator: Completed container container_e02_1525632635880_0582_01_000012 on host: cluster-nm-1a-d28x-a19182d8.us-east-1.amazon.com (state: COMPLETE, exit status: 1)
18/06/12 17:05:04 WARN yarn.YarnAllocator: Container marked as failed: container_e02_1525632635880_0582_01_000012 on host: cluster-nm-1a-d28x-a19182d8.us-east-1.amazon.com. Exit status: 1. Diagnostics: Exception from container-launch.
Container id: container_e02_1525632635880_0582_01_000012
Exit code: 1
Stack trace: ExitCodeException exitCode=1: 
	at org.apache.hadoop.util.Shell.runCommand(Shell.java:545)
	at org.apache.hadoop.util.Shell.run(Shell.java:456)
	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722)
	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1

＆＃13;

使用Etsy的statsD探查器来分析火花作业

0 个答案: