当我要将数据从SQL导入HDFS时我会发现错误说明
org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
我正在为我的终端提供终端活动
student@ubuntu:~$ sqoop import --connect jdbc:mysql://localhost:3306/p \
> --username root \
> --password student \
> --table p \
> -m \
> 1;
Warning: /home/student/Applications/sqoop/../hbase dofes not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /home/student/Applications/sqoop/../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /home/student/Applications/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /home/student/Applications/sqoop/../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
15/10/23 04:23:49 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
15/10/23 04:23:49 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/10/23 04:23:52 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
15/10/23 04:23:52 INFO tool.CodeGenTool: Beginning code generation
15/10/23 04:24:00 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `p` AS t LIMIT 1
15/10/23 04:24:01 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `p` AS t LIMIT 1
15/10/23 04:24:01 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/student/Applications/hadoop
Note: /tmp/sqoop-student/compile/d0a3526dcf308f25f4333c8558068bb8/p.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
15/10/23 04:25:03 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-student/compile/d0a3526dcf308f25f4333c8558068bb8/p.jar
15/10/23 04:25:04 WARN manager.MySQLManager: It looks like you are importing from mysql.
15/10/23 04:25:04 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
15/10/23 04:25:04 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
15/10/23 04:25:04 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
15/10/23 04:25:05 INFO mapreduce.ImportJobBase: Beginning import of p
15/10/23 04:25:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/10/23 04:25:31 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
15/10/23 04:26:02 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
15/10/23 04:26:04 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032
15/10/23 04:26:40 INFO db.DBInputFormat: Using read commited transaction isolation
15/10/23 04:26:41 INFO mapreduce.JobSubmitter: number of splits:1
15/10/23 04:26:45 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1445598425022_0001
15/10/23 04:26:53 INFO impl.YarnClientImpl: Submitted application application_1445598425022_0001
15/10/23 04:26:54 INFO mapreduce.Job: The url to track the job: http://ubuntu:8088/proxy/application_1445598425022_0001/
15/10/23 04:26:54 INFO mapreduce.Job: Running job: job_1445598425022_0001
15/10/23 04:28:24 INFO mapreduce.Job: Job job_1445598425022_0001 running in uber mode : false
15/10/23 04:28:25 INFO mapreduce.Job: map 0% reduce 0%
15/10/23 04:28:41 INFO mapreduce.Job: Task Id : attempt_1445598425022_0001_m_000000_0, Status : FAILED
Container launch failed for container_1445598425022_0001_01_000002 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
15/10/23 04:28:43 INFO mapreduce.Job: Task Id : attempt_1445598425022_0001_m_000000_1, Status : FAILED
Container launch failed for container_1445598425022_0001_01_000003 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
15/10/23 04:28:43 INFO mapreduce.Job: Task Id : attempt_1445598425022_0001_m_000000_2, Status : FAILED
Container launch failed for container_1445598425022_0001_01_000004 : org.apache.hadoop.yarn.exceptions.InvalidAuxServiceException: The auxService:mapreduce_shuffle does not exist
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.instantiateException(SerializedExceptionPBImpl.java:168)
at org.apache.hadoop.yarn.api.records.impl.pb.SerializedExceptionPBImpl.deSerialize(SerializedExceptionPBImpl.java:106)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$Container.launch(ContainerLauncherImpl.java:155)
at org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl$EventProcessor.run(ContainerLauncherImpl.java:369)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
15/10/23 04:28:47 INFO mapreduce.Job: map 100% reduce 0%
15/10/23 04:28:56 INFO mapreduce.Job: Job job_1445598425022_0001 failed with state FAILED due to: Task failed task_1445598425022_0001_m_000000
Job failed as tasks failed. failedMaps:1 failedReduces:0
15/10/23 04:29:01 INFO mapreduce.Job: Counters: 3
Job Counters
Other local map tasks=4
Total time spent by all maps in occupied slots (ms)=0
Total time spent by all reduces in occupied slots (ms)=0
15/10/23 04:29:02 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=FAILED. Redirecting to job history server
15/10/23 04:29:13 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/10/23 04:29:14 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/10/23 04:29:15 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/10/23 04:29:16 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/10/23 04:29:17 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/10/23 04:29:18 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/10/23 04:29:19 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/10/23 04:29:20 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/10/23 04:29:21 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/10/23 04:29:22 INFO ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:10020. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
15/10/23 04:29:23 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=FAILED. Redirecting to job history server
我如何克服这个？
答案 0 :(得分：6)
我遇到了同样的情况。
要解决此问题，您必须检查yarn-site.xml，使其与以下代码段匹配。
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
</configuration>