sqoop hive导入作业长时间运行,其跟踪URL为UNASSIGNED

时间:2017-12-20 17:55:09

标签: hadoop import hive hdfs sqoop

我在单个节点集群上运行hadoop并尝试sqoop将mysql数据导入到hive表中。 问题是在执行sqoop作业时,进程占用的时间和数据不会导入,如下所示:

Sqoop命令:

sqoop job --create Hcustomers --  import --connect jdbc:mysql://localhost/retaildb --username root -P --table customers --check-column id --incremental append --last-value 0 --target-dir '/user/hive/retail/Hcustomers' -m 2;

执行sqoop作业后:

chaithu@localhost:~$ sqoop job --exec Hcustomers;
Warning: /opt/cloudera/parcels/CDH-5.13.1-1.cdh5.13.1.p0.2/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
17/12/20 22:47:27 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.13.1
Enter password: 
17/12/20 22:47:32 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/12/20 22:47:32 INFO tool.CodeGenTool: Beginning code generation
Wed Dec 20 22:47:32 IST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
17/12/20 22:47:32 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `customers` AS t LIMIT 1
17/12/20 22:47:32 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `customers` AS t LIMIT 1
17/12/20 22:47:32 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce
Note: /tmp/sqoop-chaithu/compile/fa134412efb9ef64f2cb5a5ebfd29956/customers.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/12/20 22:47:34 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-chaithu/compile/fa134412efb9ef64f2cb5a5ebfd29956/customers.jar
17/12/20 22:47:38 INFO tool.ImportTool: Maximal id query for free form incremental import: SELECT MAX(`id`) FROM `customers`
17/12/20 22:47:38 INFO tool.ImportTool: Incremental import based on column `id`
17/12/20 22:47:38 INFO tool.ImportTool: Lower bound value: 0
17/12/20 22:47:38 INFO tool.ImportTool: Upper bound value: 11
17/12/20 22:47:38 WARN manager.MySQLManager: It looks like you are importing from mysql.
17/12/20 22:47:38 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
17/12/20 22:47:38 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
17/12/20 22:47:38 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
17/12/20 22:47:38 INFO mapreduce.ImportJobBase: Beginning import of customers
17/12/20 22:47:38 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
17/12/20 22:47:38 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/12/20 22:47:38 INFO client.RMProxy: Connecting to ResourceManager at localhost/127.0.0.1:8032
17/12/20 22:47:39 INFO mapreduce.JobSubmissionFiles: Permissions on staging directory /user/chaithu/.staging are incorrect: rwxrwx---. Fixing permissions to correct value rwx------
Wed Dec 20 22:47:45 IST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
17/12/20 22:47:45 INFO db.DBInputFormat: Using read commited transaction isolation
17/12/20 22:47:45 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `customers` WHERE ( `id` > 0 AND `id` <= 11 )
17/12/20 22:47:45 INFO db.IntegerSplitter: Split size: 5; Num splits: 2 from: 1 to: 11
17/12/20 22:47:45 INFO mapreduce.JobSubmitter: number of splits:2
17/12/20 22:47:46 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1513788823385_0002
17/12/20 22:47:46 INFO impl.YarnClientImpl: Submitted application application_1513788823385_0002
17/12/20 22:47:46 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1513788823385_0002/
17/12/20 22:47:46 INFO mapreduce.Job: Running job: job_1513788823385_0002

如上所示,执行卡住了,没有导入数据。附加作业申请的屏幕截图。Sqoop Job Status on Hadoop job tracker

我只想尝试从mysql导入11行到hive,我认为这不应该花费太多时间。可能是什么问题。 请建议我一样。

0 个答案:

没有答案