我正在尝试使用sqoop将表从mysql导入HDFS ..它会抛出 java.io.IOException错误无法创建目标文件夹
[root@01HW288075 hadoop]# sudo -u hdfs sqoop import --username user --password pass --connect jdbc:mysql://172.16.176.109/pocdb --table stocks --verbose
Warning: /usr/lib/hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
14/07/30 09:40:24 INFO sqoop.Sqoop: Running Sqoop version: 1.4.3-cdh4.7.0
14/07/30 09:40:24 DEBUG tool.BaseSqoopTool: Enabled debug logging.
14/07/30 09:40:24 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
14/07/30 09:40:24 DEBUG sqoop.ConnFactory: Loaded manager factory: com.cloudera.sqoop.manager.DefaultManagerFactory
14/07/30 09:40:24 DEBUG sqoop.ConnFactory: Trying ManagerFactory: com.cloudera.sqoop.manager.DefaultManagerFactory
14/07/30 09:40:24 DEBUG manager.DefaultManagerFactory: Trying with scheme: jdbc:mysql:
14/07/30 09:40:24 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
14/07/30 09:40:24 DEBUG sqoop.ConnFactory: Instantiated ConnManager org.apache.sqoop.manager.MySQLManager@1385660
14/07/30 09:40:24 INFO tool.CodeGenTool: Beginning code generation
14/07/30 09:40:24 DEBUG manager.SqlManager: Execute getColumnTypesRawQuery : SELECT t.* FROM `stocks` AS t LIMIT 1
14/07/30 09:40:24 DEBUG manager.SqlManager: No connection paramenters specified. Using regular API for making connection.
14/07/30 09:40:25 DEBUG manager.SqlManager: Using fetchSize for next query: -2147483648
14/07/30 09:40:25 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `stocks` AS t LIMIT 1
14/07/30 09:40:25 DEBUG orm.ClassWriter: selected columns:
14/07/30 09:40:25 DEBUG orm.ClassWriter: id
14/07/30 09:40:25 DEBUG orm.ClassWriter: symbol
14/07/30 09:40:25 DEBUG orm.ClassWriter: quote_date
14/07/30 09:40:25 DEBUG orm.ClassWriter: open_price
14/07/30 09:40:25 DEBUG orm.ClassWriter: high_price
14/07/30 09:40:25 DEBUG orm.ClassWriter: low_price
14/07/30 09:40:25 DEBUG orm.ClassWriter: close_price
14/07/30 09:40:25 DEBUG orm.ClassWriter: volume
14/07/30 09:40:25 DEBUG orm.ClassWriter: adj_close_price
14/07/30 09:40:25 DEBUG manager.SqlManager: Using fetchSize for next query: -2147483648
14/07/30 09:40:25 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `stocks` AS t LIMIT 1
14/07/30 09:40:25 DEBUG orm.ClassWriter: Writing source file: /tmp/sqoop-hdfs/compile/7fcf81959d64c2f761e51076be612e9d/stocks.java
14/07/30 09:40:25 DEBUG orm.ClassWriter: Table name: stocks
14/07/30 09:40:25 DEBUG orm.ClassWriter: Columns: id:4, symbol:12, quote_date:12, open_price:8, high_price:8, low_price:8, close_price:8, volume:4, adj_close_price:8,
14/07/30 09:40:25 DEBUG orm.ClassWriter: sourceFilename is stocks.java
14/07/30 09:40:25 DEBUG orm.CompilationManager: Found existing /tmp/sqoop-hdfs/compile/7fcf81959d64c2f761e51076be612e9d/
14/07/30 09:40:25 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/lib/hadoop-mapreduce
14/07/30 09:40:25 DEBUG orm.CompilationManager: Returning jar file path /usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core-2.0.0-cdh4.7.0.jar:/usr/lib/hadoop-mapreduce/hadoop-mapreduce-client-core.jar
14/07/30 09:40:25 DEBUG orm.CompilationManager: Adding source file: /tmp/sqoop-hdfs/compile/7fcf81959d64c2f761e51076be612e9d/stocks.java
14/07/30 09:40:25 DEBUG orm.CompilationManager: Invoking javac with args:
14/07/30 09:40:25 DEBUG orm.CompilationManager: -sourcepath
14/07/30 09:40:25 DEBUG orm.CompilationManager: /tmp/sqoop-hdfs/compile/7fcf81959d64c2f761e51076be612e9d/
14/07/30 09:40:25 DEBUG orm.CompilationManager: -d
14/07/30 09:40:25 DEBUG orm.CompilationManager: /tmp/sqoop-hdfs/compile/7fcf81959d64c2f761e51076be612e9d/
14/07/30 09:40:25 DEBUG orm.CompilationManager: -classpath
Note: /tmp/sqoop-hdfs/compile/7fcf81959d64c2f761e51076be612e9d/stocks.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
14/07/30 09:40:26 ERROR orm.CompilationManager: Could not make directory: /home/root1/hadoop/.
14/07/30 09:40:26 DEBUG orm.CompilationManager: Could not rename /tmp/sqoop-hdfs/compile/7fcf81959d64c2f761e51076be612e9d/stocks.java to /home/root1/hadoop/./stocks.java
java.io.IOException: Destination '/home/root1/hadoop/.' directory cannot be created
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:882)
at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:835)
at org.apache.commons.io.FileUtils.moveFile(FileUtils.java:2385)
at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:239)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:396)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:506)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:222)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:231)
at org.apache.sqoop.Sqoop.main(Sqoop.java:240)
14/07/30 09:40:26 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hdfs/compile/7fcf81959d64c2f761e51076be612e9d/stocks.jar
14/07/30 09:40:26 DEBUG orm.CompilationManager: Scanning for .class files in directory: /tmp/sqoop-hdfs/compile/7fcf81959d64c2f761e51076be612e9d
14/07/30 09:40:26 DEBUG orm.CompilationManager: Got classfile: /tmp/sqoop-hdfs/compile/7fcf81959d64c2f761e51076be612e9d/stocks.class -> stocks.class
14/07/30 09:40:26 DEBUG orm.CompilationManager: Finished writing jar file /tmp/sqoop-hdfs/compile/7fcf81959d64c2f761e51076be612e9d/stocks.jar
14/07/30 09:40:26 WARN manager.MySQLManager: It looks like you are importing from mysql.
14/07/30 09:40:26 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
14/07/30 09:40:26 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
14/07/30 09:40:26 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
14/07/30 09:40:26 DEBUG manager.MySQLManager: Rewriting connect string to jdbc:mysql://172.16.176.109/pocdb?zeroDateTimeBehavior=convertToNull
14/07/30 09:40:26 DEBUG manager.CatalogQueryManager: Retrieving primary key for table 'stocks' with query SELECT column_name FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = (SELECT SCHEMA()) AND TABLE_NAME = 'stocks' AND COLUMN_KEY = 'PRI'
14/07/30 09:40:26 DEBUG manager.CatalogQueryManager: Retrieving primary key for table 'stocks' with query SELECT column_name FROM INFORMATION_SCHEMA.COLUMNS WHERE TABLE_SCHEMA = (SELECT SCHEMA()) AND TABLE_NAME = 'stocks' AND COLUMN_KEY = 'PRI'
14/07/30 09:40:26 INFO mapreduce.ImportJobBase: Beginning import of stocks
14/07/30 09:40:26 WARN conf.Configuration: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
14/07/30 09:40:27 WARN conf.Configuration: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/07/30 09:40:27 DEBUG db.DBConfiguration: Securing password into job credentials store
14/07/30 09:40:27 DEBUG mapreduce.DataDrivenImportJob: Using table class: stocks
14/07/30 09:40:27 DEBUG mapreduce.DataDrivenImportJob: Using InputFormat: class com.cloudera.sqoop.mapreduce.db.DataDrivenDBInputFormat
14/07/30 09:40:28 WARN conf.Configuration: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/sqoop-1.4.3-cdh4.7.0.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/mysql-connector-java-5.1.13-bin.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/sqoop-1.4.3-cdh4.7.0.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/sqoop-1.4.3-cdh4.7.0.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/jackson-core-asl-1.8.8.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/netty-3.4.0.Final.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/paranamer-2.3.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/avro-mapred-1.7.4-hadoop2.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/servlet-api-2.5-20081211.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/avro-ipc-1.7.4-tests.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/avro-1.7.4.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/jetty-util-6.1.26.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/snappy-java-1.0.4.1.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/ant-contrib-1.0b3.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/commons-io-1.4.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/ant-eclipse-1.0-jvm1.2.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/mysql-connector-java-5.1.13-bin.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/commons-compress-1.4.1.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/jackson-mapper-asl-1.8.8.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/xz-1.0.jar
14/07/30 09:40:28 DEBUG mapreduce.JobBase: Adding to job classpath: file:/usr/lib/sqoop/lib/hsqldb-1.8.0.10.jar
14/07/30 09:40:28 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is inited.
14/07/30 09:40:28 INFO service.AbstractService: Service:org.apache.hadoop.yarn.client.YarnClientImpl is started.
14/07/30 09:40:29 DEBUG db.DBConfiguration: Fetching password from job credentials store
14/07/30 09:40:29 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`id`), MAX(`id`) FROM `stocks`
14/07/30 09:40:29 DEBUG db.IntegerSplitter: Splits: [ 1 to 45] into 4 parts
14/07/30 09:40:29 DEBUG db.IntegerSplitter: 1
14/07/30 09:40:29 DEBUG db.IntegerSplitter: 12
14/07/30 09:40:29 DEBUG db.IntegerSplitter: 23
14/07/30 09:40:29 DEBUG db.IntegerSplitter: 34
14/07/30 09:40:29 DEBUG db.IntegerSplitter: 45
14/07/30 09:40:29 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '`id` >= 1' and upper bound '`id` < 12'
14/07/30 09:40:29 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '`id` >= 12' and upper bound '`id` < 23'
14/07/30 09:40:29 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '`id` >= 23' and upper bound '`id` < 34'
14/07/30 09:40:29 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '`id` >= 34' and upper bound '`id` <= 45'
14/07/30 09:40:29 INFO mapreduce.JobSubmitter: number of splits:4
14/07/30 09:40:29 WARN conf.Configuration: mapred.job.classpath.files is deprecated. Instead, use mapreduce.job.classpath.files
14/07/30 09:40:29 WARN conf.Configuration: mapred.cache.files is deprecated. Instead, use mapreduce.job.cache.files
14/07/30 09:40:29 WARN conf.Configuration: mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
14/07/30 09:40:29 WARN conf.Configuration: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/07/30 09:40:29 WARN conf.Configuration: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/07/30 09:40:29 WARN conf.Configuration: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/07/30 09:40:29 WARN conf.Configuration: mapreduce.inputformat.class is deprecated. Instead, use mapreduce.job.inputformat.class
14/07/30 09:40:29 WARN conf.Configuration: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/07/30 09:40:29 WARN conf.Configuration: mapreduce.outputformat.class is deprecated. Instead, use mapreduce.job.outputformat.class
14/07/30 09:40:29 WARN conf.Configuration: mapred.cache.files.timestamps is deprecated. Instead, use mapreduce.job.cache.files.timestamps
14/07/30 09:40:29 WARN conf.Configuration: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/07/30 09:40:29 WARN conf.Configuration: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/07/30 09:40:29 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1406367087547_0013
14/07/30 09:40:30 INFO client.YarnClientImpl: Submitted application application_1406367087547_0013 to ResourceManager at /0.0.0.0:8032
14/07/30 09:40:30 INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1406367087547_0013/
14/07/30 09:40:30 INFO mapreduce.Job: Running job: job_1406367087547_0013
我有MySQl的连接。我能够在sqoop中列出数据库和list-the-tables所以连接问题不存在..
如果我转到建议的网址http://localhost:8088/proxy/application_1406367087547_0013/，我会收到消息＆＃34;请求的应用程序似乎尚未运行，并且尚未设置跟踪URL。＆＃34;
请帮助我解决这个问题。
答案 0 :(得分：1)
创建目录/ home / root1 / hadoop /并使其成为可写的。
sudo mkdir -p /home/root1/hadoop
sudo chmod +w /home/root1/hadoop
再次启动所有这些并查找结果。只有当你不是root（终端上的$符号）
时才需要sudo答案 1 :(得分：0)
我记不住了，但是当我没有指定分区列名时，也许我有类似的问题。所以请尝试以下命令：
sqoop import --connect jdbc:mysql://172.16.176.109:3306/pocdb --username user --password pass --table stocks --target-dir /tmp/target -m 1
希望有所帮助
答案 2 :(得分：0)
从日志消息中，导入目录/home/root1/hadoop/.的最后一部分是一个点。这肯定会失败。我认为这是Sqoop的一个错误。我试过Sqoop 1.4.6，导入目录永远不是一个点。如果您可以共享您的Hadoop和Sqoop版本信息，它可能会有所帮助。
您可以做的是明确提供导入路径--target-dir /user/your_name/sqoopimport。但是你应该确保你拥有这个位置的写权限。