Question

我有一个shell脚本，遍历一个表列表，在这种情况下是460个表，并且一次创建10个sqoop导入作业，等待那些完成，然后为接下来的10个表再创建10个表等等，直到所有表都已导入。

我看到一些表上的失败，每次都有不同的表，与sqoop Metastore相关联。上一次我跑了，在460张桌子中，11个工作失败了

请参阅以下其中一项工作的例外情况。他们都有相同的例外

18/06/07 21:55:16 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-mapr-1710
18/06/07 21:55:19 ERROR tool.JobTool: I/O error performing job operation: java.io.IOException: Exception creating SQL connection
        at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.init(HsqldbJobStorage.java:216)
        at org.apache.sqoop.metastore.hsqldb.AutoHsqldbStorage.open(AutoHsqldbStorage.java:112)
        at org.apache.sqoop.tool.JobTool.run(JobTool.java:274)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.sql.SQLException: File input/output error: /home/network.local/s-mapr-etl-dev/.sqoop/metastore.db.script.new
        at org.hsqldb.jdbc.Util.sqlException(Unknown Source)
        at org.hsqldb.jdbc.jdbcConnection.<init>(Unknown Source)
        at org.hsqldb.jdbcDriver.getConnection(Unknown Source)
        at org.hsqldb.jdbcDriver.connect(Unknown Source)
        at java.sql.DriverManager.getConnection(DriverManager.java:664)
        at java.sql.DriverManager.getConnection(DriverManager.java:247)
        at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.init(HsqldbJobStorage.java:176)
        ... 8 more

我已经读过这可能与被一个作业锁定的Metastore有关，如果另一个作业试图访问它，那么它将失败并且这与sqoop Metastore的默认配置如何与HSQL一起工作有关

有关类似问题，请参阅这些文章：

http://grokbase.com/t/sqoop/user/143t74crfj/getting-weird-exception-while-running-sqoop-jobs

https://community.hortonworks.com/questions/41385/concurrent-access-to-the-sqoop-metastore.html

我也遇到过这个错误： https://community.hortonworks.com/questions/144276/sqoop-suddenly-fails-to-connect-to-meta-store-user.html

是否有其他人遇到此问题并有解决方案/解决方法？

并行Sqoop导入失败

0 个答案: