<sqoop xmlns="uri:oozie:sqoop-action:0.3">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<arg>job</arg>
<arg>--meta-connect</arg>
<arg>jdbc:mysql://FQDN:3306/sqoop</arg>
<arg>--exec</arg>
<arg>fabric_inventory</arg>
</sqoop>
现在,要传递--meta-connect的用户名和密码,
如果我在oozie.xml中传递它:
<arg>jdbc:mysql://FQDN:3306/sqoop?user=sqoop&password=sqoop</arg>
或
<arg>jdbc:mysql://FQDN:3306/sqoop?user=sqoop&password=sqoop</arg>
它提供以下例外:
运行导入作业遇到IOException:java.io.IOException:没有为ClassWriter生成的列 2.它不需要
--username and --password prop and val in arg tag.
如何以正确的方式传递值?
这就是完整的异常:
2016-05-11 10:21:44,920 ERROR [main] tool.ImportTool (ImportTool.java:run(613)) - Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:107)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:228)
at org.apache.sqoop.tool.JobTool.run(JobTool.java:283)
at org.apache.sqoop.Sqoop.run(Sqoop.java:148)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:184)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:226)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:235)
at org.apache.sqoop.Sqoop.main(Sqoop.java:244)
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:197)
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:177)
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:241)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
答案 0 :(得分:1)
如果你弄清楚Sqoop 1 和Sqoop 2 之间的混淆,那么你也必须跳过Oozie障碍。
我的建议:停止使用命令行参数并使用标准的Hadoop配置文件。
1。在您的网关节点上(针对单元测试),修改/etc/sqoop/conf/sqoop-site.xml
以设置客户端属性
<?xml version="1.0" encoding="UTF-8"?>
<configuration>
<!-- some stuff here that does not matter for Metastore -->
...
<!-- now the Metastore config -->
<property>
<name>sqoop.metastore.client.enable.autoconnect</name>
<value>true</value>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.url</name>
<value>jdbc:hsqldb:hsql://FQDN:16000/sqoop</value>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.username</name>
<value>sa</value>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.password</name>
<value></value>
</property>
<property>
<name>sqoop.metastore.client.record.password</name>
<value>false</value>
</property>
</configuration>
1b。将该文件上传到HDFS(用于Oozie作业)
2. 在将实际运行全局Metastore DB的节点上,还要编辑该文件,并添加两个额外的服务器属性(在此示例中,DB文件存储在{ {1}})
/var/lib/...
2b。确保您不时检查该数据库(重置&#34;脚本&#34;文件并刷新&#34;重做日志&#34;文件)然后备份&#34;脚本&#34;文件作为当前数据库状态的快照,在某个安全的地方,以防您丢失节点及其磁盘 - 是的,这些事情发生了
3. 在Oozie Sqoop操作中,使用 <property>
<name>sqoop.metastore.server.port</name>
<value>16000</value>
</property>
<property>
<name>sqoop.metastore.server.location</name>
<value>/var/lib/sqoop/data/shared.db</value>
</property>
条目设置客户端属性,以HDFS中的配置文件为目标。
如果您对处理这些道具和Metastore客户端的实际Sqoop源代码感兴趣,请查看there
答案 1 :(得分:0)
- 元连接应具有jdbc:hsqldb:hsql://<server-name>:<port>/sqoop
格式的值(默认为hsqldb)。在您的情况下,您可以将其更改为mysql。
您可以在sqoop-site.xml中覆盖以下属性以传递Metastore用户名和密码:
<property>
<name>sqoop.metastore.client.autoconnect.username</name>
<value>SA</value>
<description>The username to bind to the metastore.
</description>
</property>
<property>
<name>sqoop.metastore.client.autoconnect.password</name>
<value></value>
<description>The password to bind to the metastore.
</description>
</property>
- 元连接将从此处读取用户名和密码,并将连接到数据库。
您可以参考default sqoop-site.xml了解更多详情