我已经尝试了多种方法来在oozie中创建一个蜂巢作业,我正在使用Hortonworks-Sandbox进行开发环境。我遵循https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/bk_workflow-management/content/hive_parameters.html网站上的指示(还有许多其他指示,但没有运气)。
我的workflow.xml文件是:
<?xml version="1.0" encoding="UTF-8" standalone="no"?><workflow-app xmlns="uri:oozie:workflow:0.5" name="test">
<start to="Create_external_table"/>
<action name="Create_external_table">
<hive xmlns="uri:oozie:hive-action:0.6">
<job-tracker>${resourceManager}</job-tracker>
<name-node>${nameNode}</name-node>
<job-xml>/user/oozie/workflows/hive-conf.xml</job-xml>
<configuration>
<property>
<name>hive.metastore.uris</name>
<value>thrift://sandbox-hdp.hortonworks.com:9083</value>
</property>
</configuration>
<query>CREATE EXTERNAL TABLE IF NOT EXISTS behavior_data.log_files (Id STRING, PortalId int, CampaignId int,BannerAssetId int, BannerAssetFileId int, Timestamp int, EventType int, Price float,Msisdn STRING,UserId STRING,hitDate STRING, hitTime int) COMMENT "log_files" ROW FORMAT DELIMITED FIELDS TERMINATED BY "|" STORED AS TEXTFILE LOCATION "/user/root/log_files/flumefiles/20180911/13";</query>
<file>/user/root/log_files/flumefiles/20180911/13</file>
</hive>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>${wf:errorMessage(wf:lastErrorNode())}</message>
</kill>
<end name="end"/>
配置如下所示。
我从ambari-workflow管理器中得到的错误是:
`-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] Start action [0000013-180914115041330-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-09-14 15:44:36,823 INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] [***0000013-180914115041330-oozie-oozi-W@:start:***]Action status=DONE
2018-09-14 15:44:36,823 INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] [***0000013-180914115041330-oozie-oozi-W@:start:***]Action updated in DB!
2018-09-14 15:44:36,888 INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000013-180914115041330-oozie-oozi-W@:start:
2018-09-14 15:44:36,889 INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000013-180914115041330-oozie-oozi-W
2018-09-14 15:44:36,918 INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Start action [0000013-180914115041330-oozie-oozi-W@Create_external_table] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-09-14 15:44:39,662 INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 15:44:39,712 INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]
2018-09-14 15:44:39,716 INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] [***0000013-180914115041330-oozie-oozi-W@Create_external_table***]Action status=RUNNING
2018-09-14 15:44:39,716 INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] [***0000013-180914115041330-oozie-oozi-W@Create_external_table***]Action updated in DB!
2018-09-14 15:44:39,720 INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] No Notification URL is defined. Therefore nothing to notify for job 0000013-180914115041330-oozie-oozi-W@Create_external_table
2018-09-14 15:55:00,272 INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 15:55:00,295 INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]
2018-09-14 16:06:00,600 INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 16:06:00,623 INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]
2018-09-14 16:17:00,299 INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 16:17:00,319 INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]`
从/var/log/oozie/oozie-error.log文件中:
2018-09-14 16:12:57,397 WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2018-09-14 16:12:57,408 WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/oozie/workfl
ows/workflow_otinanai/lib] does not exist
2018-09-18 08:20:38,605 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]
2018-09-18 08:20:38,655 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]
2018-09-18 08:20:38,726 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]
2018-09-18 08:20:38,846 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]
2018-09-18 08:20:38,892 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]
2018-09-18 08:20:38,942 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]
2018-09-18 08:20:38,943 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]
2018-09-18 08:20:39,035 WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming
2018-09-18 08:20:42,131 WARN Services:523 - SERVER[] Previous services singleton active, destroying it
2018-09-18 08:20:42,189 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]
2018-09-18 08:20:42,204 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]
2018-09-18 08:20:42,213 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]
2018-09-18 08:20:42,214 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]
2018-09-18 08:20:42,214 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]
2018-09-18 08:20:42,214 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]
2018-09-18 08:20:42,215 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]
2018-09-18 08:20:42,216 WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming
2018-09-18 08:20:45,892 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]
2018-09-18 08:20:45,895 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]
2018-09-18 08:20:45,902 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]
2018-09-18 08:20:45,902 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]
2018-09-18 08:20:45,903 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]
2018-09-18 08:20:45,904 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]
2018-09-18 08:20:45,904 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]
2018-09-18 08:20:45,909 WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming
2018-09-18 08:20:48,136 WARN Services:523 - SERVER[] Previous services singleton active, destroying it
2018-09-18 08:20:48,359 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]
2018-09-18 08:20:48,391 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]
2018-09-18 08:20:48,445 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]
2018-09-18 08:20:48,454 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]
2018-09-18 08:20:48,469 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]
2018-09-18 08:20:48,498 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]
2018-09-18 08:20:48,500 WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]
2018-09-18 08:20:48,504 WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming
2018-09-18 08:20:55,819 WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] System property [oozie.https.keystore.pass] no defined in Oozie configuration, ignored
2018-09-18 08:20:55,821 WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] System property [oozie.admin.port] no defined in Oozie configuration, ignored
2018-09-18 08:20:55,822 WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] System property [oozie.https.keystore.file] no defined in Oozie configuration, ignored
2018-09-18 08:20:55,828 WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]
2018-09-18 08:20:55,831 WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]
2018-09-18 08:20:55,834 WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]
2018-09-18 08:20:55,835 WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]
2018-09-18 08:20:55,836 WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [hadoop.security.credential.provider.path]
2018-09-18 08:20:55,837 WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [credentialStoreClassPath]
2018-09-18 08:20:55,838 WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]
2018-09-18 08:20:55,844 WARN Services:523 - SERVER[sandbox-hdp.hortonworks.com] System ID [oozie-oozi] exceeds maximum length [10], trimming
2018-09-18 08:20:56,334 WARN ConfigUtils:523 - SERVER[sandbox-hdp.hortonworks.com] Using a deprecated configuration property [oozie.service.AuthorizationService.security.enabled], should use [oozie.service.Auth
orizationService.authorization.enabled]. Please delete the deprecated property in order for the new property to take effect.
2018-09-18 08:21:07,459 WARN SparkConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Spark Configuration could not be loaded for *: /usr/hdp/current/oozie-server/conf/spark-conf does not exist