使用工作流管理器在oozie中创建配置单元作业时出错

时间:2018-09-18 11:42:08

标签: hive sandbox oozie-workflow

我已经尝试了多种方法来在oozie中创建一个蜂巢作业,我正在使用Hortonworks-Sandbox进行开发环境。我遵循https://docs.hortonworks.com/HDPDocuments/Ambari-2.7.0.0/bk_workflow-management/content/hive_parameters.html网站上的指示(还有许多其他指示,但没有运气)。

我的workflow.xml文件是:

<?xml version="1.0" encoding="UTF-8" standalone="no"?><workflow-app xmlns="uri:oozie:workflow:0.5" name="test">
<start to="Create_external_table"/>
<action name="Create_external_table">
    <hive xmlns="uri:oozie:hive-action:0.6">
        <job-tracker>${resourceManager}</job-tracker>
        <name-node>${nameNode}</name-node>
        <job-xml>/user/oozie/workflows/hive-conf.xml</job-xml>
        <configuration>
            <property>
                <name>hive.metastore.uris</name>
                <value>thrift://sandbox-hdp.hortonworks.com:9083</value>
            </property>
        </configuration>
        <query>CREATE EXTERNAL TABLE IF NOT EXISTS behavior_data.log_files (Id STRING, PortalId int, CampaignId int,BannerAssetId int, BannerAssetFileId int, Timestamp int, EventType int, Price float,Msisdn STRING,UserId STRING,hitDate STRING, hitTime int) COMMENT "log_files" ROW FORMAT DELIMITED FIELDS TERMINATED BY "|" STORED AS TEXTFILE LOCATION "/user/root/log_files/flumefiles/20180911/13";</query>
        <file>/user/root/log_files/flumefiles/20180911/13</file>
    </hive>
    <ok to="end"/>
    <error to="kill"/>
</action>
<kill name="kill">
    <message>${wf:errorMessage(wf:lastErrorNode())}</message>
</kill>
<end name="end"/>

配置如下所示。

我从ambari-workflow管理器中得到的错误是:

    `-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] Start action [0000013-180914115041330-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-09-14 15:44:36,823  INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] [***0000013-180914115041330-oozie-oozi-W@:start:***]Action status=DONE
2018-09-14 15:44:36,823  INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] [***0000013-180914115041330-oozie-oozi-W@:start:***]Action updated in DB!
2018-09-14 15:44:36,888  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000013-180914115041330-oozie-oozi-W@:start:
2018-09-14 15:44:36,889  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000013-180914115041330-oozie-oozi-W
2018-09-14 15:44:36,918  INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Start action [0000013-180914115041330-oozie-oozi-W@Create_external_table] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2018-09-14 15:44:39,662  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 15:44:39,712  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]
2018-09-14 15:44:39,716  INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] [***0000013-180914115041330-oozie-oozi-W@Create_external_table***]Action status=RUNNING
2018-09-14 15:44:39,716  INFO ActionStartXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] [***0000013-180914115041330-oozie-oozi-W@Create_external_table***]Action updated in DB!
2018-09-14 15:44:39,720  INFO WorkflowNotificationXCommand:520 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] No Notification URL is defined. Therefore nothing to notify for job 0000013-180914115041330-oozie-oozi-W@Create_external_table
2018-09-14 15:55:00,272  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 15:55:00,295  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]
2018-09-14 16:06:00,600  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 16:06:00,623  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]
2018-09-14 16:17:00,299  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] Trying to get job [job_1536925639400_0005], attempt [1]
2018-09-14 16:17:00,319  INFO HiveActionExecutor:520 - SERVER[sandbox-hdp.hortonworks.com] USER[admin] GROUP[-] TOKEN[] APP[test] JOB[0000013-180914115041330-oozie-oozi-W] ACTION[0000013-180914115041330-oozie-oozi-W@Create_external_table] checking action, hadoop job ID [job_1536925639400_0005] status [RUNNING]`

从/var/log/oozie/oozie-error.log文件中:

    2018-09-14 16:12:57,397  WARN ParameterVerifier:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] The application does not define formal parameters in its XML definition
2018-09-14 16:12:57,408  WARN LiteWorkflowAppService:523 - SERVER[sandbox-hdp.hortonworks.com] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] libpath [hdfs://sandbox-hdp.hortonworks.com:8020/user/oozie/workfl
ows/workflow_otinanai/lib] does not exist                                                                                                                                                                          
2018-09-18 08:20:38,605  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]                                                           
2018-09-18 08:20:38,655  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]                                                              
2018-09-18 08:20:38,726  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]                                                          
2018-09-18 08:20:38,846  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]                                                             
2018-09-18 08:20:38,892  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]                                                                        
2018-09-18 08:20:38,942  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]                                                                                        
2018-09-18 08:20:38,943  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]                                                             
2018-09-18 08:20:39,035  WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming                                                                                                 
2018-09-18 08:20:42,131  WARN Services:523 - SERVER[] Previous services singleton active, destroying it                                                                                                            
2018-09-18 08:20:42,189  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]                                                           
2018-09-18 08:20:42,204  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]                                                              
2018-09-18 08:20:42,213  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]                                                          
2018-09-18 08:20:42,214  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]                                                             
2018-09-18 08:20:42,214  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]                                                                        
2018-09-18 08:20:42,214  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]                                                                                        
2018-09-18 08:20:42,215  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]                                                             
2018-09-18 08:20:42,216  WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming                                                                                                 
2018-09-18 08:20:45,892  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]                                                           
2018-09-18 08:20:45,895  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]                                                              
2018-09-18 08:20:45,902  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]                                                          
2018-09-18 08:20:45,902  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]                                                             
2018-09-18 08:20:45,903  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]                                                                        
2018-09-18 08:20:45,904  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]                                                                                        
2018-09-18 08:20:45,904  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]                                                             
2018-09-18 08:20:45,909  WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming                                                                                                 
2018-09-18 08:20:48,136  WARN Services:523 - SERVER[] Previous services singleton active, destroying it                                                                                                            
2018-09-18 08:20:48,359  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]                                                           
2018-09-18 08:20:48,391  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]                                                              
2018-09-18 08:20:48,445  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]                                                          
2018-09-18 08:20:48,454  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]                                                             
2018-09-18 08:20:48,469  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [hadoop.security.credential.provider.path]                                                                        
2018-09-18 08:20:48,498  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [credentialStoreClassPath]                                                                                        
2018-09-18 08:20:48,500  WARN ConfigurationService:523 - SERVER[] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]                                                             
2018-09-18 08:20:48,504  WARN Services:523 - SERVER[] System ID [oozie-oozi] exceeds maximum length [10], trimming                                                                                                 
2018-09-18 08:20:55,819  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] System property [oozie.https.keystore.pass] no defined in Oozie configuration, ignored                                
2018-09-18 08:20:55,821  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] System property [oozie.admin.port] no defined in Oozie configuration, ignored                                         
2018-09-18 08:20:55,822  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] System property [oozie.https.keystore.file] no defined in Oozie configuration, ignored                                
2018-09-18 08:20:55,828  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.hosts]                                
2018-09-18 08:20:55,831  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.hosts]                                   
2018-09-18 08:20:55,834  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.falcon.groups]                               
2018-09-18 08:20:55,835  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.AuthorizationService.security.enabled]                                  
2018-09-18 08:20:55,836  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [hadoop.security.credential.provider.path]                                             
2018-09-18 08:20:55,837  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [credentialStoreClassPath]                                                             
2018-09-18 08:20:55,838  WARN ConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Invalid configuration defined, [oozie.service.ProxyUserService.proxyuser.hue.groups]                                  
2018-09-18 08:20:55,844  WARN Services:523 - SERVER[sandbox-hdp.hortonworks.com] System ID [oozie-oozi] exceeds maximum length [10], trimming                                                                      
2018-09-18 08:20:56,334  WARN ConfigUtils:523 - SERVER[sandbox-hdp.hortonworks.com] Using a deprecated configuration property [oozie.service.AuthorizationService.security.enabled], should use [oozie.service.Auth
orizationService.authorization.enabled].  Please delete the deprecated property in order for the new property to take effect.                                                                                      
2018-09-18 08:21:07,459  WARN SparkConfigurationService:523 - SERVER[sandbox-hdp.hortonworks.com] Spark Configuration could not be loaded for *: /usr/hdp/current/oozie-server/conf/spark-conf does not exist

有帮助吗?enter image description here

0 个答案:

没有答案