Pentaho数据集成如何在carte集群上运行厨房工作？

时间：2017-11-28 12:17:54

标签： cluster-computing pentaho kettle pentaho-spoon

我已经建立了一个菜单集群（1个主人和2个奴隶）并用勺子在菜单集群上运行工作。但是当我使用kitchen命令或carte http访问时，它以独立方式运行（只在主节点中运行）。

我是否错过了配置中的任何内容？或者它不支持群集模式？

以下是我的尝试：

  1. 我的配置：

    2. enter image description here

    enter image description here

    1. 用“Enviroment Type - Local”用勺子跑了

      主输出：

      2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml
Tue Nov 28 04:47:09 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must   be       established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to   explicitly disable SSL       by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2017/11/28 04:47:10 - sortcluster111 (master) - Dispatching started for transformation [sortcluster111 (master)]
Tue Nov 28 04:47:10 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must   be       established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to   explicitly disable SSL       by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Tue Nov 28 04:47:10 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must   be       established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to   explicitly disable SSL       by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2017/11/28 04:47:10 - output.0 - Connected to database [102] (commit=1000)
2017/11/28 04:47:10 - input.0 - Finished reading query, closing connection.
2017/11/28 04:47:10 - input.0 - Finished processing (I=47, O=0, R=0, W=47, U=0, E=0)
2017/11/28 04:47:10 - input.0 - Server socket accepted for port [40001], reading from server Dynamic slave [kettleslave02:8083]
2017/11/28 04:47:10 - input.0 - Server socket accepted for port [40000], reading from server Dynamic slave [kettleslave01:8082]
2017/11/28 04:47:10 - output.0 - Finished processing (I=47, O=47, R=0, W=47, U=0, E=0)

      slave01输出：

      2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml
Tue Nov 28 04:47:09 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must  be        established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to  explicitly disable SSL        by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2017/11/28 04:47:10 - sortcluster111 (cluster1:Dynamic slave [kettleslave01:8082]) - Dispatching started for transformation [sortcluster111 (cluster1:Dynamic slave [kettleslave01:8082])]
2017/11/28 04:47:10 - sort.0 - Server socket accepted for port [40000], reading from server kettlemaster01
2017/11/28 04:47:10 - sort.0 - Finished processing (I=24, O=0, R=0, W=24, U=0, E=0)

      slave02输出：

      2017/11/28 04:47:09 - RepositoriesMeta - Reading repositories XML file: /root/.kettle/repositories.xml
2017/11/28 04:47:09 - General - Unable to connect to the repository with name 'Mysqlrep'
2017/11/28 04:47:10 - sortcluster111 (cluster1:Dynamic slave [kettleslave02:8083]) - Dispatching started for transformation [sortcluster111 (cluster1:Dynamic slave [kettleslave02:8083])]
2017/11/28 04:47:10 - sort.0 - Server socket accepted for port [40000], reading from server kettlemaster01
2017/11/28 04:47:10 - sort.0 - Finished processing (I=23, O=0, R=0, W=23, U=0, E=0)

    2. 和厨房一起跑：

      kitchen.sh -rep=Mysqlrep -user=admin -pass=admin -job trans1

      主输出：

      2017/11/28 04:10:19 - trans1 - Starting entry [sorttrans]
2017/11/28 04:10:19 - sorttrans - Loading transformation from repository [sortcluster111] in directory [/]
2017/11/28 04:10:19 - sorttrans - Using run configuration [cluster config]
2017/11/28 04:10:19 - sorttrans - Using legacy execution engine
2017/11/28 04:10:19 - sortcluster111 - Dispatching started for transformation [sortcluster111]
Tue Nov 28 04:10:19 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be         established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL         by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Tue Nov 28 04:10:19 EST 2017 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be         established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL         by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2017/11/28 04:10:19 - output.0 - Connected to database [102] (commit=1000)
2017/11/28 04:10:19 - input.0 - Finished reading query, closing connection.
2017/11/28 04:10:19 - input.0 - Finished processing (I=47, O=0, R=0, W=47, U=0, E=0)
2017/11/28 04:10:19 - sort.0 - Finished processing (I=0, O=0, R=47, W=47, U=0, E=0)
2017/11/28 04:10:19 - output.0 - Finished processing (I=0, O=47, R=47, W=47, U=0, E=0)
2017/11/28 04:10:19 - trans1 - Starting entry [finish]
2017/11/28 04:10:19 - trans1 - Finished job entry [finish] (result=[true])
2017/11/28 04:10:19 - trans1 - Finished job entry [sorttrans] (result=[true])
2017/11/28 04:10:19 - trans1 - Finished job entry [SQL] (result=[true])
2017/11/28 04:10:19 - trans1 - Job execution finished
2017/11/28 04:10:19 - Kitchen - Finished!
2017/11/28 04:10:19 - Kitchen - Start=2017/11/28 04:10:00.586, Stop=2017/11/28 04:10:19.739
2017/11/28 04:10:19 - Kitchen - Processing ended after 19 seconds.

      在奴隶中没有输出

      3. 此致

      约翰

2 个答案:

答案 0 :(得分：0)

首先，在远程服务器上启动carte：

  • ./carte.sh hostname port
  • 检查您的浏览器：hostname:port/kettle/status/以空状态页面回复

其次，用勺子定义菜单：

    勺子视图中的
  • （左侧面板）：点击Slave serve r，然后点击new
  • 服务器名称= carte
  • hostname =先前定义的主机名
  • port =先前定义的端口
  • 网络应用名称= 空白（重要！）
  • username = cluster
  • 密码=群集 是主人= 是

第三，配置勺子运行：

    勺子视图中的
  • （左侧面板）：点击Run configuration，然后点击new
  • 名称：carte / servername
  • 引擎：Pentaho（默认）
  • 从属服务器：是
  • 位置：carte（第二步中定义的服务器名称）
  • 将资源发送到此服务器：是（除非您read the doc）

Forth，运行transformaton / job

  • “运行配置”下拉框允许您选择Pentaho local或carte/hostname或第三步中定义的任何配置。
  • 如果您禁用了Run option面板，则下拉运行工具或顶级菜单/操作/运行选项或F8。并查看Always show dialog on run，以后您将需要它。

第五

    浏览器中的
  • ：hostname:port/kettle/status/
  • 玩得开心。

答案 1 :(得分：0)

pdi的新版本中存在一个错误： 选项“以集群模式运行此转换吗？”不存在，因此要解决此问题，您需要打开作业xml文件并删除要在群集模式下运行的转换条目的 run_configuration 属性，并将 cluster 设置为Y。

相关问题
最新问题