我目前在pio训练和火花方面遇到问题。
要单独运行火花,我必须运行下一个命令spark-submit --driver-class-path spark-timeseries/python/sparkts/sparkts-0.4.0-SNAPSHOT-jar-with-dependencies.jar train.py
对于集成,我尝试做pio train -- --driver-class-path spark-timeseries/python/sparkts/sparkts-0.4.0-SNAPSHOT-jar-with-dependencies.jar train.py
(单个--
用于解析为spark-submit的单独参数),但是出现以下错误ImportError: No module named pypio.utils
是pio使用的模块。
我尝试过的另一种选择是pio train -- --driver-class-path /PredictionIO-0.12.1/conf:/PredictionIO-0.12.1/vendors/hbase-1.2.6/conf:spark-timeseries/python/sparkts/sparkts-0.4.0-SNAPSHOT-jar-with-dependencies.jar train.py
,但是我遇到了同样的错误。
我认为我手动给出的--driver-class-path
参数掩盖了pio的某些依赖性。
有人有主意吗?
PS。仅运行pio train时,这是完整的提交命令:
[INFO] [Runner$] Submission command: /PredictionIO-0.12.1/vendors/spark-2.1.1-bin-hadoop2.6/bin/spark-submit --class org.apache.predictionio.workflow.CreateWorkflow --jars file:/engines/arima_model/target/scala-2.11/poisson-regression-assembly-0.1-SNAPSHOT-deps.jar,file:/engines/arima_model/target/scala-2.11/template-iris-assembly-0.1-SNAPSHOT-deps.jar,file:/engines/arima_model/target/scala-2.11/poisson-regression_2.11-0.1-SNAPSHOT.jar,file:/engines/arima_model/target/scala-2.11/template-iris_2.11-0.1-SNAPSHOT.jar,file:/PredictionIO-0.12.1/lib/spark/pio-data-jdbc-assembly-0.12.1.jar,file:/PredictionIO-0.12.1/lib/spark/pio-data-hbase-assembly-0.12.1.jar,file:/PredictionIO-0.12.1/lib/spark/pio-data-localfs-assembly-0.12.1.jar,file:/PredictionIO-0.12.1/lib/spark/pio-data-s3-assembly-0.12.1.jar,file:/PredictionIO-0.12.1/lib/spark/pio-data-hdfs-assembly-0.12.1.jar,file:/PredictionIO-0.12.1/lib/spark/pio-data-elasticsearch-assembly-0.12.1.jar --files file:/PredictionIO-0.12.1/conf/log4j.properties,file:/PredictionIO-0.12.1/vendors/hbase-1.2.6/conf/hbase-site.xml --driver-class-path /PredictionIO-0.12.1/conf:/PredictionIO-0.12.1/vendors/hbase-1.2.6/conf --driver-java-options -Dpio.log.dir=/home/pio file:/PredictionIO-0.12.1/lib/pio-assembly-0.12.1.jar --engine-id org.jpioug.template.python.Engine --engine-version 21ee0e10f5dc458111a5a19e3963e06f196a707f --engine-variant file:/engines/arima_model/engine.json --verbosity 0 --json-extractor Both --env PIO_STORAGE_SOURCES_HBASE_TYPE=hbase,PIO_ENV_LOADED=1,PIO_STORAGE_REPOSITORIES_METADATA_NAME=pio_meta,PIO_VERSION=0.12.1,PIO_FS_BASEDIR=/home/pio/.pio_store,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOSTS=localhost,PIO_STORAGE_SOURCES_HBASE_HOME=/PredictionIO-0.12.1/vendors/hbase-1.2.6,PIO_HOME=/PredictionIO-0.12.1,PIO_FS_ENGINESDIR=/home/pio/.pio_store/engines,PIO_STORAGE_SOURCES_LOCALFS_PATH=/home/pio/.pio_store/models,PIO_STORAGE_SOURCES_ELASTICSEARCH_TYPE=elasticsearch,PIO_STORAGE_REPOSITORIES_METADATA_SOURCE=ELASTICSEARCH,PIO_STORAGE_REPOSITORIES_MODELDATA_SOURCE=LOCALFS,PIO_STORAGE_REPOSITORIES_EVENTDATA_NAME=pio_event,PIO_STORAGE_SOURCES_ELASTICSEARCH_CLUSTERNAME=predictionio,PIO_STORAGE_SOURCES_ELASTICSEARCH_HOME=/PredictionIO-0.12.1/vendors/elasticsearch-5.5.2,PIO_FS_TMPDIR=/home/pio/.pio_store/tmp,PIO_STORAGE_REPOSITORIES_MODELDATA_NAME=pio_model,PIO_STORAGE_REPOSITORIES_EVENTDATA_SOURCE=HBASE,PIO_CONF_DIR=/PredictionIO-0.12.1/conf,PIO_STORAGE_SOURCES_ELASTICSEARCH_PORTS=9200,PIO_STORAGE_SOURCES_LOCALFS_TYPE=localfs