Question

我想在同时安装Jupyter和DataLab的DataProc中创建一个集群（我知道它们非常相似，但是团队成员的偏好不同）。我可以使用其中任何一个创建集群：

具有Jupyter的集群：

gcloud dataproc clusters create $DATAPROC_CLUSTER_NAME_JUPYTER \
--project $PROJECT \
--bucket $BUCKET \
--zone $ZONE \
--initialization-actions gs://dataproc-initialization-actions/connectors/connectors.sh,gs://dataproc-initialization-actions/jupyter/jupyter.sh \
--metadata gcs-connector-version=$GCS_CONNECTOR_VERSION \
--metadata bigquery-connector-version=$BQ_CONNECTOR_VERSION \
--metadata JUPYTER_PORT=$JUPYTER_PORT,JUPYTER_CONDA_PACKAGES=numpy:scipy:pandas:scikit-learn

具有DataLab的集群：

gcloud dataproc clusters create $DATAPROC_CLUSTER_NAME_DATALAB \
--project $PROJECT \
--bucket $BUCKET \
--zone $ZONE \
--master-boot-disk-size $MASTER_DISK_SIZE \
--worker-boot-disk-size $WORKER_DISK_SIZE \
--initialization-actions gs://dataproc-initialization-actions/connectors/connectors.sh,gs://dataproc-initialization-actions/datalab/datalab.sh \
--metadata gcs-connector-version=$GCS_CONNECTOR_VERSION \
--metadata bigquery-connector-version=$BQ_CONNECTOR_VERSION \
--scopes cloud-platform,bigquery

两者都很好。但是，当我尝试同时使用它们创建集群时，它会失败：

gcloud dataproc clusters create test \
--project $PROJECT \
--bucket $BUCKET \
--zone $ZONE \
--initialization-actions gs://dataproc-initialization-actions/connectors/connectors.sh,gs://dataproc-initialization-actions/datalab/datalab.sh,gs://dataproc-initialization-actions/jupyter/jupyter.sh \
--metadata gcs-connector-version=$GCS_CONNECTOR_VERSION \
--metadata bigquery-connector-version=$BQ_CONNECTOR_VERSION \
--metadata JUPYTER_PORT=$JUPYTER_PORT,JUPYTER_CONDA_PACKAGES=numpy:scipy:pandas:scikit-learn \
--scopes cloud-platform,bigquery

错误消息是：

ERROR: (gcloud.dataproc.clusters.create) Operation [projects/abc/regions/global/operations/d34943dc-5bda-386f-af91-db6e0516e2c5] failed: Multiple Errors:
 - Initialization action failed. Failed action 'gs://dataproc-initialization-actions/jupyter/jupyter.sh', see output in: gs://abc/google-cloud-dataproc-metainfo/266175ef-e595-4732-b351-335837a3f30e/test-m/dataproc-initialization-script-2_output
 - Initialization action failed. Failed action 'gs://dataproc-initialization-actions/jupyter/jupyter.sh', see output in: gs://abc/google-cloud-dataproc-metainfo/266175ef-e595-4732-b351-335837a3f30e/test-w-0/dataproc-initialization-script-2_output
 - Initialization action failed. Failed action 'gs://dataproc-initialization-actions/jupyter/jupyter.sh', see output in: gs://abc/google-cloud-dataproc-metainfo/266175ef-e595-4732-b351-335837a3f30e/test-w-1/dataproc-initialization-script-2_output.

test-m中的文件如下：

++ /usr/share/google/get_metadata_value attributes/dataproc-role
+ readonly ROLE=Worker
+ ROLE=Worker
++ /usr/share/google/get_metadata_value attributes/INIT_ACTIONS_REPO
++ echo https://github.com/GoogleCloudPlatform/dataproc-initialization-actions.git
+ readonly INIT_ACTIONS_REPO=https://github.com/GoogleCloudPlatform/dataproc-initialization-actions.git
+ INIT_ACTIONS_REPO=https://github.com/GoogleCloudPlatform/dataproc-initialization-actions.git
++ /usr/share/google/get_metadata_value attributes/INIT_ACTIONS_BRANCH
++ echo master
+ readonly INIT_ACTIONS_BRANCH=master
+ INIT_ACTIONS_BRANCH=master
++ /usr/share/google/get_metadata_value attributes/JUPYTER_CONDA_CHANNELS
+ readonly JUPYTER_CONDA_CHANNELS=
+ JUPYTER_CONDA_CHANNELS=
++ /usr/share/google/get_metadata_value attributes/JUPYTER_CONDA_PACKAGES
+ readonly JUPYTER_CONDA_PACKAGES=numpy:scipy:pandas:scikit-learn
+ JUPYTER_CONDA_PACKAGES=numpy:scipy:pandas:scikit-learn
+ echo 'Cloning fresh dataproc-initialization-actions from repo https://github.com/GoogleCloudPlatform/dataproc-initialization-actions.git and branch master...'
Cloning fresh dataproc-initialization-actions from repo https://github.com/GoogleCloudPlatform/dataproc-initialization-actions.git and branch master...
+ git clone -b master --single-branch https://github.com/GoogleCloudPlatform/dataproc-initialization-actions.git
fatal: destination path 'dataproc-initialization-actions' already exists and is not an empty directory.

好像有一个克隆步骤会阻止安装成功。我该如何解决？任何建议表示赞赏，谢谢。

Answer 1

这似乎是init操作中的错误，导致我们无法两次git clone存储库。我们将解决此问题。

同时，您可以尝试使用datalab初始化操作来Jupyter optional component。

为什么不能同时安装Jupyter和DataLab来创建Google DataProc集群？

1 个答案: