我创建了StreamSets服务器的Docker容器,并且可以正常启动它。
但是,在创建第一个管道时,系统会要求我安装streamsets-datacollector-dev-lib
阶段库。
这样做并重新启动容器后,该服务在控制台中显示以下错误:
Abnormal exit: java.lang.RuntimeException: Missing services: Stage com_streamsets_pipeline_stage_devtest_rawdata_RawDataDSource is missing service com.streamsets.pipeline.api.service.dataformats.DataFormatParserServiceCheck STDERR for more details
streamsets_streamsets.1.9oqwp3hfpzox@stephane-pc |
streamsets_streamsets.1.9oqwp3hfpzox@stephane-pc | java.lang.RuntimeException: Missing services: Stage com_streamsets_pipeline_stage_devtest_rawdata_RawDataDSource is missing service com.streamsets.pipeline.api.service.dataformats.DataFormatParserService
删除streamsets-datacollector-dev-lib
文件时,容器可以正常重启。
Dockerfile
文件:
FROM streamsets/datacollector
docker-compose-dev.yml
文件:
version: "3.7"
services:
streamsets:
image: localhost:5000/streamsets:latest
ports:
- "18630:18630"
networks:
streamsets:
common:
volumes:
- "~/dev/docker/projects/streamsets/volumes/data:/data"
- "~/dev/docker/projects/streamsets/volumes/stage-libs:/opt/streamsets-datacollector-3.18.1/streamsets-libs"
- "~/dev/docker/projects/streamsets/volumes/stage-libs-extra:/opt/stage-libs-extra"
- "~/dev/docker/projects/streamsets/volumes/logs:/logs"
environment:
SDC_CONF_STREAMSETS_LIBRARIES_EXTRA_DIR: "/opt/stage-libs-extra"
HOST_USER_ID: ${CURRENT_UID}
HOST_GROUP_ID: ${CURRENT_GID}
deploy:
replicas: 1
restart_policy:
condition: any
delay: 5s
max_attempts: 3
window: 10s
healthcheck:
test: curl --fail http://localhost:18630 || exit 1
interval: 1m
timeout: 3s
retries: 3
networks:
streamsets:
name: streamsets
common:
external: true
name: common
启动命令:
docker stack deploy --compose-file docker-compose-dev.yml streamsets
体积:
stephane@stephane-pc:~/dev/docker/projects/streamsets$ tree volumes/stage-libs
volumes/stage-libs
├── streamsets-datacollector-dev-lib
│ └── lib
│ ├── automaton-1.11-8.jar
│ ├── commons-lang3-3.5.jar
│ ├── generex-1.0.2.jar
│ ├── jackson-annotations-2.8.0.jar
│ ├── jackson-core-2.8.9.jar
│ ├── jackson-databind-2.8.9.jar
│ ├── javafaker-0.13.jar
│ ├── streamsets-datacollector-dev-lib-3.18.1.jar
│ └── streamsets-datacollector-stagesupport-3.18.1.jar
└── streamsets-datacollector-mysql-binlog-lib
└── lib
├── guava-19.0.jar
├── HikariCP-3.2.0.jar
├── jackson-annotations-2.8.0.jar
├── jackson-core-2.8.8.jar
├── jackson-databind-2.8.8.jar
├── log4j-1.2.17.jar
├── mysql-binlog-connector-java-0.13.0.jar
├── slf4j-log4j12-1.7.7.jar
├── streamsets-datacollector-mysql-binlog-lib-3.18.1.jar
└── streamsets-datacollector-stagesupport-3.18.1.jar
4 directories, 19 files
stephane@stephane-pc:~/dev/docker/projects/streamsets$ tree volumes/stage-libs-extra/
volumes/stage-libs-extra/
0 directories, 0 files
更新:我注意到当没有任何卷映射时,启动容器将显示以下库:
bash-4.4$ ls -l /opt/streamsets-datacollector-3.18.1/streamsets-libs
total 20
drwxrwxr-x 3 1000 root 4096 Aug 26 18:55 streamsets-datacollector-basic-lib
drwxrwxr-x 3 1000 root 4096 Aug 26 18:55 streamsets-datacollector-dataformats-lib
drwxrwxr-x 3 1000 root 4096 Aug 26 18:55 streamsets-datacollector-dev-lib
drwxrwxr-x 3 1000 root 4096 Aug 26 18:55 streamsets-datacollector-stats-lib
drwxrwxr-x 3 1000 root 4096 Aug 26 18:55 streamsets-datacollector-windows-lib
但是具有卷映射时:
- "~/dev/docker/projects/streamsets/volumes/stage-libs:/opt/streamsets-datacollector-3.18.1/streamsets-libs"
- "~/dev/docker/projects/streamsets/volumes/stage-libs-extra:/opt/stage-libs-extra"
主机目录中没有这些库。
更新:无法解决这些权限问题,我求助于创建Docker卷而不是使用主机卷。
docker volume create --name sdc-data;
docker volume create --name sdc-stagelibs;
在docker-compose-dev.yml
文件中,我现在有:
volumes:
- "sdc-data:/data"
- "sdc-stagelibs:/opt/streamsets-datacollector-3.18.1/streamsets-libs"
现在可以安装其他舞台库了。