我正在尝试在单个节点Hadoop框(Hortonworks Sandbox)上安装Streamsets。 Streamsets website
上的安装过程非常简单下载核心tar文件,解压缩然后运行 “streamsets-datacollector-3.1.2.0 / bin / streamsets dc”在端口18630上启动DataCollector。
不知何故,此端口永远不会打开,因此连接失败。我使用netcat来验证此端口是否保持关闭状态。我已阅读日志,它确实显示“StandaloneAndClusterPipelineManager - 停止生产管道管理器”,但我不确定这是一个问题或如何解决它。
请帮帮我。
由于
阿迪尔
命令行输出:
[root@sandbox-hdp streams]# streamsets-datacollector-3.1.2.0/bin/streamsets dc
Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS
Logging initialized @2385ms to org.eclipse.jetty.util.log.Slf4jLog
Running on URI : 'http://sandbox-hdp.hortonworks.com:18630'
日志文件
2018-03-21 11:03:20,046 [user:] [pipeline:] [runner:] [thread:main] INFO Main - -----------------------------------------------------------------
2018-03-21 11:03:20,047 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Build info:
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Version : 3.1.2.0
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Date : 2018-03-20T18:19Z
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Built by : root
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Repo SHA : 2f3225224e5ce2beff355254e44ffa9e3c48863a
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Source MD5 : 1aa7555e88e9e601b82336af48d3f97
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - -----------------------------------------------------------------
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Runtime info:
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Java version : 1.8.0_161-b14
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - SDC ID : 76d2066d-2cf7-11e8-adc1-4d8940914dad
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Runtime dir : /streams/streamsets-datacollector-3.1.2.0
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Config dir : /streams/streamsets-datacollector-3.1.2.0/etc
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Data dir : /streams/streamsets-datacollector-3.1.2.0/data
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Log dir : /streams/streamsets-datacollector-3.1.2.0/log
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - -----------------------------------------------------------------
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Security Manager : ENABLED, policy file: file:///streams/streamsets-datacollector-3.1.2.0/etc/sdc-security.policy
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - -----------------------------------------------------------------
2018-03-21 11:03:20,048 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Starting ...
2018-03-21 11:03:20,066 [user:] [pipeline:] [runner:] [thread:main] INFO Main - -----------------------------------------------------------------
2018-03-21 11:03:20,066 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Kerberos enabled: false
2018-03-21 11:03:20,073 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Unlimited cryptography enabled: true
2018-03-21 11:03:20,073 [user:] [pipeline:] [runner:] [thread:main] INFO Main - -----------------------------------------------------------------
2018-03-21 11:03:20,073 [user:] [pipeline:] [runner:] [thread:main] INFO Main - Starting ...
2018-03-21 11:03:20,136 [user:] [pipeline:] [runner:] [thread:main] INFO ClassLoaderStageLibraryTask - Validating classpath of all stages
2018-03-21 11:03:20,230 [user:] [pipeline:] [runner:] [thread:main] INFO ClassLoaderStageLibraryTask - Classpath of all stages passed validation
2018-03-21 11:03:22,884 [user:] [pipeline:] [runner:] [thread:main] INFO LineagePublisherTaskImpl - No publishers configured
2018-03-21 11:03:43,573 [user:] [pipeline:] [runner:] [thread:main] INFO Reflections - Reflections took 125 ms to scan 2 urls, producing 1235 keys and 2032 values
2018-03-21 11:03:44,594 [user:] [pipeline:] [runner:] [thread:main] INFO Reflections - Reflections took 51 ms to scan 2 urls, producing 1235 keys and 2032 values
2018-03-21 11:03:45,190 [user:] [pipeline:] [runner:] [thread:main] INFO WebServerTask - Running on URI : 'http://sandbox-hdp.hortonworks.com:18630'
2018-03-21 13:42:41,888 [user:] [pipeline:] [runner:] [thread:Main.shutdownHook] INFO StandaloneAndClusterPipelineManager - Stopped Production Pipeline Manager
OpenJDK 64-Bit Server VM (25.161-b14) for linux-amd64 JRE (1.8.0_161-b14), built on Jan 18 2018 10:55:10 by "mockbuild" with gcc 4.4.7 20120313 (Red Hat 4.4.7-18)
Memory: 4k page, physical 12205060k(386800k free), swap 5119996k(4399392k free)
CommandLine flags: -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/streams/streamsets-datacollector-3.1.2.0/log/sdc_heapdump_1521630198.hprof -XX:InitialHeapSize=1073741824 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=348966912 -XX:MaxTenuringThreshold=6 -XX:NewSize=348966912 -XX:OldPLABSize=16 -XX:OldSize=697933824 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
2018-03-21T11:03:21.162+0000: 2.295: [GC (Allocation Failure) 2018-03-21T11:03:21.162+0000: 2.295: [ParNew: 272640K->18771K(306688K), 0.0477338 secs] 272640K->18771K(1014528K), 0.0478240 secs] [Times: user=0.12 sys=0.05, real=0.05 secs]
2018-03-21T11:03:21.710+0000: 2.843: [GC (Allocation Failure) 2018-03-21T11:03:21.710+0000: 2.843: [ParNew: 291411K->9579K(306688K), 0.0523179 secs] 291411K->19386K(1014528K), 0.0523734 secs] [Times: user=0.08 sys=0.03, real=0.05 secs]
2018-03-21T11:03:21.773+0000: 2.906: [GC (CMS Initial Mark) [1 CMS-initial-mark: 9806K(707840K)] 30182K(1014528K), 0.0069336 secs] [Times: user=0.01 sys=0.01, real=0.01 secs]
2018-03-21T11:03:21.780+0000: 2.913: [CMS-concurrent-mark-start]
2018-03-21T11:03:21.796+0000: 2.929: [CMS-concurrent-mark: 0.015/0.015 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-03-21T11:03:21.796+0000: 2.929: [CMS-concurrent-preclean-start]
2018-03-21T11:03:21.797+0000: 2.930: [CMS-concurrent-preclean: 0.001/0.001 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-03-21T11:03:21.797+0000: 2.930: [CMS-concurrent-abortable-preclean-start]
2018-03-21T11:03:22.088+0000: 3.221: [GC (Allocation Failure) 2018-03-21T11:03:22.088+0000: 3.221: [ParNew: 282219K->8640K(306688K), 0.0050788 secs] 292026K->18447K(1014528K), 0.0051379 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-03-21T11:03:22.093+0000: 3.226: [CMS-concurrent-abortable-preclean: 0.123/0.296 secs] [Times: user=1.01 sys=0.02, real=0.30 secs]
2018-03-21T11:03:22.094+0000: 3.227: [GC (CMS Final Remark) [YG occupancy: 14026 K (306688 K)]2018-03-21T11:03:22.094+0000: 3.227: [Rescan (parallel) , 0.0027376 secs]2018-03-21T11:03:22.097+0000: 3.230: [weak refs processing, 0.0000588 secs]2018-03-21T11:03:22.097+0000: 3.230: [class unloading, 0.0030661 secs]2018-03-21T11:03:22.100+0000: 3.233: [scrub symbol table, 0.0014896 secs]2018-03-21T11:03:22.102+0000: 3.235: [scrub string table, 0.0004684 secs][1 CMS-remark: 9806K(707840K)] 23832K(1014528K), 0.0086906 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-03-21T11:03:22.103+0000: 3.236: [CMS-concurrent-sweep-start]
2018-03-21T11:03:22.106+0000: 3.239: [CMS-concurrent-sweep: 0.003/0.003 secs] [Times: user=0.00 sys=0.00, real=0.00 secs]
2018-03-21T11:03:22.106+0000: 3.239: [CMS-concurrent-reset-start]
2018-03-21T11:03:22.150+0000: 3.283: [CMS-concurrent-reset: 0.044/0.044 secs] [Times: user=0.13 sys=0.05, real=0.05 secs]
2018-03-21T11:03:22.380+0000: 3.513: [GC (Allocation Failure) 2018-03-21T11:03:22.380+0000: 3.513: [ParNew: 281280K->9977K(306688K), 0.0092280 secs] 291085K->19782K(1014528K), 0.0093244 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
2018-03-21T11:03:22.723+0000: 3.856: [GC (Allocation Failure) 2018-03-21T11:03:22.723+0000: 3.856: [ParNew: 282617K->12670K(306688K), 0.0153239 secs] 292422K->22475K(1014528K), 0.0153801 secs] [Times: user=0.05 sys=0.00, real=0.01 secs]
2018-03-21T11:03:44.232+0000: 25.365: [GC (Allocation Failure) 2018-03-21T11:03:44.232+0000: 25.365: [ParNew: 285310K->34048K(306688K), 0.0843839 secs] 295115K->44018K(1014528K), 0.0844584 secs] [Times: user=0.13 sys=0.07, real=0.08 secs]
Heap
par new generation total 306688K, used 243737K [0x00000000c0000000, 0x00000000d4cc0000, 0x00000000d4cc0000)
eden space 272640K, 76% used [0x00000000c0000000, 0x00000000cccc66b0, 0x00000000d0a40000)
from space 34048K, 100% used [0x00000000d0a40000, 0x00000000d2b80000, 0x00000000d2b80000)
to space 34048K, 0% used [0x00000000d2b80000, 0x00000000d2b80000, 0x00000000d4cc0000)
concurrent mark-sweep generation total 707840K, used 9970K [0x00000000d4cc0000, 0x0000000100000000, 0x0000000100000000)
Metaspace used 34235K, capacity 35140K, committed 35400K, reserved 1081344K
class space used 4175K, capacity 4363K, committed 4424K, reserved 1048576K