火花驱动程序已断开连接并自动连接到另一个端口。如何停止断开连接

时间:2018-09-07 09:28:34

标签: apache-spark apache-kafka spark-submit

火花流媒体应用程序是由两个人组成的独立集群提交的。在我的应用程序中,从kafka主题读取数据并处理一些计算并发送到结果主题。结果主题中出现重复的记录。在控制台中看到以下异常。设置为s3,甚至将auto .offset.reset设置为最新的“ kafkaParams.put(” auto.offset.reset“,” latest“);”那为什么我在结果主题中得到重复的记录。

2018-08-28 16:32:36 INFO  JobScheduler:54 - Finished job streaming job 1535454155000 ms.0 from job set of time 1535454155000 ms
    2018-08-28 16:32:36 INFO  JobScheduler:54 - Total delay: 1.870 s for time 1535454155000 ms (execution: 0.201 s)
    2018-08-28 16:32:42 INFO  CheckpointWriter:54 - Checkpoint for time 1535454142000 ms saved to file 's3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000', took 5590 bytes and 8507 ms
    2018-08-28 16:32:42 INFO  CheckpointWriter:54 - Saving checkpoint for time 1535454143000 ms to file 's3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000'
    2018-08-28 16:32:43 INFO  NativeS3FileSystem:237 - OutputStream for key 'cffccheck/cffc2/temp' writing to tempfile '\tmp\hadoop-DELL\s3\output-7698969883279329562.tmp'
    2018-08-28 16:32:43 INFO  NativeS3FileSystem:271 - OutputStream for key 'cffccheck/cffc2/temp' closed. Now beginning upload
    2018-08-28 16:32:44 INFO  NativeS3FileSystem:283 - OutputStream for key 'cffccheck/cffc2/temp' upload complete
    2018-08-28 16:32:45 INFO  CheckpointWriter:54 - CheckpointWriter executor terminated? false, waited for 10008 ms.
    2018-08-28 16:32:45 INFO  JobGenerator:54 - Stopped JobGenerator
    2018-08-28 16:32:45 INFO  JobScheduler:54 - Stopped JobScheduler
    2018-08-28 16:32:45 INFO  ContextHandler:910 - Stopped o.s.j.s.ServletContextHandler@120d3fd{/streaming,null,UNAVAILABLE,@Spark}
    2018-08-28 16:32:45 INFO  ContextHandler:910 - Stopped o.s.j.s.ServletContextHandler@410ae5ac{/streaming/batch,null,UNAVAILABLE,@Spark}
    2018-08-28 16:32:45 INFO  ContextHandler:910 - Stopped o.s.j.s.ServletContextHandler@4fd05028{/static/streaming,null,UNAVAILABLE,@Spark}
    2018-08-28 16:32:45 INFO  StreamingContext:54 - StreamingContext stopped successfully
    2018-08-28 16:32:45 INFO  AbstractConnector:318 - Stopped Spark@562c877a{HTTP/1.1,[http/1.1]}{0.0.0.0:4040}
    2018-08-28 16:32:45 INFO  SparkUI:54 - Stopped Spark web UI at http://DELL-PC:4040
    2018-08-28 16:32:45 INFO  StandaloneSchedulerBackend:54 - Shutting down all executors
    2018-08-28 16:32:45 INFO  CoarseGrainedSchedulerBackend$DriverEndpoint:54 - Asking each executor to shut down
    2018-08-28 16:32:45 INFO  MapOutputTrackerMasterEndpoint:54 - MapOutputTrackerMasterEndpoint stopped!
    2018-08-28 16:32:45 WARN  TransportChannelHandler:78 - Exception in connection from /192.168.1.179:58235
    java.io.IOException: An existing connection was forcibly closed by the remote host
        at sun.nio.ch.SocketDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)
    2018-08-28 16:32:45 WARN  TransportChannelHandler:78 - Exception in connection from /192.168.1.179:58251
    java.io.IOException: An existing connection was forcibly closed by the remote host
        at sun.nio.ch.SocketDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:43)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
        at sun.nio.ch.IOUtil.read(IOUtil.java:192)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
        at io.netty.buffer.PooledUnsafeDirectByteBuf.setBytes(PooledUnsafeDirectByteBuf.java:288)
        at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1106)
        at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:343)
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:123)
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
        at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)
        at java.lang.Thread.run(Thread.java:748)
    2018-08-28 16:32:45 INFO  MemoryStore:54 - MemoryStore cleared
    2018-08-28 16:32:45 INFO  BlockManager:54 - BlockManager stopped
    2018-08-28 16:32:45 INFO  BlockManagerMaster:54 - BlockManagerMaster stopped
    2018-08-28 16:32:45 INFO  OutputCommitCoordinator$OutputCommitCoordinatorEndpoint:54 - OutputCommitCoordinator stopped!
    2018-08-28 16:32:45 INFO  SparkContext:54 - Successfully stopped SparkContext
    2018-08-28 16:32:50 INFO  CheckpointWriter:54 - Checkpoint for time 1535454143000 ms saved to file 's3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000', took 5590 bytes and 8615 ms
    2018-08-28 16:32:50 INFO  ShutdownHookManager:54 - Shutdown hook called
    2018-08-28 16:32:50 INFO  ShutdownHookManager:54 - Deleting directory E:\temp8094\localdir\spark-aa95d0d5-6b7a-4d5b-ae51-a832fd2b4074
    2018-08-28 16:32:50 INFO  ShutdownHookManager:54 - Deleting directory C:\Users\DELL\AppData\Local\Temp\spark-28b3a422-0a53-4049-9fb0-2425af002354
    2018-08-28 16:32:53 INFO  SecurityManager:54 - Changing view acls to: DELL
    2018-08-28 16:32:53 INFO  SecurityManager:54 - Changing modify acls to: DELL
    2018-08-28 16:32:53 INFO  SecurityManager:54 - Changing view acls groups to: 
    2018-08-28 16:32:53 INFO  SecurityManager:54 - Changing modify acls groups to: 
    2018-08-28 16:32:53 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(DELL); groups with view permissions: Set(); users  with modify permissions: Set(DELL); groups with modify permissions: Set()
    2018-08-28 16:32:53 INFO  Utils:54 - Successfully started service 'Driver' on port 58363.
    2018-08-28 16:32:53 INFO  DriverWrapper:54 - Driver address: 192.168.1.179:58363
    2018-08-28 16:32:53 INFO  WorkerWatcher:54 - Connecting to worker spark://Worker@192.168.1.179:53448
2018-08-28 16:32:53 INFO  SecurityManager:54 - Changing view acls to: DELL
2018-08-28 16:32:53 INFO  SecurityManager:54 - Changing modify acls to: DELL
2018-08-28 16:32:53 INFO  SecurityManager:54 - Changing view acls groups to: 
2018-08-28 16:32:53 INFO  SecurityManager:54 - Changing modify acls groups to: 
2018-08-28 16:32:53 INFO  SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(DELL); groups with view permissions: Set(); users  with modify permissions: Set(DELL); groups with modify permissions: Set()
2018-08-28 16:32:53 INFO  WorkerWatcher:54 - Successfully connected to spark://Worker@192.168.1.179:53448
2018-08-28 16:32:53 INFO  TransportClientFactory:267 - Successfully created connection to /192.168.1.179:53448 after 129 ms (0 ms spent in bootstraps)
2018-08-28 16:32:56 INFO  CheckpointReader:54 - Checkpoint files found: s3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000,s3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000.bk,s3://cffcs5/cffccheck/cffc2/checkpoint-1535454142000
2018-08-28 16:32:56 INFO  CheckpointReader:54 - Attempting to load checkpoint from file s3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000
2018-08-28 16:32:56 INFO  NativeS3FileSystem:615 - Opening 's3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000' for reading
2018-08-28 16:32:57 INFO  Checkpoint:54 - Checkpoint for time 1535454143000 ms validated
2018-08-28 16:32:57 INFO  CheckpointReader:54 - Checkpoint successfully loaded from file s3://cffcs5/cffccheck/cffc2/checkpoint-1535454143000
2018-08-28 16:32:57 INFO  CheckpointReader:54 - Checkpoint was generated at time 1535454143000 ms
2018-08-28 16:32:57 INFO  SparkContext:54 - Running Spark version 2.3.0

0 个答案:

没有答案