Flume:来自下游的意外异常。 java.io.IOException:由peer重置连接

时间:2013-09-16 12:11:07

标签: java hadoop flume

当我尝试将多个日志发送到单个端口时,我遇到了异常。有没有人知道我的配置是否有问题,或者我是否需要将其作为一个bug提出来。我试图配置多个端口,但仍然有相同的异常。任何帮助都会非常棒,因为我已经坚持了一周。

[WARN -   org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught(NettyServer.java:201)    ] Unexpected exception from downstream.
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
at sun.nio.ch.IOUtil.read(IOUtil.java:193)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:359)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:66)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)

收集器配置

hdfs-agent.sources = avro-collect<br>
hdfs-agent.sinks = hdfs-write<br>
hdfs-agent.channels = fileChannel<br>


hdfs-agent.sources.avro-collect.type = avro<br>
hdfs-agent.sources.avro-collect.bind = <<System IP>><br>
hdfs-agent.sources.avro-collect.port = 41414<br>
hdfs-agent.sources.avro-collect.channels = fileChannel<br>

hdfs-agent.sinks.hdfs-write.type = hdfs<br>
hdfs-agent.sinks.hdfs-write.hdfs.path = hdfs://hadoop:54310/flume/%{host}/%Y%m%d/%{logFileType}<br>
hdfs-agent.sinks.hdfs-write.hdfs.rollSize = 209715200<br>
hdfs-agent.sinks.hdfs-write.hdfs.rollCount = 6000<br>
hdfs-agent.sinks.hdfs-write.hdfs.fileType = DataStream<br>
hdfs-agent.sinks.hdfs-write.hdfs.writeFormat = Text<br>
hdfs-agent.sinks.hdfs-write.hdfs.filePrefix = %{host}<br>
hdfs-agent.sinks.hdfs-write.hdfs.maxOpenFiles = 100000<br>
hdfs-agent.sinks.hdfs-write.hdfs.batchSize = 5000<br>
hdfs-agent.sinks.hdfs-write.hdfs.rollInterval = 75<br>
hdfs-agent.sinks.hdfs-write.hdfs.callTimeout = 5000000<br>
hdfs-agent.sinks.hdfs-write.channel = fileChannel<br>


hdfs-agent.channels.fileChannel.type=file<br>
hdfs-agent.channels.fileChannel.dataDirs=/u01/Collector/flume_channel/dataDir13<br>
hdfs-agent.channels.fileChannel.checkpointDir=/u01/Collector/flume_channel/checkpointDir13<br>
hdfs-agent.channels.fileChannel.transactionCapacity = 50000<br>
hdfs-agent.channels.fileChannel.capacity = 9000000<br>
hdfs-agent.channels.fileChannel.write-timeout = 250000<br>

发件人配置

app-agent.sources = tail tailapache<br>
app-agent.channels = fileChannel<br>
app-agent.sinks = avro-forward-sink avro-forward-sink-apache<br>

app-agent.sources.tail.type = exec<br>
app-agent.sources.tail.command = tail -f /server/default/log/server.log<br>
app-agent.sources.tail.channels = fileChannel<br>

app-agent.sources.tailapache.type = exec<br>
app-agent.sources.tailapache.command = tail -f /logs/access_log<br>
app-agent.sources.tailapache.channels = fileChannel<br>

app-agent.sources.tail.interceptors = ts st stt<br>
app-agent.sources.tail.interceptors.ts.type =     org.apache.flume.interceptor.TimestampInterceptor$Builder<br>
app-agent.sources.tail.interceptors.st.type = static<br>
app-agent.sources.tail.interceptors.st.key = logFileType<br>
app-agent.sources.tail.interceptors.st.value = jboss<br>
app-agent.sources.tail.interceptors.stt.type = static<br>
app-agent.sources.tail.interceptors.stt.key = host<br>
app-agent.sources.tail.interceptors.stt.value = Mart<br>

app-agent.sources.tailapache.interceptors = ts1 i1 st1<br>
app-agent.sources.tailapache.interceptors.ts1.type =     org.apache.flume.interceptor.TimestampInterceptor$Builder<br>
app-agent.sources.tailapache.interceptors.i1.type = static<br>
app-agent.sources.tailapache.interceptors.i1.key = logFileType<br>
app-agent.sources.tailapache.interceptors.i1.value = apache<br>
app-agent.sources.tailapache.interceptors.st1.type = static<br>
app-agent.sources.tailapache.interceptors.st1.key = host<br>
app-agent.sources.tailapache.interceptors.st1.value = Mart<br>

app-agent.sinks.avro-forward-sink.type = avro<br>
app-agent.sinks.avro-forward-sink.hostname = <<Host IP>><br>
app-agent.sinks.avro-forward-sink.port = 41414<br>
app-agent.sinks.avro-forward-sink.channel = fileChannel<br>

app-agent.sinks.avro-forward-sink-apache.type = avro<br>
app-agent.sinks.avro-forward-sink-apache.hostname = <<Host IP>><br>
app-agent.sinks.avro-forward-sink-apache.port = 41414<br>
app-agent.sinks.avro-forward-sink-apache.channel = fileChannel<br>

app-agent.channels.fileChannel.type=file<br>
app-agent.channels.fileChannel.dataDirs=/usr/local/lib/flume-ng/flume_channel/dataDir13<br>
app-agent.channels.fileChannel.checkpointDir=/usr/local/lib/flume-ng/flume_channel/checkpointDir13<br>
app-agent.channels.fileChannel.transactionCapacity = 50000<br>
app-agent.channels.fileChannel.capacity = 9000000
app-agent.channels.fileChannel.write-timeout = 250000
app-agent.channles.fileChannel.keep-alive=600

1 个答案:

答案 0 :(得分:-1)

从这里开始:When is "java.io.IOException:Connection reset by peer" thrown?

BalusC的回答:

  

另一方突然中断了连接中间的连接   交易。这可能有许多无法控制的原因   服务器端。例如。最终用户决定关闭客户端或   在仍与服务器交互时突然更改服务器,   或客户端程序崩溃,或最终用户的互联网   连接断开,或者终端用户的计算机崩溃等等。