Flume无法初始化Log on [channel = chan1] java.lang.UnsupportedOperationException:这应该被子类覆盖

时间:2017-04-27 18:32:36

标签: hadoop hdfs flume flume-ng

我遇到了flume频道的问题,它无法在频道初始化日志文件。我正在尝试使用Flume将其文件加载到本地目录。

以下是配置文件信息和日志文件信息

配置文件

#### Define a source, a channel, and a sink
agent.sources = src1
agent.channels = chan1
agent.sinks = sink1


#### Set the source type to Spooling Directory and set the directory
#### location to /home/flume/ingestion/


agent.sources.src1.type = exec
agent.sources.src1.command= cat /xxxxx/xxxxxx/Xxxxx/xxxxx/test.dat



#### Configure the channel as simple in-memory queue
agent.channels.chan1.type = file
agent.channels.chan1.checkpointDir = /home/xxxxx/xxxxxxx/flume/checkpoint
agent.channels.chan1.dataDirs= /home/xxxxxxx/xxxxx/flume/data


#### Define the HDFS sink and set its path to your target HDFS directory
agent.sinks.sink1.type = file_roll
agent.sinks.sink1.sink.directory = /home/xxxxx/xxxxxx/xxxxx/flume/files
agent.sinks.sink1.sink.rollInterval = 0
agent.sinks.sink1.sink.rollSize = 0
agent.sinks.sink1.idleTimeout = 0


#### Connect source and sink
agent.sources.src1.channels = chan1
agent.sinks.sink1.channel = chan1

水槽日志文件

27 Apr 2017 10:49:40,503 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61)  - Configuration provider starting
27 Apr 2017 10:49:40,521 INFO  [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133)  - Reloading configuration file:/usr/lib/gphd/apache-flume-1.4.0-bin/conf/flume-agent1.conf
27 Apr 2017 10:49:40,530 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:sink1
27 Apr 2017 10:49:40,531 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:sink1
27 Apr 2017 10:49:40,531 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930)  - Added sinks: sink1 Agent: agent
27 Apr 2017 10:49:40,531 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:sink1
27 Apr 2017 10:49:40,532 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016)  - Processing:sink1
27 Apr 2017 10:49:40,674 INFO  [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140)  - Post-validation flume configuration contains configuration for agents: [agent]
27 Apr 2017 10:49:40,675 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:150)  - Creating channels
27 Apr 2017 10:49:40,687 INFO  [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:40)  - Creating instance of channel chan1 type file
27 Apr 2017 10:49:40,691 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205)  - Created channel chan1
27 Apr 2017 10:49:40,692 INFO  [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:39)  - Creating instance of source src1, type exec
27 Apr 2017 10:49:40,701 INFO  [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40)  - Creating instance of sink: sink1, type: file_roll
27 Apr 2017 10:49:40,706 INFO  [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:119)  - Channel chan1 connected to [src1, sink1]
27 Apr 2017 10:49:40,716 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138)  - Starting new configuration:{ sourceRunners:{src1=EventDrivenSourceRunner: { source:org.apache.flume.source.ExecSource{name:src1,state:IDLE} }} sinkRunners:{sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@27d03700 counterGroup:{ name:null counters:{} } }} channels:{chan1=FileChannel chan1 { dataDirs: [/home/phdftpuser/dev_phase/flume/data] }} }
27 Apr 2017 10:49:40,730 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145)  - Starting Channel chan1
27 Apr 2017 10:49:40,733 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:282)  - Starting FileChannel chan1 { dataDirs: [/home/phdftpuser/dev_phase/flume/data] }...
27 Apr 2017 10:49:40,747 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.<init>:322)  - Encryption is not enabled
27 Apr 2017 10:49:40,748 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:369)  - Replay started
27 Apr 2017 10:49:40,750 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:381)  - Found NextFileID 0, from []
27 Apr 2017 10:49:40,782 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFile.<init>:91)  - Preallocated /root/.flume/file-channel/checkpoint/checkpoint to 8008232 for capacity 1000000
27 Apr 2017 10:49:40,785 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>:53)  - Starting up with /root/.flume/file-channel/checkpoint/checkpoint and /root/.flume/file-channel/checkpoint/checkpoint.meta
27 Apr 2017 10:49:40,813 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:476)  - Failed to initialize Log on [channel=chan1]
java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
        at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
        at org.apache.flume.channel.file.proto.ProtosFactory$Checkpoint.getSerializedSize(ProtosFactory.java:233)
        at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
        at org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>(EventQueueBackingStoreFileV3.java:111)
        at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:70)
        at org.apache.flume.channel.file.Log.replay(Log.java:412)
        at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
27 Apr 2017 10:49:40,816 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:313)  - Failed to start the file channel [channel=chan1]
java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
        at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
        at org.apache.flume.channel.file.proto.ProtosFactory$Checkpoint.getSerializedSize(ProtosFactory.java:233)
        at com.google.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:84)
        at org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>(EventQueueBackingStoreFileV3.java:111)
        at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:70)
        at org.apache.flume.channel.file.Log.replay(Log.java:412)
        at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
        at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
27 Apr 2017 10:49:40,817 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:173)  - Starting Sink sink1
27 Apr 2017 10:49:40,818 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.sink.RollingFileSink.start:104)  - Starting org.apache.flume.sink.RollingFileSink{name:sink1, channel:chan1}...
27 Apr 2017 10:49:40,818 INFO  [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:184)  - Starting Source src1
27 Apr 2017 10:49:40,819 INFO  [lifecycleSupervisor-1-0] (org.apache.flume.source.ExecSource.start:163)  - Exec source starting with command:cat /wadpfs4/filehub/DTCC/inbound/DTC.A001.dat
27 Apr 2017 10:49:40,820 INFO  [lifecycleSupervisor-1-1] (org.apache.flume.instrumentation.MonitoredCounterGroup.register:110)  - Monitoried counter group for type: SINK, name: sink1, registered successfully.

请你帮我解决一下。

0 个答案:

没有答案