我有一个用法，我必须使用FLUME将cat文件加载到本地目录。我知道Flume不是为这种用例设计的，但我没有任何流数据。所以我决定使用这个并看看水槽如何工作我创建了我的配置文件，其中sources.type为exec，channels.type为file，sink.types为file_roll，我还为checkpointDirs和dataDir等通道配置了适当的配置。但是，当我运行代理时，它在源头上失败了说（flume.log）
FYI conf文件
#### Define a source, a channel, and a sink
agent.sources = src1
agent.channels = chan1
agent.sinks = sink1
#### Set the source type to Spooling Directory and set the directory
#### location to /home/flume/ingestion/
agent.sources.src1.type = exec
agent.sources.src1.commands= cat /xxxxx/xxxxxx/Xxxxx/xxxxx/test.dat
#### Configure the channel as simple in-memory queue
agent.channels.chan1.type = file
agent.channels.chan1.checkpointDir = /home/xxxxx/xxxxxxx/flume/checkpoint
agent.channels.chan1.dataDirs= /home/xxxxxxx/xxxxx/flume/data
#### Define the HDFS sink and set its path to your target HDFS directory
agent.sinks.sink1.type = file_roll
agent.sinks.sink1.sink.directory = /home/xxxxx/xxxxxx/xxxxx/flume/files
agent.sinks.sink1.sink.rollInterval = 0
agent.sinks.sink1.sink.rollSize = 0
agent.sinks.sink1.idleTimeout = 0
#### Connect source and sink
agent.sources.src1.channels = chan1
agent.sinks.sink1.channel = chan1
FYI水槽日志
27 Apr 2017 08:16:33,831 INFO [lifecycleSupervisor-1-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider.start:61) - Configuration provider starting
27 Apr 2017 08:16:33,849 INFO [conf-file-poller-0] (org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run:133) - Reloading configuration file:/usr/lib/gphd/apache-flume-1.4.0-bin/conf/flume-agent1.con
f
27 Apr 2017 08:16:33,859 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:sink1
27 Apr 2017 08:16:33,860 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:sink1
27 Apr 2017 08:16:33,860 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:sink1
27 Apr 2017 08:16:33,861 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:930) - Added sinks: sink1 Agent: agent
27 Apr 2017 08:16:33,861 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:sink1
27 Apr 2017 08:16:33,861 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:sink1
27 Apr 2017 08:16:33,861 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.addProperty:1016) - Processing:sink1
27 Apr 2017 08:16:33,918 INFO [conf-file-poller-0] (org.apache.flume.conf.FlumeConfiguration.validateConfiguration:140) - Post-validation flume configuration contains configuration for agents: [agent]
27 Apr 2017 08:16:33,919 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:150) - Creating channels
27 Apr 2017 08:16:33,934 INFO [conf-file-poller-0] (org.apache.flume.channel.DefaultChannelFactory.create:40) - Creating instance of channel chan1 type file
27 Apr 2017 08:16:33,938 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadChannels:205) - Created channel chan1
27 Apr 2017 08:16:33,939 INFO [conf-file-poller-0] (org.apache.flume.source.DefaultSourceFactory.create:39) - Creating instance of source src1, type exec
27 Apr 2017 08:16:33,941 ERROR [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.loadSources:366) - Source src1 has been removed due to an error during configuration
java.lang.IllegalStateException: The parameter command must be specified
at com.google.common.base.Preconditions.checkState(Preconditions.java:145)
at org.apache.flume.source.ExecSource.configure(ExecSource.java:221)
at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
at org.apache.flume.node.AbstractConfigurationProvider.loadSources(AbstractConfigurationProvider.java:331)
at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:102)
at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
27 Apr 2017 08:16:33,944 INFO [conf-file-poller-0] (org.apache.flume.sink.DefaultSinkFactory.create:40) - Creating instance of sink: sink1, type: file_roll
27 Apr 2017 08:16:33,952 INFO [conf-file-poller-0] (org.apache.flume.node.AbstractConfigurationProvider.getConfiguration:119) - Channel chan1 connected to [sink1]
27 Apr 2017 08:16:33,962 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:138) - Starting new configuration:{ sourceRunners:{} sinkRunners:{sink1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProc
essor@54560854 counterGroup:{ name:null counters:{} } }} channels:{chan1=FileChannel chan1 { dataDirs: [/home/phdftpuser/dev_phase/flume/data] }} }
27 Apr 2017 08:16:33,976 INFO [conf-file-poller-0] (org.apache.flume.node.Application.startAllComponents:145) - Starting Channel chan1
27 Apr 2017 08:16:33,976 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.FileChannel.start:282) - Starting FileChannel chan1 { dataDirs: [/home/phdftpuser/dev_phase/flume/data] }...
27 Apr 2017 08:16:33,996 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.<init>:322) - Encryption is not enabled
27 Apr 2017 08:16:33,998 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:369) - Replay started
27 Apr 2017 08:16:34,000 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:381) - Found NextFileID 0, from []
27 Apr 2017 08:16:34,008 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>:53) - Starting up with /home/phdftpuser/dev_phase/flume/checkpoint/checkpoint and /home/phdftpuser/dev_phase/flu
me/checkpoint/checkpoint.meta
27 Apr 2017 08:16:34,009 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>:57) - Reading checkpoint metadata from /home/phdftpuser/dev_phase/flume/checkpoint/checkpoint.meta
27 Apr 2017 08:16:34,031 WARN [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:442) - Checkpoint may not have completed successfully. Forcing full replay, this may take a while.
org.apache.flume.channel.file.BadCheckpointException: The checkpoint metadata file does not exist or has zero length
at org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>(EventQueueBackingStoreFileV3.java:61)
at org.apache.flume.channel.file.EventQueueBackingStoreFactory.get(EventQueueBackingStoreFactory.java:75)
at org.apache.flume.channel.file.Log.replay(Log.java:412)
at org.apache.flume.channel.file.FileChannel.start(FileChannel.java:302)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
27 Apr 2017 08:16:34,032 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Serialization.deleteAllFiles:105) - Skipping in_use.lock because it is in excludes set
27 Apr 2017 08:16:34,039 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Serialization.deleteAllFiles:118) - Deleted the following files: , checkpoint, checkpoint.meta.
27 Apr 2017 08:16:34,064 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFile.<init>:91) - Preallocated /home/phdftpuser/dev_phase/flume/checkpoint/checkpoint to 8008232 for capacity 1000000
27 Apr 2017 08:16:34,064 INFO [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.EventQueueBackingStoreFileV3.<init>:53) - Starting up with /home/phdftpuser/dev_phase/flume/checkpoint/checkpoint and /home/phdftpuser/dev_phase/flu
me/checkpoint/checkpoint.meta
27 Apr 2017 08:16:34,068 ERROR [lifecycleSupervisor-1-0] (org.apache.flume.channel.file.Log.replay:476) - Failed to initialize Log on [channel=chan1]
java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.
at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)
我多次尝试更改配置文件，但结果没有任何好处。有人可以帮我解决这个问题。
答案 0 :(得分：0)
java.lang.IllegalStateException：参数命令必须是 指定的
对于 exec 来源，command是必需属性。您的配置文件改为commands。
将配置修改为，
agent.sources.src1.type = exec
agent.sources.src1.command = cat /xxxxx/xxxxxx/Xxxxx/xxxxx/test.dat
这些属性不属于file_roll接收器，请将其删除。
agent.sinks.sink1.sink.rollSize = 0
agent.sinks.sink1.idleTimeout = 0