Question

我使用以下代码将源目录中的文件写入hdfs。

  # Initialize agent's source, channel and sink
agent.sources = test
agent.channels = memoryChannel
agent.sinks = flumeHDFS

# Setting the source to spool directory where the file exists
agent.sources.test.type = spooldir
agent.sources.test.spoolDir = /Data

# Setting the channel to memory
agent.channels.memoryChannel.type = memory
# Max number of events stored in the memory channel
agent.channels.memoryChannel.capacity = 10000
# agent.channels.memoryChannel.batchSize = 15000
agent.channels.memoryChannel.transactioncapacity = 1000000

# Setting the sink to HDFS
agent.sinks.flumeHDFS.type = hdfs
agent.sinks.flumeHDFS.hdfs.path = /user/team
agent.sinks.flumeHDFS.hdfs.fileType = DataStream

# Write format can be text or writable
agent.sinks.flumeHDFS.hdfs.writeFormat = Text

# use a single csv file at a time
agent.sinks.flumeHDFS.hdfs.maxOpenFiles = 1

# rollover file based on maximum size of 10 MB
agent.sinks.flumeHDFS.hdfs.rollCount=0
agent.sinks.flumeHDFS.hdfs.rollInterval=2000
agent.sinks.flumeHDFS.hdfs.rollSize = 0
agent.sinks.flumeHDFS.hdfs.batchSize =1000000

# never rollover based on the number of events
agent.sinks.flumeHDFS.hdfs.rollCount = 0

# rollover file based on max time of 1 min
#agent.sinks.flumeHDFS.hdfs.rollInterval = 0
# agent.sinks.flumeHDFS.hdfs.idleTimeout = 600

# Connect source and sink with channel
agent.sources.TwitterExampleDir.channels = memoryChannel
agent.sinks.flumeHDFS.channel = memoryChannel

但是我收到以下错误

：无法配置组件！ org.apache.flume.conf.ConfigurationException：配置失败零件！在org.apache.flume.conf.source.SourceConfiguration.configure（SourceConfigurati on.java:110）在org.apache.flume.conf.FlumeConfiguration $ AgentConfiguration.validateSources（ FlumeConfiguration.java:566）在org.apache.flume.conf.FlumeConfiguration $ AgentConfiguration.isValid（FlumeCon） figuration.java:345）在org.apache.flume.conf.FlumeConfiguration $ AgentConfiguration.access $ 000（Flume Configuration.java:212）在org.apache.flume.conf.FlumeConfiguration.validateConfiguration（FlumeConfigur） ation.java:126）在org.apache.flume.conf.FlumeConfiguration。（FlumeConfiguration.java:108）在org.apache.flume.node.PropertiesFileConfigurationProvider.getFlumeConfigurat 离子（PropertiesFileConfigurationProvider.java:193） at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration（Abstrac） tConfigurationProvider.java:94）在org.apache.flume.node.PollingPropertiesFileConfigurationProvider $ FileWatcher Runnable.run（PollingPropertiesFileConfigurationProvider.java:140） at java.util.concurrent.Executors $ RunnableAdapter.call（Executors.java:511） at java.util.concurrent.FutureTask.runAndReset（FutureTask.java:308） at java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.access $ 301（ScheduledThreadPoolExecutor.java:180） at java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.run（Sch eduledThreadPoolExecutor.java:294）在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:11 42） at java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:6 17） at java.lang.Thread.run（Thread.java:745）引起：org.apache.flume.conf.ConfigurationException：没有为测试设置通道在org.apache.flume.conf.source.SourceConfiguration.configure（SourceConfigurati on.java:68） ......还有15个

任何人都可以帮助我将我的数据从源目录/ Data放到hdfs目录/ user / team吗？

Answer 1

Stack Trace提及：

No channels set for test

您指定的sources为test

agent.sources = test

但是在将其连接到channel

时

agent.sources.TwitterExampleDir.channels = memoryChannel

所以你必须提到test而不是TwitterExampleDir

Apache Flume spoolDirectory配置失败

1 个答案: