Question

我正在使用cloudera flume的假脱机目录源和HDFS作为接收器。我正面临 Serializer已关闭错误。我只复制一个文件，这是在我使用 scp

复制第一个文件后发生的

我的经纪人如下：

    agentaccesscombined.sources=spooldir-accesscombinedsource
    agentaccesscombined.sinks=hdfs-accesscombinedsink
    agentaccesscombined.channels=chaccesscombined

    # flume spooldir source
    agentaccesscombined.sources.spooldir-accesscombinedsource.type=spooldir
    agentaccesscombined.sources.spooldir-accesscombinedsource.spoolDir=/var/spoolAccessCombinedDir
    agentaccesscombined.sources.spooldir-accesscombinedsource.ignorePattern=\\w.*.filepart
    agentaccesscombined.sources.spooldir-accesscombinedsource.deletePolicy=immediate
    agentaccesscombined.sources.spooldir-accesscombinedsource.fileSuffix=.SPOOL
    agentaccesscombined.sources.spooldir-accesscombinedsource.fileHeader=true
    agentaccesscombined.sources.spooldir-accesscombinedsource.bufferMaxLineLength=70000
    agentaccesscombined.sources.spooldir-accesscombinedsource.bufferMaxLines=10000
    agentaccesscombined.sources.spooldir-accesscombinedsource.batchSize=1000
    agentaccesscombined.sources.spooldir-accesscombinedsource.fileHeaderKey=file

    #flume hdfs-sink
    agentaccesscombined.sinks.hdfs-accesscombinedsink.type=hdfs
    agentaccesscombined.sinks.hdfs-accesscombinedsink.hdfs.path=hdfs://cldx-1044:1200:8020/flumeOut_spoolDir_access_combined
    agentaccesscombined.sinks.hdfs-accesscombinedsink.hdfs.rollSize=12553700
    agentaccesscombined.sinks.hdfs-accesscombinedsink.hdfs.rollCount=12553665
    agentaccesscombined.sinks.hdfs-accesscombinedsink.hdfs.rollInterval=100000
    agentaccesscombined.sinks.hdfs-accesscombinedsink.hdfs.fileType=DataStream
    agentaccesscombined.sinks.hdfs-accesscombinedsink.hdfs.writeFormat=Text
    agentaccesscombined.sinks.hdfs-accesscombinedsink.round = true
    agentaccesscombined.sinks.hdfs-accesscombinedsink.roundValue=50
    agentaccesscombined.sinks.hdfs-accesscombinedsink.roundUnit=minute
    agentaccesscombined.sinks.hdfs-accesscombinedsink.hdfs.idleTimeout=5

    #flume channel 
    agentaccesscombined.channels.chaccesscombined.type=file
    agentaccesscombined.channels.chaccesscombined.capacity=1000000
    agentaccesscombined.channels.chaccesscombined.transactionCapacity = 1000
    agentaccesscombined.channels.chaccesscombined.checkpointInterval=30000
    agentaccesscombined.channels.chaccesscombined.maxFileSize=2146435071
    agentaccesscombined.channels.chaccesscombined.minimumRequiredSpace=524288000
    agentaccesscombined.channels.chaccesscombined.keep-alive=30
    agentaccesscombined.channels.chaccesscombined.write-timeout=30
    agentaccesscombined.channels.chaccesscombined.checkpoint-timeout=6000
    agentaccesscombined.channels.chaccesscombined.checkpointDir=/tmp/flume/java/checkpoint_accesscombined
    agentaccesscombined.channels.chaccesscombined.dataDirs=/tmp/flume/java/data_accesscombined


agentaccesscombined.sources.spooldir-accesscombinedsource.channels=chaccesscombined
agentaccesscombined.sinks.hdfs-accesscombinedsink.channel=chaccesscombined

如果我使用 winscp 复制文件，它可以正常工作但不使用 scp 。请帮帮我。

提前致谢。

Answer 1

要解决此问题，请重新启动您的水槽代理。然后使用复制原子文件的方法。

假脱机目录源需要一旦文件开始读取它就不会改变。如果文件发生更改，则会记录错误消息并开始产生错误，如上图所示。

cp不是原子的。我不知道scp等等。也许复制到临时目录然后使用mv。

Answer 2

您可以使用winscp将文件上传到临时目录，然后通过“mv”移动到水槽监控目录。 mv操作是原子的。您可能需要inotify进行自动化。

假脱机目录源异常[序列化器已关闭]

2 个答案: