Question

我有3个节点集群，使用5.9版本的最新cloudera parcel。所有三个操作系统都是CentOS 6.7。我是第一次使用Flume。

我的目的是将网页数据流式传输到HDFS。然而，这个网页是我的第三方网站，新闻网站，所以我不知道用哪个端口连接。

Curl和telnet发生在端口80上，因此我使用它。但是得到错误。

我的Flume.conf是：

tier1.sources  = http-source
tier1.channels = mem-channel-1
tier1.sinks    = hdfs-sink
tier1.sources.http-source.type     = http
tier1.sources.http-source.handler = org.apache.flume.source.http.JSONHandler
tier1.sources.http-source.bind     = 132.247.1.32
tier1.sources.http-source.port     = 80
tier1.sources.http-source.channels = mem-channel-1
tier1.channels.mem-channel-1.type   = memory
tier1.sinks.hdfs-sink.type         = hdfs
tier1.sinks.hdfs-sink.channel      = mem-channel-1
tier1.sinks.hdfs-sink.hdfs.path    = /flume/events/%y-%m-%d/%H%M/%S
# Other properties are specific to each type of
# source, channel, or sink. In this case, we
# specify the capacity of the memory channel.
tier1.channels.mem-channel-1.capacity = 100

错误

2016-12-19 16:45:00,353 WARN org.mortbay.log: failed SelectChannelConnector@132.247.1.32:80: java.net.BindException: Cannot assign requested address
2016-12-19 16:45:00,353 WARN org.mortbay.log: failed Server@36772002: java.net.BindException: Cannot assign requested address
2016-12-19 16:45:00,353 ERROR org.apache.flume.source.http.HTTPSource: Error while starting HTTPSource. Exception follows.
java.net.BindException: Cannot assign requested address**
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.mortbay.jetty.nio.SelectChannelConnector.open(SelectChannelConnector.java:216)
at org.mortbay.jetty.nio.SelectChannelConnector.doStart2016-12-19 16:45:00,364 ERROR org.apache.flume.lifecycle.LifecycleSupervisor: Unable to start EventDrivenSourceRunner: { source:org.apache.flume.source.http.HTTPSource{
name:http-source,state:IDLE} } - Exception follows.
java.lang.RuntimeException: java.net.BindException: Cannot assign requested address(SelectChannelConnector.java:315)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.mortbay.jetty.Server.doStart(Server.java:235)
at org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
at org.apache.flume.source.http.HTTPSource.start(HTTPSource.java:207)
at org.apache.flume.source.EventDrivenSourceRunner.start(EventDrivenSourceRunner.java:44)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

Answer 1

尝试更改源配置，如下所示： httpagent.sources.http-source.port = 80 httpagent.sources.http-source.bind = localhost httpagent.sources.http-source.url = 132.247.1.32

注意：如果132.247.1.32不起作用，请尝试提供主机名。

Answer 2

Flume的HTTP源提供了一种使用GET / POST将数据发送到flume代理的方法。 HTTP源不会出来并为您从网站获取数据，它会设置HTTP服务器并等待GET / POST并接受来自该端点的数据。

我的建议是创建一个自定义来源，以获取您需要的网页。

使用Flume将网页数据流式传输到HDFS

2 个答案: