Question

您能否为我提供配置Flume for Facebook数据流的步骤。

我们已成功配置水槽以从Twitter提取数据。

请查看我们为从twitter提取数据而创建的配置。

Twitter的Flume配置

TwitterAgent.sources = Twitter 
TwitterAgent.channels = MemChannel 
TwitterAgent.sinks = HDFS

# Describing/Configuring the source 
TwitterAgent.sources.Twitter.type =          org.apache.flume.source.twitter.TwitterSource
TwitterAgent.sources.Twitter.consumerKey =  4ENqf3q23iwdTSDJchv7w 
TwitterAgent.sources.Twitter.consumerSecret =      bAPTWfbRildBMWsEHo56SmZeXkftvZNCgvjHXbcUfAKoKzQjY0VIUOftTh6c 
TwitterAgent.sources.Twitter.accessToken = 736128293661855746-   rQIQYZNGCh9lW8XHCkjcnvwZH1BItnGi0XJ0gHM26F
TwitterAgent.sources.Twitter.accessTokenSecret = ehTsqX7GcU1aBqmekDcwPuu1csFOnfgzxc2EPtS0kudXOADeAAI 
TwitterAgent.sources.Twitter.keywords = modi, india elections, bjp,   congress, tdp,jana sena, pwan kalyan, mohanlal

# Describing/Configuring the sink 

TwitterAgent.sinks.HDFS.type = hdfs 
TwitterAgent.sinks.HDFS.hdfs.path = hdfs://env11-hadoop-  master.trv.flytxt.com:54310/user/Hadoop/twitter_data
TwitterAgent.sinks.HDFS.hdfs.fileType = DataStream
TwitterAgent.sinks.HDFS.hdfs.writeFormat = writable 
TwitterAgent.sinks.HDFS.hdfs.batchSize = 1
TwitterAgent.sinks.HDFS.hdfs.rollSize = 1
TwitterAgent.sinks.HDFS.hdfs.rollCount = 1

# Describing/Configuring the channel TwitterAgent.channels.MemChannel.type =   memory 
TwitterAgent.channels.MemChannel.capacity = 10000 
TwitterAgent.channels.MemChannel.transactionCapacity = 100
TwitterAgent.channels.MemChannel.type = file

# Binding the source and sink to the channel 
TwitterAgent.sources.Twitter.channels = MemChannel

TwitterAgent.sinks.HDFS.channel = MemChannel

For configuring twitter we need the following configurations

TwitterAgent.sources.Twitter.consumerKey =   4ENqf3q23iwsdfmhadfjafjkemliSYs7w 
TwitterAgent.sources.Twitter.consumerSecret =   bAPTWfbRildangxvasxvhaxjasbxkjtvUfAKoKzQjY0VIUOftTh6c 
TwitterAgent.sources.Twitter.accessToken = 7361282936618557ZNbcvHJxjxbnH1BItnGi0XJ0gHM26F
TwitterAgent.sources.Twitter.accessTokenSecret = ehTsASNMGCxvashgvcxjAHvcSFGcjahgPuu1csFO2EPtS0kudXOADeAAI

但是如何获取Facebook的相同内容？或者使用Flume

为Facebook数据流提供工作配置

Answer 1

是的，您可以通过Facebook的抄写工具以日志的形式从Facebook接收数据。

对于scribe的安装部分，您可以参考：http://blog.octo.com/en/scribe-installation/

对于工作部分，您可以参考：http://blog.octo.com/en/scribe-a-way-to-aggregate-data-and-why-not-to-directly-fill-the-hdfs/

如何为Facebook数据流配置apache flume

TwitterAgent.sinks.HDFS.channel = MemChannel

1 个答案: