Question

我正在使用python 我通过卡夫卡收到json词典到火花流， JSON就像{“a”：10} {“a”：20}（一个dict意味着一个kafka消息），关键将是“a”总是但有多少字典，这是不确定的。现在我想在上面的情况下平均10和20。据我所知，averageByKey可能很有用。但是如何使用，我不知道。任何帮助都会很棒！

感谢您的阅读。。。。

更新

from __future__ import print_function

import sys

from pyspark import SparkContext
from pyspark.streaming import StreamingContext
from pyspark.streaming.kafka import KafkaUtils
import json

def createContext():
    sc = SparkContext(appName="PythonSparkStreamingKafka_RM_02")
    sc.setLogLevel("WARN")
    ssc = StreamingContext(sc, 60)

    kafkaStream = KafkaUtils.createStream(ssc, 'localhost:2181', 'spark-streaming-consumer', {'':1})
    raw = kafkaStream.map(lambda kafkaS: kafkaS[1])
    clean = raw.map(lambda v: json.loads(v))
    print (dir(clean))
    clean.pprint()

    add=clean.map(lambda xs: ('Total',xs['hello'])).reduceByKey(lambda a, b: a+b)
    add.pprint()

    count_var = clean.count()
    count_var.pprint()
    average = add.map(lambda tpl: tpl[1]/float(60))
    average.pprint()
    return ssc

if __name__ == "__main__":

    ssc = StreamingContext.getOrCreate('/path/checkpoint_v'+sys.argv[1],lambda: createContext())
    ssc.start()
    ssc.awaitTermination()

现在

 in above program I'm getting add.pprint() output as below for example:
Stream is like:
{u'hello': 26}
{u'hello': 28}
{u'hello': 31}
{u'hello': 35}
{u'hello': 40}
{u'hello': 46}

>('Total',206) 

and output of count_var.pprint() as below for example:
> 6

The question is, in below line 

> average = add.map(lambda tpl: tpl[1]/float(60))
I want to use value of count_var.pprint()(which is 6) instead of static value 60

So how can I use stream object as integer in above operation

Answer 1

首先，您需要将事件映射到某个可处理类型，例如元组。然后你可以使用经典的“map - ＆gt; reduceByKey - ＆gt; map”来计算这样的平均值：

import json

ssc = StreamingContext(spark.sparkContext, 1)
dstream = KafkaUtils.createDirectStream(ssc, ['topic'], client_configuration, 
              valueDecoder=lambda s: json.loads(s.decode('ascii')))

def map_event(raw):
    item = list(raw[1].items())[0]
    return (item[0], (1, item[1]))

dstream.map(map_event).reduceByKey(lambda a, b: (a[0] + b[0], a[1] + b[1])) \
    .map(lambda r: (r[0], float(r[1][1]) / r[1][0])) \
    .pprint()

ssc.start()

使用apache spark streaming平均数据

1 个答案: