Question

我正在构建一个python脚本，它将通过Apache spark执行，我将从存储在S3存储桶上的json文件生成RDD。我需要根据json文档中的一些数据过滤掉json RDD，从而生成一个新的json文件，其中包含过滤的json文件。需要将json文件上传到S3存储桶。因此，请通过pyspark为我的实施建议适当的解决方案。

输入json

{
"_id" : ObjectId("55a787ee9efccaeb288b457f"),
"data" : {
    "N◦ CATEGORIA" : 102.0, 
    "NOMBRE CATEGORIA" : "GASEOSAS", 
    "VARIABLE" : "TOP OF HEART", 
    "VAR." : "TOH", 
    "MARCA" : "COCA COLA ZERO", 
    "MES" : "ENERO", 
    "MES_N" : 1.0, 
    "AÑO" : 2014.0, 
    "UNIVERSO_TOTAL" : 1.0433982E7, 
    "UNIVERSO_FEMENINO" : 5529024.0, 
    "UNIVERSO_MASCULINO" : 4904958.0, 
    "PORCENTAJE_TOTAL" : 0.0066, 
    "PORCENTAJE_FEMENINO" : 0.0125, 
    "PORCENTAJE_MASCULINO" : null
}, 
"app_id" : ObjectId("5376349e11bc073138c33163"), 
"category" : "excel_RAC", 
"subcategory" : "RAC", 
"created_time" : NumberLong(1437042670), 
"instance_id" : null, 
"metric_date" : NumberLong(1437042670), 
"campaign_id" : ObjectId("5386602ba102b6cd4528ed93"), 
"datasource_id" : ObjectId("559f5c8f9efccacf0a3c9875"), 
"duplicate_id" : "695a3f5f562d0a02f1820fe5d91642a5"
}

输入json数据需要根据VARIABLE：“TOP OF HEART”进行过滤，然后生成输出json，如下所示

输出Json

{ 
 "_id" : ObjectId("55b5d19e9efcca86118b45a2"), 
"widget_type" : "rac_toh_excel", 
"campaign_id" : ObjectId("558554b29efccab00a3c987c"), 
"datasource_id" : ObjectId("55b5d18f9efcca770b3c986a"), 
"date_time" : NumberLong(1388530800), 
"data" : {
    "key" : "COCA COLA ZERO", 
    "values" : {
        "x" : NumberLong(1388530800), 
        "y" : 1.0433982E7, 
        "data" : {
            "id" : ObjectId("553a151e5c93ffe0408b46f9"), 
            "month" : 1.0, 
            "year" : 2014.0, 
            "total" : 1.0433982E7, 
            "variable" : "TOH", 
            "total_percentage" : 0.0066
        }
    }
}, 
"filter" : [

]
}

使用Pyspark处理json数据

0 个答案: