Question

我们拥有4,000万份简历文件和100万个职位。我们对两个都使用数据框，然后交叉连接两个文档。然后我们使用finaldata.rdd.map（lambda row：process_row（row））该功能花费太多时间。它不是并行执行的。

sc = SparkContext()
    collection_file = sc.textFile("hdfs://ip:port/user/Mongo_11July")
    sqlContext = SQLContext(sc)
    collection_df = sqlContext.read.json(collection_file)
    collection_df.createOrReplaceTempView("temp1")


    jobtitle_file = sc.textFile("hdfs://ip:port/user/Jobtitle_11July")


    jobtitle_df = sqlContext.read.json(jobtitle_file)
    jobtitle_df.createOrReplaceTempView("jobtitle")



    df_cases = sqlContext.sql("SELECT _id,HTMLtext FROM temp1 ")
    df_job_titles = sqlContext.sql("SELECT distinct(Name) FROM jobtitle")



    schema = StructType([
        StructField("_id", StructType([StructField("$oid",StringType(),True)]), True),
        StructField("HTMLtext", StringType(), True),
        StructField("Name", StringType(), True),
        StructField("Tags_Add", ArrayType(StringType()), True)])   

    finaldata = df_cases.crossJoin(df_job_titles) 
    new_rdd = finaldata.rdd.map(lambda row: process_row(row)) 
    new_df = sqlContext.createDataFrame(new_rdd,schema) 
    final_new_df = new_df.groupBy("_id").agg(collect_list("Tags_Add").alias("Tags_Add"))
    flattenUdf = F.udf(fudf, ArrayType(StringType()))
    outputdataframe=final_new_df.select("_id", flattenUdf("Tags_Add").alias("Tags_Add"))

Apache Spark Python两个数据框rdd映射未并行执行

0 个答案: