Question

我正在尝试运行pyspark脚本。我正在此脚本中进行并行处理。在执行任务时，出现一些错误。我试图运行sql并创建该sql输出的DF，然后尝试将df保存到文件中。每次我遇到相同的错误。你能帮我解决这个问题吗？让我知道您是否需要更多信息。

我正在为此使用python 3.4.2。

代码：

        query = ''
        for index,row in query_to_df_job_main.iterrows():
            query += row['query1'].strip('\"')+' union '
            print(query)
            #print(table_list)
        query = query.rstrip(' union ')
        logging.info("Query for Master job" +" " +  row['Master Job Name'] +" " +query)
        df = sqlContext.sql(query)

        print("Target:  "+row['TargetPath'] )
        print(df.printSchema())
        #count = df.count()
        #print(count)
        df.repartition(1).write.format('com.databricks.spark.csv').mode('overwrite').option("dateFormat", "MM-dd-yyyy hh:mm:ss").save(row['TargetPath'], quote='',sep='|',header='True',nullValue=None)#,compression="gzip"

错误：

__str__ returned non-string (type JavaObject)
Query Execution is Failed
An error occurred while calling o194.schema. Trace:
py4j.Py4JException: Method treeString([]) does not exist
        at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:318)
        at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:326)
        at py4j.Gateway.invoke(Gateway.java:274)
        at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
        at py4j.commands.CallCommand.execute(CallCommand.java:79)
        at py4j.GatewayConnection.run(GatewayConnection.java:238)
        at java.lang.Thread.run(Thread.java:748)

获取错误调用o206.toString时发生错误。在运行pyspark脚本时进行跟踪

0 个答案: