Question

我已经定义了一个基本脚本来创建一个DF，其中的数据来自我在redshift中的一个表。我运行这个过程但是我一直在努力争取一段我无法解释的信息。

日志中的错误输出为：

"/mnt/yarn/usercache/root/appcache/application_1525803778049_0004/container_1525803778049_0004_01_000001/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o60.getDynamicFrame. : java.lang.UnsupportedOperationException: empty.reduceLeft at scala.collection.

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.dynamicframe import DynamicFrame, DynamicFrameReader, DynamicFrameWriter, DynamicFrameCollection
from pyspark.sql.functions import lit
from awsglue.job import Job

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)

table = glueContext.create_dynamic_frame.from_options(connection_type="redshift", connection_options = 
    {"url": "jdbc:redshift://xxxxx.yyyyy.us-east-1.redshift.amazonaws.com:5439/db",
    "user": "yyyy",
    "password": "yyyyy",
    "dbtable": "schema.table_name",
    "redshiftTmpDir": "s3://aws-glue-temporary-accountnumber-us-east-1/"},
    format="orc", 
    transformation_ctx="table" )

table.show()

dfred = table.toDF().createOrReplaceTempView("table_df")

job.commit()

请感谢您提供给我的任何帮助。非常感谢

Glue AWS：调用o60.getDynamicFrame时发生错误

0 个答案: