Question

我正在尝试将pyspark DataFrame写入Redshift，但是会导致错误：-

java.util.ServiceConfigurationError：org.apache.spark.sql.sources.DataSourceRegister：提供程序org.apache.spark.sql.avro.AvroFileFormat无法实例化

由以下原因引起：java.lang.NoSuchMethodError：org.apache.spark.sql.execution.datasources.FileFormat。$ init $（Lorg / apache / spark / sql / execution / datasources / FileFormat;）V

火花版本：2.4.1

Spark-submit命令：spark-submit --master local [*] --jars〜/ Downloads / spark-avro_2.12-2.4.0.jar，〜/ Downloads / aws-java-sdk-1.7.4 .jar，〜/ Downloads / RedshiftJDBC42-no-awssdk-1.2.20.1043.jar，〜/ Downloads / hadoop-aws-2.7.3.jar，〜/ Downloads / hadoop-common-2.7.3.jar --packages com .databricks：spark-redshift_2.11：2.0.1，com.amazonaws：aws-java-sdk：1.7.4，org.apache.hadoop：hadoop-aws：2.7.3，org.apache.hadoop：hadoop-common ：2.7.3，org.apache.spark：spark-avro_2.12：2.4.0 script.py

from pyspark.sql import DataFrameReader
from pyspark.context import SparkContext
from pyspark.sql.session import SparkSession
from pyspark.sql import SQLContext
from pyspark.sql.functions import pandas_udf, PandasUDFType
from pyspark.sql.types import *

import sys
import os

pe_dl_dbname            = os.environ.get("REDSHIFT_DL_DBNAME")
pe_dl_host              = os.environ.get("REDSHIFT_DL_HOST")
pe_dl_port              = os.environ.get("REDSHIFT_DL_PORT")
pe_dl_user              = os.environ.get("REDSHIFT_DL_USER")
pe_dl_password          = os.environ.get("REDSHIFT_DL_PASSWORD")

s3_bucket_path = "s3-bucket-name/sub-folder/sub-sub-folder"
tempdir = "s3a://{}".format(s3_bucket_path)

driver = "com.databricks.spark.redshift"
sc = SparkContext.getOrCreate()
sqlContext = SQLContext(sc)
spark = SparkSession(sc)
spark.conf.set("spark.sql.execution.arrow.enabled", "true")

sc._jsc.hadoopConfiguration().set("fs.s3.impl","org.apache.hadoop.fs.s3native.NativeS3FileSystem")

datalake_jdbc_url = 'jdbc:redshift://{}:{}/{}?user={}&password={}'.format(pe_dl_host, pe_dl_port, pe_dl_dbname, pe_dl_user, pe_dl_password)

"""
The table is created in Redshift as follows:
create table adhoc_analytics.testing (name varchar(255), age integer);
"""
l = [('Alice', 1)]
df = spark.createDataFrame(l, ['name', 'age'])
df.show()
df.write \
  .format("com.databricks.spark.redshift") \
  .option("url", datalake_jdbc_url) \
  .option("dbtable", "adhoc_analytics.testing") \
  .option("tempdir", tempdir) \
  .option("tempformat", "CSV") \
  .save()

Answer 1

Databricks Spark-Redshift不适用于Spark版本2.4.1，这是我维护使其与Spark 2.4.1一起使用的版本 https://github.com/goibibo/spark-redshift

如何使用它：

pyspark-软件包“ com.github.goibibo：spark-redshift：v4.1.0”-存储库“ https://jitpack.io”

如何编写一个pyspark-dataframe进行redshift？

1 个答案: