Question

我尝试了下面的代码及其组合，以便读取S3文件夹中提供的所有文件，但似乎无济于事..敏感的信息/代码已从下面的脚本中删除。每个文件有6个，大小为6.5 GB。

#Spark Connection
sc<-spark_connect(master = "local" , config=config)


rd_1<-spark_read_csv(sc,name = "Retail_1",path = "s3a://mybucket/xyzabc/Retail_Industry/*/*",header = F,delimiter = "|")


# This is the S3 bucket/folder for files [One of the file names Industry_Raw_Data_000]
s3://mybucket/xyzabc/Retail_Industry/Industry_Raw_Data_000

这是我得到的错误

Error: org.apache.spark.sql.AnalysisException: Path does not exist: s3a://mybucket/xyzabc/Retail_Industry/*/*;
at org.apache.spark.sql.execution.datasources.DataSource$.org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary(DataSource.scala:710)

Answer 1

花了几个星期在谷歌上搜索该问题后，它才得以解决。解决方法在这里。

Sys.setenv(AWS_ACCESS_KEY_ID="abc") 
Sys.setenv(AWS_SECRET_ACCESS_KEY="xyz")

config<-spark_config()

config$sparklyr.defaultPackages <- c(
"com.databricks:spark-csv_2.10:1.5.0",
"com.amazonaws:aws-java-sdk-pom:1.10.34",
"org.apache.hadoop:hadoop-aws:2.7.3")



#Spark Connection
sc<-spark_connect(master = "local" , config=config)

# hadoop configurations
ctx <- spark_context(sc)
jsc <- invoke_static( sc,
"org.apache.spark.api.java.JavaSparkContext",
"fromSparkContext",
ctx
)

hconf <- jsc %>% invoke("hadoopConfiguration")  
hconf %>% invoke("set", "com.amazonaws.services.s3a.enableV4", "true")
hconf %>% invoke("set", "fs.s3a.fast.upload", "true")

folder_files<-"s3a://mybucket/abc/xyz"

rd_11<-spark_read_csv(sc,name = "Retail",path=folder_files,infer_schema = TRUE,header = F,delimiter = "|")


spark_disconnect(sc)

如何在R中使用sparklyr读取S3文件夹/存储桶中的所有文件？

1 个答案: