Question

在控制台上

aws s3 cp s3://com.philadelphia-reflections.php/.htaccess htaccess

做了预期的事情：它下载文本文件

然而，

在pyspark，

sc.hadoopFile(path = 's3n://com.philadelphia-reflections.php/.htaccess', 
              conf = {
                      'fs.s3n.awsAccessKeyId': '...',
                      'fs.s3n.awsSecretAccessKey': '...',
                    },
              inputFormatClass = "org.apache.hadoop.mapred.TextInputFormat" , 
              keyClass = "org.apache.hadoop.io.Text",
              valueClass = "org.apache.hadoop.io.LongWritable"
             )

产生以下内容，其中必不可少的部分是＆＃34; Input path does not exist: s3n://com.philadelphia-reflections.php/.htaccess＆＃34;

s3：// ...生成＆＃34; java.io.IOException: /.htaccess doesn't exist＆＃34;
s3a：// ...生成＆＃34; org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: s3a://com.philadelphia-reflections.php/.htaccess＆＃34;

---------------------------------------------------------------------------
Py4JJavaError                             Traceback (most recent call last)
<ipython-input-14-67f3549c21f9> in <module>()
      6               inputFormatClass = "org.apache.hadoop.mapred.TextInputFormat" ,
      7               keyClass = "org.apache.hadoop.io.Text",
----> 8               valueClass = "org.apache.hadoop.io.LongWritable"
      9              )

/home/george/spark-2.1.1-bin-hadoop2.7/python/pyspark/context.py in hadoopFile(self, path, inputFormatClass, keyClass, valueClass, keyConverter, valueConverter, conf, batchSize)
    706         jrdd = self._jvm.PythonRDD.hadoopFile(self._jsc, path, inputFormatClass, keyClass,
    707                                               valueClass, keyConverter, valueConverter,
--> 708                                               jconf, batchSize)
    709         return RDD(jrdd, self)
    710 

/home/george/spark-2.1.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py in __call__(self, *args)
   1131         answer = self.gateway_client.send_command(command)
   1132         return_value = get_return_value(
-> 1133             answer, self.gateway_client, self.target_id, self.name)
   1134 
   1135         for temp_arg in temp_args:

/home/george/spark-2.1.1-bin-hadoop2.7/python/pyspark/sql/utils.py in deco(*a, **kw)
     61     def deco(*a, **kw):
     62         try:
---> 63             return f(*a, **kw)
     64         except py4j.protocol.Py4JJavaError as e:
     65             s = e.java_exception.toString()

/home/george/spark-2.1.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
    317                 raise Py4JJavaError(
    318                     "An error occurred while calling {0}{1}{2}.\n".
--> 319                     format(target_id, ".", name), value)
    320             else:
    321                 raise Py4JError(

Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.hadoopFile.
: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: s3n://com.philadelphia-reflections.php/.htaccess
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:287)
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:229)
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:315)
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:202)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
    at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:252)
    at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:250)
    at scala.Option.getOrElse(Option.scala:121)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:250)
    at org.apache.spark.rdd.RDD$$anonfun$take$1.apply(RDD.scala:1333)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
    at org.apache.spark.rdd.RDD.take(RDD.scala:1327)
    at org.apache.spark.api.python.SerDeUtil$.pairRDDToPython(SerDeUtil.scala:203)
    at org.apache.spark.api.python.PythonRDD$.hadoopFile(PythonRDD.scala:625)
    at org.apache.spark.api.python.PythonRDD.hadoopFile(PythonRDD.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
    at py4j.Gateway.invoke(Gateway.java:280)
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
    at py4j.commands.CallCommand.execute(CallCommand.java:79)
    at py4j.GatewayConnection.run(GatewayConnection.java:214)
    at java.lang.Thread.run(Thread.java:748)

Answer 1

是raise_on_unfiltered_parameters你的水桶的名字？如果是这样，它不适用于s3n或s3a客户端，这需要存储桶具有简单的主机名，例如com.philadelphia-reflections.php

使用简单的“主机名”样式存储桶名称创建存储桶，并将数据复制到那里以供使用。

pyspark Spark 2.1.1 s3n输入路径不存在

1 个答案: