我正在使用Pyspark从ElasticSearch索引中读取地理点数据。我正在使用以下命令创建我的DataFrame。
df.loc[df.age==0, 'bal']=df.loc[df.age<=0].groupby('cohort').bal.sum().values
当我打印DataFrame的Schema时，我得到架构为
us_df = spark.read.format('es').option('es.query', us_q).option('es.read.field.as.array.include', 'extra_tags').load('index').select('centroid.lat', 'centroid.lon')但是当我尝试获得前10条记录时
root
|-- lat: double (nullable = true)
|-- lon: double (nullable = true)
我收到错误
us_df.take(10)我已经看到了跟踪，经度和经度作为ES的字符串返回
18/03/11 21:10:38 ERROR TaskSetManager: Task 0 in stage 6.0 failed 1 times; aborting job
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/ksripada/spark-2.1.1-bin-hadoop2.7/python/pyspark/sql/dataframe.py", line 429, in take
return self.limit(num).collect()
File "/Users/ksripada/spark-2.1.1-bin-hadoop2.7/python/pyspark/sql/dataframe.py", line 391, in collect
port = self._jdf.collectToPython()
File "/Users/ksripada/spark-2.1.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/java_gateway.py", line 1133, in __call__
File "/Users/ksripada/spark-2.1.1-bin-hadoop2.7/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/Users/ksripada/spark-2.1.1-bin-hadoop2.7/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o260.collectToPython.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 6.0 failed 1 times, most recent failure: Lost task 0.0 in stage 6.0 (TID 9, localhost, executor driver): java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Double
at scala.runtime.BoxesRunTime.unboxToDouble(BoxesRunTime.java:114)
at org.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class.getDouble(rows.scala:44)
at org.apache.spark.sql.catalyst.expressions.GenericInternalRow.getDouble(rows.scala:194)
at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231)
at org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
at org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
被推断为double，并且在读取数据时，它会抛出错误。
如何在创建DataFrame时强制将纬度和经度解释为字符串而不是双倍。
Spark版本 - 2.1.1 ES-Hadoop jar - elasticsearch-spark-20_2.11-5.2.2.jar ES版本 - 5.2.2
由于