我正在尝试从最终用户提供的参数（通过REST api）创建数据框，以便从我的模型中获取预测。但是我在创建数据框时遇到错误。
** Approach#1** (using tuple of values and list of columns)
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/02/10 13:01:13 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/02/10 13:01:14 WARN Utils: Your hostname, pyspark-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3)
18/02/10 13:01:14 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
18/02/10 13:01:17 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
### Tuple is [(800, 0, 0.3048, 71.3, 0.0026634)]
### schema --> struct<Freq_Hz:int,displ_thick_m:double,Chord_m:double,V_inf_mps:double,AoA_Deg:int>
### session --> <pyspark.sql.conf.RuntimeConfig object at 0x7f1b68086860>
### data frame --> MapPartitionsRDD[8] at toJavaRDD at NativeMethodAccessorImpl.java:0
127.0.0.1 - - [10/Feb/2018 13:01:37] "GET /test HTTP/1.1" 500 -
Traceback (most recent call last):
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1615, in full_dispatch_request
return self.finalize_request(rv)
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1630, in finalize_request
response = self.make_response(rv)
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1740, in make_response
rv = self.response_class.force_type(rv, request.environ)
File "/home/pyspark/.local/lib/python3.5/site-packages/werkzeug/wrappers.py", line 921, in force_type
response = BaseResponse(*_run_wsgi_app(response, environ))
File "/home/pyspark/.local/lib/python3.5/site-packages/werkzeug/wrappers.py", line 59, in _run_wsgi_app
return _run_wsgi_app(*args)
File "/home/pyspark/.local/lib/python3.5/site-packages/werkzeug/test.py", line 923, in run_wsgi_app
app_rv = app(environ, start_response)
TypeError: 'RDD' object is not callable
错误（使用元组和模式）
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
18/02/10 12:56:47 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/02/10 12:56:48 WARN Utils: Your hostname, pyspark-VirtualBox resolves to a loopback address: 127.0.1.1; using 10.0.2.15 instead (on interface enp0s3)
18/02/10 12:56:48 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
18/02/10 12:56:51 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
### Tuple is [(800, 0, 0.3048, 71.3, 0.0026634)]
### schema --> struct<displ_thick_m:double,Chord_m:double,Freq_Hz:int,AoA_Deg:int,V_inf_mps:double>
### session --> <pyspark.sql.conf.RuntimeConfig object at 0x7efd4df9e860>
127.0.0.1 - - [10/Feb/2018 12:56:53] "GET /test HTTP/1.1" 500 -
Traceback (most recent call last):
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1997, in __call__
return self.wsgi_app(environ, start_response)
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1985, in wsgi_app
response = self.handle_exception(e)
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1540, in handle_exception
reraise(exc_type, exc_value, tb)
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/_compat.py", line 33, in reraise
raise value
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/home/pyspark/.local/lib/python3.5/site-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/home/pyspark/Desktop/building_py_rec/lin_reg/server.py", line 48, in test
df = session.createDataFrame(tup, schema)
File "/home/pyspark/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/session.py", line 522, in createDataFrame
rdd, schema = self._createFromLocal(map(prepare, data), schema)
File "/home/pyspark/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/session.py", line 383, in _createFromLocal
data = list(data)
File "/home/pyspark/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/session.py", line 505, in prepare
verify_func(obj, schema)
File "/home/pyspark/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/types.py", line 1360, in _verify_type
_verify_type(v, f.dataType, f.nullable)
File "/home/pyspark/spark-2.1.0-bin-hadoop2.7/python/pyspark/sql/types.py", line 1324, in _verify_type
raise TypeError("%s can not accept object %r in type %s" % (dataType, obj, type(obj)))
TypeError: DoubleType can not accept object 800 in type <class 'int'>
在这里，我理解提供给createDataframe的值（元组中的值）与schema中的顺序不匹配的顺序。因此，TypeError。
相关代码
@app.route('/test')
def test():
# create spark Session
session = SparkSession.builder.appName('lin_reg_api').getOrCreate();
# Approach#1
tup = [(800,0,0.3048,71.3,0.0026634)]
cols = ["Freq_Hz", "AoA_Deg", "Chord_m", "V_inf_mps", "displ_thick_m"];
print(' ### Tuple is ', tup);
#Approach#2
schema = StructType({
StructField("Freq_Hz", IntegerType(), False),
StructField("AoA_Deg", IntegerType(), False),
StructField("Chord_m", DoubleType(), False),
StructField("V_inf_mps", DoubleType(), False),
StructField("displ_thick_m", DoubleType(), False),
});
print(' ### schema -->', schema.simpleString());
# session = linReg.getSession(); # returns the spark session
print(' ### session -->', session.conf);
# Approach 1
#df = session.createDataFrame(tup, cols)
# Approach 2
df = session.createDataFrame(tup, schema)
print(' ### data frame -->', df.toJSON())
return df.toJSON()
我想了解如何让这两种方法都适合我。
答案 0 :(得分：2)
在您发布的代码中但在应用初始化期间未发生第一个错误。使用您发布的代码无法重现。
第二个问题在例外情况中明确指出：
TypeError：DoubleType无法接受类型
中的对象800
这是因为您使用set（{...}）来定义架构，并且字段的顺序未定义。使用具有已定义顺序的序列，例如list：
schema = StructType([
StructField("Freq_Hz", IntegerType(), False),
StructField("AoA_Deg", IntegerType(), False),
StructField("Chord_m", DoubleType(), False),
StructField("V_inf_mps", DoubleType(), False),
StructField("displ_thick_m", DoubleType(), False),
])