Question

以下各项的正确PySpark模式是什么？第一个返回是一个列表，我已经正确编写了代码，但是第二个返回是一个元组列表，我不确定如何在一个模式中编写其对应的Type。

df = spark.createDataFrame([("Alive", 4)], ["Name", "Number"])


def example(n):
    return Row('Out1', 'Out2')([n + 2, n+4], [(n, n+2), (n, n+4)])


schema = StructType([
    StructField("Out1", ArrayType(IntegerType()), False),
    StructField("Out2", ArrayType(IntegerType(), False), True)])

example_udf = F.udf(example, schema)

newDF = df.withColumn("Output", F.explode(F.array(example_udf(df["Number"]))))
newDF = newDF.select("Name", "Number", "Output.*")

非常感谢！

Answer 1

我们可以将第二个返回类型用作array(array(integer))，

df = spark.createDataFrame([("Alive", 4)], ["Name", "Number"])
def example(n):
    return ([n + 2, n+4], [(n, n+2), (n, n+4)])

schema =StructType([StructField('out1',ArrayType(IntegerType())),StructField('out2',ArrayType(ArrayType(IntegerType())))])

example_udf = F.udf(example, schema)

df.select(example_udf(df.number).alias('s_col')).select('s_col.out1','s_col.out2').show()
+------+--------------+
| out1 |  out2        |
+---+-----------------+
|[6,8] |[[4,6],[4,8]] |
+------+--------------+

Answer 2

感谢@Suresh！我的问题的另一个答案是：

E/AndroidRuntime: FATAL EXCEPTION: main
    Process: com.example.javaproject, PID: 12835
    java.lang.RuntimeException: Unable to start activity ComponentInfo{com.example.javaproject/com.example.javaproject.LondonLevel1}: java.lang.IllegalStateException: congrats_layout must not be null
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3271)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3410)
        at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:83)
        at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135)
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95)
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2017)
        at android.os.Handler.dispatchMessage(Handler.java:107)
        at android.os.Looper.loop(Looper.java:214)
        at android.app.ActivityThread.main(ActivityThread.java:7397)
        at java.lang.reflect.Method.invoke(Native Method)
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492)
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:935)
     Caused by: java.lang.IllegalStateException: congrats_layout must not be null
        at com.example.javaproject.LondonLevel1.onCreate(LondonLevel1.kt:36)
        at android.app.Activity.performCreate(Activity.java:7802)
        at android.app.Activity.performCreate(Activity.java:7791)
        at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1300)
        at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:3246)
        at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:3410) 
        at android.app.servertransaction.LaunchActivityItem.execute(LaunchActivityItem.java:83) 
        at android.app.servertransaction.TransactionExecutor.executeCallbacks(TransactionExecutor.java:135) 
        at android.app.servertransaction.TransactionExecutor.execute(TransactionExecutor.java:95) 
        at android.app.ActivityThread$H.handleMessage(ActivityThread.java:2017) 
        at android.os.Handler.dispatchMessage(Handler.java:107) 
        at android.os.Looper.loop(Looper.java:214) 
        at android.app.ActivityThread.main(ActivityThread.java:7397) 
        at java.lang.reflect.Method.invoke(Native Method) 
        at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:492) 
        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:935) 
I/Process: Sending signal. PID: 12835 SIG: 9

元组PySpark模式列表

2 个答案: