使用typedcolumn选择Spark数据集

时间:2016-07-28 16:39:50

标签: scala apache-spark apache-spark-dataset

查看spark DataSet上的select()函数,有各种生成的函数签名:

(c1: TypedColumn[MyClass, U1],c2: TypedColumn[MyClass, U2] ....)

这似乎暗示我应该能够直接引用MyClass的成员并且类型安全,但我不确定如何......

ds.select("member")当然有效..似乎ds.select(_.member)也可能以某种方式起作用?

1 个答案:

答案 0 :(得分:14)

iOS/Pods/TwilioCommon/TwilioCommon.framework/TwilioCommon, file was built for unsupported file format ( 0x76 0x65 0x72 0x73 0x69 0x6F 0x6E 0x20 0x68 0x74 0x74 0x70 0x73 0x3A 0x2F 0x2F ) which is not the architecture being linked (x86_64): /Users/somendra/Documents/iOS/Pods/TwilioCommon/TwilioCommon.framework/TwilioCommon iOS/Pods/TwilioConversationsClient/TwilioConversationsClient.framework/TwilioConversationsClient, file was built for unsupported file format ( 0x76 0x65 0x72 0x73 0x69 0x6F 0x6E 0x20 0x68 0x74 0x74 0x70 0x73 0x3A 0x2F 0x2F ) which is not the architecture being linked (x86_64): /Users/somendra/Documents/iOS/Pods/TwilioConversationsClient/TwilioConversationsClient.framework/TwilioConversationsClient Undefined symbols for architecture x86_64: "_OBJC_CLASS_$_TWCCameraCapturer", referenced from: type metadata accessor for __ObjC.TWCCameraCapturer in TwilioManager.o "_OBJC_CLASS_$_TWCConversation", referenced from: type metadata accessor for __ObjC.TWCConversation in TwilioManager.o "_OBJC_CLASS_$_TWCIncomingInvite", referenced from: type metadata accessor for __ObjC.TWCIncomingInvite in TwilioManager.o "_OBJC_CLASS_$_TWCLocalMedia", referenced from: type metadata accessor for __ObjC.TWCLocalMedia in TwilioManager.o "_OBJC_CLASS_$_TWCOutgoingInvite", referenced from: type metadata accessor for __ObjC.TWCOutgoingInvite in TwilioManager.o "_OBJC_CLASS_$_TwilioAccessManager", referenced from: type metadata accessor for __ObjC.TwilioAccessManager in TwilioManager.o "_OBJC_CLASS_$_TwilioConversationsClient", referenced from: type metadata accessor for __ObjC.TwilioConversationsClient in TwilioManager.o ld: symbol(s) not found for architecture x86_64 clang: error: linker command failed with exit code 1 (use -v to see invocation) 的Scala DSL中,有很多方法可以识别select

  • 来自符号:Column
  • 来自字符串:'name$"name"
  • 来自表达式:col(name)

要从expr("nvl(name, 'unknown') as renamed")获得TypedColumn,您只需使用Column

例如:myCol.as[T]