需要将json文件加载到spark中

时间:2018-02-13 11:11:48

标签: json scala apache-spark

对Spark和Scala来说,我有一些场景需要将下面的文件传递给Spark sql上下文,我不知道如何将这个文件处理成spark。

如果有人帮助我会非常有帮助

{
 "source": [
{
  "id": "source1",
  "type": "csv",
  "path": "/home/siva/f1.txt",
  "delimiter": ",",
  "tableName": "table1",
  "schema": "f1,f2,f3,f4,f5"
}
],
"pipe": [
{
  "id": "pipe1",
  "sql": "select f1,f2,f5 from table1"
}
],
"sinks": [
{
  "id": "output1",
  "type": "hdfs",
  "path": "/home/siva/out",
  "format": "json"
},

]
}

请帮我继续。

先谢谢

0 个答案:

没有答案