sqlcontext.read.json produces a DataFrame with values from a directory in its path

时间:2018-12-03 12:54:41

标签: scala apache-spark

when reading json files into a DataFrame, this way:

val frame = sqlContext.read.json("path/ts=100/somefile.json")

in the DataFrame I receive, the ts values are all equal "100", even though there are different "ts" values in the file.

the weird thing is that if i use : sqlContext.read.json("path/ts=100") I get all files into the frame, and the "ts" value is not overridden. any idea how to work through it?

examples:

values in data frame:

ts : 100

values in json file:

{"ts" : "5002342004"}

Thanks.

0 个答案:

没有答案