when reading json files into a DataFrame, this way:
val frame = sqlContext.read.json("path/ts=100/somefile.json")
in the DataFrame I receive, the ts values are all equal "100", even though there are different "ts" values in the file.
the weird thing is that if i use : sqlContext.read.json("path/ts=100")
I get all files into the frame, and the "ts" value is not overridden.
any idea how to work through it?
examples:
values in data frame:
ts : 100
values in json file:
{"ts" : "5002342004"}
Thanks.