Question

我有一个格式为timeStamp \t JSON的文件。我能够读取该文件并将其转换为RDD。

val jml = sc.textFile("/Users/varun.risbud/jenkinsMonitor")

def tf(x: String) = {
  val y = x.split("\\t")
  (y{0}, y{1})
}

tf: (x: String)(String, String) 

scala> val test = jml.map(tf)
test: org.apache.spark.rdd.RDD[(String, String)] = MapPartitionsRDD[137] at map at <console>:31

scala> val l = test.toDF
l: org.apache.spark.sql.DataFrame = [_1: string, _2: string]

scala> l.show(5)
+-------------+--------------------+
|           _1|                  _2|
+-------------+--------------------+
|1495828323032|{"items":[{"actio...|
|1495828382459|{"items":[{"actio...|
|1495828442891|{"items":[{"actio...|
|1495828504130|{"items":[{"actio...|
|1495828562754|{"items":[{"actio...|
+-------------+--------------------+
only showing top 5 rows

我想解析JSON中的_2并将JSON中的字段作为列。我能够解析JSON并将其转换为DataFrame而没有_1（timeStamp）列

val res1 = jml.map(l => l.split("\t"){1})

import spark.sqlContext.implicits._

val jm = spark.sqlContext.read.json(res1)
val flattenedJM = jm.withColumn("items", explode($"items"))
val i = flattenedJM.select($"items.*")
i.printSchema

root
|-- actions: array (nullable = true)
|    |-- element: struct (containsNull = true)
|    |    |-- causes: array (nullable = true)
|    |    |    |-- element: struct (containsNull = true)
|    |    |    |    |-- shortDescription: string (nullable = true)
|    |    |    |    |-- upstreamBuild: long (nullable = true)
|    |    |    |    |-- upstreamProject: string (nullable = true)
|    |    |    |    |-- upstreamUrl: string (nullable = true)
|    |    |    |    |-- userName: string (nullable = true)
|    |    |-- parameters: array (nullable = true)
|    |    |    |-- element: struct (containsNull = true)
|    |    |    |    |-- name: string (nullable = true)
|    |    |    |    |-- value: string (nullable = true)
|-- blocked: boolean (nullable = true)
|-- buildable: boolean (nullable = true)
|-- buildableStartMilliseconds: long (nullable = true)
|-- id: long (nullable = true)
|-- inQueueSince: long (nullable = true)
|-- params: string (nullable = true)
|-- stuck: boolean (nullable = true)
|-- task: struct (nullable = true)
|    |-- color: string (nullable = true)
|    |-- name: string (nullable = true)
|    |-- url: string (nullable = true)
|-- timestamp: long (nullable = true)
|-- why: string (nullable = true)

我不知道如何将JSON解析为DataFrame的timeStamp列。

修改请求l.show(1, false)的输出

_1 | _2

|1495828323032|{"items":[{"actions":[{"parameters":[{"name":"HDFS_PATH","value":"/user/reco/RecommendationEngineCommand/group-a/output/0000579-170518175227238-oozie-jett-W/formatted-recos/part-00000"},{"name":"BUILD","value":true},{"name":"FORMULA","value":"asset-collab-filter_group-a"}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6619,"inQueueSince":1495700652290,"params":"\n(StringParameterValue) HDFS_PATH='/user/reco/RecommendationEngineCommand/group-a/output/0000579-170518175227238-oozie-jett-W/formatted-recos/part-00000'\n(BooleanParameterValue) BUILD='true'\n(StringParameterValue) FORMULA='asset-collab-filter_group-a'","stuck":false,"task":{"name":"add-genre-based-tags","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/add-genre-based-tags/","color":"red"},"why":null,"buildableStartMilliseconds":1495700657291},{"actions":[{"parameters":[{"name":"HDFS_PATH","value":"/user/reco/TFIDFUserRecommendationCommand/output/NEWS_KEYWORD/20170525052044-0400-0.5952090573038591/part-*"},{"name":"BUILD","value":true},{"name":"FORMULA","value":""}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6635,"inQueueSince":1495706476146,"params":"\n(StringParameterValue) HDFS_PATH='/user/reco/TFIDFUserRecommendationCommand/output/NEWS_KEYWORD/20170525052044-0400-0.5952090573038591/part-*'\n(BooleanParameterValue) BUILD='true'\n(StringParameterValue) FORMULA=''","stuck":false,"task":{"name":"add-genre-based-tags","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/add-genre-based-tags/","color":"red"},"why":null,"buildableStartMilliseconds":1495706481148},{"actions":[{"parameters":[{"name":"HDFS_PATH","value":"/user/reco/RecommendationEngineCommand/group-c/output/0000600-170518175227238-oozie-jett-W/formatted-recos/part-00000"},{"name":"BUILD","value":true},{"name":"FORMULA","value":"asset-collab-filter_group-c"}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6646,"inQueueSince":1495712959073,"params":"\n(StringParameterValue) HDFS_PATH='/user/reco/RecommendationEngineCommand/group-c/output/0000600-170518175227238-oozie-jett-W/formatted-recos/part-00000'\n(BooleanParameterValue) BUILD='true'\n(StringParameterValue) FORMULA='asset-collab-filter_group-c'","stuck":false,"task":{"name":"add-genre-based-tags","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/add-genre-based-tags/","color":"red"},"why":null,"buildableStartMilliseconds":1495712964075},{"actions":[{"parameters":[{"name":"HDFS_PATH","value":"/user/reco/TFIDFUserRecommendationCommand/output/NEWS_KEYWORD/20170525091621-0400-0.41675916440498506/part-*"},{"name":"BUILD","value":true},{"name":"FORMULA","value":""}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6666,"inQueueSince":1495719019057,"params":"\n(StringParameterValue) HDFS_PATH='/user/reco/TFIDFUserRecommendationCommand/output/NEWS_KEYWORD/20170525091621-0400-0.41675916440498506/part-*'\n(BooleanParameterValue) BUILD='true'\n(StringParameterValue) FORMULA=''","stuck":false,"task":{"name":"add-genre-based-tags","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/add-genre-based-tags/","color":"red"},"why":null,"buildableStartMilliseconds":1495719024058},{"actions":[{"parameters":[{"name":"HDFS_PATH","value":"/user/reco/RecommendationEngineCommand/group-b/output/0000618-170518175227238-oozie-jett-W/formatted-recos/part-00000"},{"name":"BUILD","value":true},{"name":"FORMULA","value":"asset-collab-filter_group-b"}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6674,"inQueueSince":1495720890472,"params":"\n(StringParameterValue) HDFS_PATH='/user/reco/RecommendationEngineCommand/group-b/output/0000618-170518175227238-oozie-jett-W/formatted-recos/part-00000'\n(BooleanParameterValue) BUILD='true'\n(StringParameterValue) FORMULA='asset-collab-filter_group-b'","stuck":false,"task":{"name":"add-genre-based-tags","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/add-genre-based-tags/","color":"red"},"why":null,"buildableStartMilliseconds":1495720895474},{"actions":[{"parameters":[{"name":"HDFS_PATH","value":"/user/reco/RecommendationEngineCommand/group-d/output/0000638-170518175227238-oozie-jett-W/formatted-recos/part-00000"},{"name":"BUILD","value":true},{"name":"FORMULA","value":"asset-collab-filter_group-d"}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6698,"inQueueSince":1495730414670,"params":"\n(StringParameterValue) HDFS_PATH='/user/reco/RecommendationEngineCommand/group-d/output/0000638-170518175227238-oozie-jett-W/formatted-recos/part-00000'\n(BooleanParameterValue) BUILD='true'\n(StringParameterValue) FORMULA='asset-collab-filter_group-d'","stuck":false,"task":{"name":"add-genre-based-tags","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/add-genre-based-tags/","color":"red"},"why":null,"buildableStartMilliseconds":1495730419672},{"actions":[{"parameters":[{"name":"HDFS_PATH","value":"/user/reco/TFIDFUserRecommendationCommand/output/NEWS_KEYWORD/20170525130057-0400-0.6889772107486807/part-*"},{"name":"BUILD","value":true},{"name":"FORMULA","value":""}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6707,"inQueueSince":1495732273684,"params":"\n(StringParameterValue) HDFS_PATH='/user/reco/TFIDFUserRecommendationCommand/output/NEWS_KEYWORD/20170525130057-0400-0.6889772107486807/part-*'\n(BooleanParameterValue) BUILD='true'\n(StringParameterValue) FORMULA=''","stuck":false,"task":{"name":"add-genre-based-tags","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/add-genre-based-tags/","color":"red"},"why":null,"buildableStartMilliseconds":1495732278686},{"actions":[{"parameters":[{"name":"HDFS_PATH","value":"/user/reco/RecommendationEngineCommand/group-a/output/0000657-170518175227238-oozie-jett-W/formatted-recos/part-00000"},{"name":"BUILD","value":true},{"name":"FORMULA","value":"asset-collab-filter_group-a"}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6721,"inQueueSince":1495739172918,"params":"\n(StringParameterValue) HDFS_PATH='/user/reco/RecommendationEngineCommand/group-a/output/0000657-170518175227238-oozie-jett-W/formatted-recos/part-00000'\n(BooleanParameterValue) BUILD='true'\n(StringParameterValue) FORMULA='asset-collab-filter_group-a'","stuck":false,"task":{"name":"add-genre-based-tags","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/add-genre-based-tags/","color":"red"},"why":null,"buildableStartMilliseconds":1495739177920},{"actions":[{"parameters":[{"name":"HDFS_PATH","value":"/user/reco/RecommendationEngineCommand/group-b/output/0000674-170518175227238-oozie-jett-W/formatted-recos/part-00000"},{"name":"BUILD","value":true},{"name":"FORMULA","value":"asset-collab-filter_group-b"}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6741,"inQueueSince":1495747040074,"params":"\n(StringParameterValue) HDFS_PATH='/user/reco/RecommendationEngineCommand/group-b/output/0000674-170518175227238-oozie-jett-W/formatted-recos/part-00000'\n(BooleanParameterValue) BUILD='true'\n(StringParameterValue) FORMULA='asset-collab-filter_group-b'","stuck":false,"task":{"name":"add-genre-based-tags","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/add-genre-based-tags/","color":"red"},"why":null,"buildableStartMilliseconds":1495747045076},{"actions":[{"parameters":[{"name":"GROUP","value":"group-a"}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6744,"inQueueSince":1495747228124,"params":"\n(StringParameterValue) GROUP='group-a'","stuck":false,"task":{"name":"recommendation-engine","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/recommendation-engine/","color":"red"},"why":null,"buildableStartMilliseconds":1495747238049},{"actions":[{"parameters":[{"name":"GROUP","value":"group-b"}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6745,"inQueueSince":1495747228131,"params":"\n(StringParameterValue) GROUP='group-b'","stuck":false,"task":{"name":"recommendation-engine","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/recommendation-engine/","color":"red"},"why":null,"buildableStartMilliseconds":1495747238049},{"actions":[{"parameters":[{"name":"HDFS_PATH","value":"/user/reco/TFIDFUserRecommendationCommand/output/NEWS_KEYWORD/20170525172108-0400-0.39591519476120673/part-*"},{"name":"BUILD","value":true},{"name":"FORMULA","value":""}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6748,"inQueueSince":1495748215643,"params":"\n(StringParameterValue) HDFS_PATH='/user/reco/TFIDFUserRecommendationCommand/output/NEWS_KEYWORD/20170525172108-0400-0.39591519476120673/part-*'\n(BooleanParameterValue) BUILD='true'\n(StringParameterValue) FORMULA=''","stuck":false,"task":{"name":"add-genre-based-tags","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/add-genre-based-tags/","color":"red"},"why":null,"buildableStartMilliseconds":1495748220645},{"actions":[{"causes":[{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"}]}],"blocked":true,"buildable":false,"id":6776,"inQueueSince":1495825239472,"params":"","stuck":false,"task":{"name":"extract-attribute-events","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/extract-attribute-events/","color":"blue_anime"},"why":"Build #20,208 is already in progress (ETA:N/A)","buildableStartMilliseconds":1495825239473},{"actions":[{"parameters":[{"name":"LISTABLE","value":"asset"},{"name":"CATEGORIES","value":"article video"},{"name":"JOB_TYPE","value":"articleVideos"}]},{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":true,"buildable":false,"id":6779,"inQueueSince":1495825839474,"params":"\n(StringParameterValue) LISTABLE='asset'\n(StringParameterValue) CATEGORIES='article video'\n(StringParameterValue) JOB_TYPE='articleVideos'","stuck":false,"task":{"name":"content-based-similar-asset-video-job","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/content-based-similar-asset-video-job/","color":"red_anime"},"why":"Build #96 is already in progress (ETA:N/A)","buildableStartMilliseconds":1495825839578},{"actions":[{"parameters":[{"name":"GROUP","value":"ALL_CLIENTS"},{"name":"BUCKET","value":"popular-series-movies"}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":true,"buildable":false,"id":6783,"inQueueSince":1495827688771,"params":"\n(StringParameterValue) GROUP='ALL_CLIENTS'\n(StringParameterValue) BUCKET='popular-series-movies'","stuck":false,"task":{"name":"mostpopular-job","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/mostpopular-job/","color":"red"},"why":null,"buildableStartMilliseconds":1495827693773},{"actions":[{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6777,"inQueueSince":1495825239472,"params":"","stuck":false,"task":{"name":"mostpopular-event-aggregator","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/mostpopular-event-aggregator/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495825239473},{"actions":[{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6782,"inQueueSince":1495827639474,"params":"","stuck":true,"task":{"name":"event-propagator","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/event-propagator/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495827639475},{"actions":[{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6781,"inQueueSince":1495827039473,"params":"","stuck":false,"task":{"name":"retrieve-events-from-kafka","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/retrieve-events-from-kafka/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495827039576},{"actions":[{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6780,"inQueueSince":1495826739474,"params":"","stuck":false,"task":{"name":"asset-data-retriever","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/asset-data-retriever/","color":"red"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495826739475},{"actions":[{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6778,"inQueueSince":1495825239472,"params":"","stuck":true,"task":{"name":"report-hdfs-stats","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/report-hdfs-stats/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495825239473},{"actions":[{"parameters":[{"name":"CLUSTER","value":"*"},{"name":"DATE_TIME","value":""},{"name":"DATACENTER","value":"*"}]},{"causes":[{"shortDescription":"Started by upstream project \"retrieve-events-from-kafka\" build number 16,536","upstreamBuild":16536,"upstreamProject":"retrieve-events-from-kafka","upstreamUrl":"job/retrieve-events-from-kafka/"}]}],"blocked":false,"buildable":true,"id":6775,"inQueueSince":1495825006185,"params":"\n(StringParameterValue) CLUSTER='*'\n(StringParameterValue) DATE_TIME=''\n(StringParameterValue) DATACENTER='*'","stuck":true,"task":{"name":"event-copier","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/event-copier/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495825011188},{"actions":[{"parameters":[{"name":"LISTABLE","value":"content series"},{"name":"TAG","value":"movie"},{"name":"JOB_TYPE","value":"movieSeries"}]},{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6767,"inQueueSince":1495789239475,"params":"\n(StringParameterValue) LISTABLE='content series'\n(StringParameterValue) TAG='movie'\n(StringParameterValue) JOB_TYPE='movieSeries'","stuck":false,"task":{"name":"content-based-similar-movie-series-job","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/content-based-similar-movie-series-job/","color":"red"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495789239476},{"actions":[{"parameters":[{"name":"HDFS_PATH","value":"/user/reco/TFIDFUserRecommendationCommand/output/NEWS_KEYWORD/20170525012316-0400-0.12471714652202148/part-*"},{"name":"BUILD","value":true},{"name":"FORMULA","value":""}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":false,"buildable":true,"id":6594,"inQueueSince":1495691563700,"params":"\n(StringParameterValue) HDFS_PATH='/user/reco/TFIDFUserRecommendationCommand/output/NEWS_KEYWORD/20170525012316-0400-0.12471714652202148/part-*'\n(BooleanParameterValue) BUILD='true'\n(StringParameterValue) FORMULA=''","stuck":true,"task":{"name":"add-genre-based-tags","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/add-genre-based-tags/","color":"red"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495691568702},{"actions":[{"parameters":[{"name":"GROUP","value":"ALL_CLIENTS"},{"name":"BUCKET","value":"popular-one-day-article-clip"}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"},{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":false,"buildable":true,"id":6750,"inQueueSince":1495748774477,"params":"\n(StringParameterValue) GROUP='ALL_CLIENTS'\n(StringParameterValue) BUCKET='popular-one-day-article-clip'","stuck":true,"task":{"name":"mostpopular-job","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/mostpopular-job/","color":"red"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495748779479},{"actions":[{"parameters":[{"name":"GROUP","value":"group-d"}]},{"causes":[{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"},{"shortDescription":"Started by user anonymous","userId":null,"userName":"anonymous"}]}],"blocked":false,"buildable":true,"id":6702,"inQueueSince":1495731636544,"params":"\n(StringParameterValue) GROUP='group-d'","stuck":true,"task":{"name":"recommendation-engine","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/recommendation-engine/","color":"red"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495731642623},{"actions":[{"parameters":[{"name":"REDUCERS","value":"20"},{"name":"TTL","value":"180000"}]},{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6774,"inQueueSince":1495821639474,"params":"\n(StringParameterValue) REDUCERS='20'\n(StringParameterValue) TTL='180000'","stuck":false,"task":{"name":"recommendation-score-news-genres","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/recommendation-score-news-genres/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495821639475},{"actions":[{"parameters":[{"name":"REDUCERS","value":"20"},{"name":"TTL","value":"180000"}]},{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6773,"inQueueSince":1495807239474,"params":"\n(StringParameterValue) REDUCERS='20'\n(StringParameterValue) TTL='180000'","stuck":true,"task":{"name":"recommendation-score-news-keywords","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/recommendation-score-news-keywords/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495807239578},{"actions":[{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6772,"inQueueSince":1495805019474,"params":"","stuck":true,"task":{"name":"recommendation-engine-report","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/recommendation-engine-report/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495805019578},{"actions":[{"parameters":[{"name":"DAY_OFFSET","value":"1"}]},{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6771,"inQueueSince":1495800039475,"params":"\n(StringParameterValue) DAY_OFFSET='1'","stuck":true,"task":{"name":"recommendation-event-analytics","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/recommendation-event-analytics/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495800039580},{"actions":[{"parameters":[{"name":"INPUT_DATE","value":"1 day ago"},{"name":"OVERWRITE","value":""}]},{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6770,"inQueueSince":1495800039474,"params":"\n(StringParameterValue) INPUT_DATE='1 day ago'\n(StringParameterValue) OVERWRITE=''","stuck":true,"task":{"name":"analytics-cohort-daily-summary","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/analytics-cohort-daily-summary/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495800039475},{"actions":[{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6769,"inQueueSince":1495792839473,"params":"","stuck":true,"task":{"name":"maint-log-remover","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/maint-log-remover/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495792839577},{"actions":[{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6768,"inQueueSince":1495791039473,"params":"","stuck":true,"task":{"name":"retrieve-latest-user-settings","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/retrieve-latest-user-settings/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495791039474},{"actions":[{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6766,"inQueueSince":1495782039475,"params":"","stuck":true,"task":{"name":"maint-file-merger","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/maint-file-merger/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495782039475},{"actions":[{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6765,"inQueueSince":1495774839473,"params":"","stuck":true,"task":{"name":"recommendation-directory-cleanup","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/recommendation-directory-cleanup/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495774839474},{"actions":[{"parameters":[{"name":"daysToCheck","value":"90"},{"name":"daysToKeepAsHours","value":"1"},{"name":"monthsToKeepAsDays","value":"1"},{"name":"allowLargerOutput","value":""},{"name":"dateMatchesArg","value":""},{"name":"includeAllFilesArg","value":""}]},{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6764,"inQueueSince":1495773939472,"params":"\n(StringParameterValue) daysToCheck='90'\n(StringParameterValue) daysToKeepAsHours='1'\n(StringParameterValue) monthsToKeepAsDays='1'\n(StringParameterValue) allowLargerOutput=''\n(StringParameterValue) dateMatchesArg=''\n(StringParameterValue) includeAllFilesArg=''","stuck":true,"task":{"name":"event-file-consolidator","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/event-file-consolidator/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495773939473},{"actions":[{"causes":[{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6763,"inQueueSince":1495771239475,"params":"","stuck":true,"task":{"name":"mostpopular-events-cleanup","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/mostpopular-events-cleanup/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495771239578},{"actions":[{"causes":[{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6762,"inQueueSince":1495757739473,"params":"","stuck":true,"task":{"name":"local-recommendation-job","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/local-recommendation-job/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495757739576},{"actions":[{"parameters":[{"name":"RECOMMENDABLE_TYPE","value":"NEWS_KEYWORD"}]},{"causes":[{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6761,"inQueueSince":1495756839474,"params":"\n(StringParameterValue) RECOMMENDABLE_TYPE='NEWS_KEYWORD'","stuck":true,"task":{"name":"score-assets-for-users","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/score-assets-for-users/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495756839577},{"actions":[{"causes":[{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6760,"inQueueSince":1495756839473,"params":"","stuck":true,"task":{"name":"recommendation-engine-preparer","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/recommendation-engine-preparer/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495756839474},{"actions":[{"parameters":[{"name":"DEMO_CLIENTS","value":"showcase,hometown-cable,zonealarm,reinbeck,attwifi,hometelecom,myclick2go,zonealarm,cookieshield,hbc,longlines,hotwire"},{"name":"UNIQUE_CLIENTS","value":"toshiba-es"},{"name":"PROXY","value":"172.17.7.152"},{"name":"RANDOMIZE","value":"--random"}]},{"causes":[{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6759,"inQueueSince":1495756839473,"params":"\n(StringParameterValue) DEMO_CLIENTS='showcase,hometown-cable,zonealarm,reinbeck,attwifi,hometelecom,myclick2go,zonealarm,cookieshield,hbc,longlines,hotwire'\n(StringParameterValue) UNIQUE_CLIENTS='toshiba-es'\n(StringParameterValue) PROXY='172.17.7.152'\n(StringParameterValue) RANDOMIZE='--random'","stuck":true,"task":{"name":"COPYDOWN","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/COPYDOWN/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495756839474},{"actions":[{"causes":[{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"},{"shortDescription":"Started by timer"}]}],"blocked":false,"buildable":true,"id":6758,"inQueueSince":1495752339472,"params":"","stuck":true,"task":{"name":"retrieve-vam-asset-metadata-local","url":"http://hadoop-oozie01.star.dev.opal.synacor.com:8080/jenkins/job/retrieve-vam-asset-metadata-local/","color":"blue"},"why":"Waiting for next available executor","buildableStartMilliseconds":1495752339575}]}

Answer 1

你有这个方法来解析输入文件。

def tf(x: String) = {
  val y = x.split("\\t")
  (y{0}, y{1})
}

因此，操纵y{1}部分 - 在JSON的上下文中添加时间戳。

将y{1}转换为JsonObject实例
将时间戳 - Y{0}作为json元素添加到json实例
返回(y{0}, json-instance.tostring)

这将解决您的问题。我在这里错过了什么吗？

Answer 2

您可以将时间戳添加到json本身，然后将其转换为数据帧。

SPARK：如何将DataFrame中的字符串字段转换为JSON

2 个答案: