Question

对于火花结构化流式传输读取过程：

2019-04-24T05:37:09.064416+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/" host=young-thicket-75018.herokuapp.com request_id=8d2ebd1c-0e87-4048-aa63-667672ad11a2 fwd="129.128.60.85" dyno= connect= service= status=503 bytes= protocol=https

Error: Cannot find module '../wireprotocol/constants' at Function.Module._resolveFilename (internal/modules/cjs/loader.js:581:15) at Function.Module._load (internal/modules/cjs/loader.js:507:25) at Module.require (internal/modules/cjs/loader.js:637:17) at require (internal/modules/cjs/helpers.js:22:18) at Object.<anonymous> (/app/node_modules/mongodb-core/lib/sdam/topology_description.js:5:24) at Module._compile (internal/modules/cjs/loader.js:689:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:700:10) at Module.load (internal/modules/cjs/loader.js:599:32) at tryModuleLoad (internal/modules/cjs/loader.js:538:12) at Function.Module._load (internal/modules/cjs/loader.js:530:3) npm ERR! code ELIFECYCLE npm ERR! errno 1 npm ERR! todolist-v1@1.0.0 start: `node app.js` npm ERR! Exit status 1 npm ERR! npm ERR! Failed at the todolist-v1@1.0.0 start script. npm ERR! This is probably not a problem with npm. There is likely additional logging output above.正在正确写入其输出，如下所示：

sdf.writeStream
  .outputMode(outputMode)
  .format("console")
  .trigger(Trigger.ProcessingTime("2 seconds"))
  .start())

但是，这只是结果的一部分。通过format(console)设置是否与Batch: 3 +----------+------+-------+-----------------+ |OnTimeRank|Origin|Carrier| OnTimePct| +----------+------+-------+-----------------+ | 1| BWI| EV| 90.0| | 2| BWI| US|88.54072251715655| | 3| BWI| CO|88.52097130242826| | 4| BWI| YV| 87.2168284789644| | 5| BWI| DL|86.21888471700737| | 6| BWI| NW|86.04866030181707| | 7| BWI| 9E|85.83545377438507| | 8| BWI| AA|85.71428571428571| | 9| BWI| FL|83.25366684127816| | 10| BWI| UA|81.32427843803056| | 1| CMI| MQ|81.92159607980399| | 1| IAH| NW| 91.6242895602752| | 2| IAH| F9|88.62350722815839| | 3| IAH| US|87.54764930114358| | 4| IAH| 9E|84.33613445378151| | 5| IAH| OO| 84.2836946277097| | 6| IAH| DL|83.46420323325636| | 7| IAH| UA|83.40671436433682| | 8| IAH| XE|81.35189010909355| | 9| IAH| OH|80.61558611656844| +----------+------+-------+-----------------+等效-沿dataframe.show(NumRows, truncate)的行：

option

Answer 1

该选项称为numRows，例如.option("numRows",1000)

来源https://github.com/apache/spark/blob/2a80a4cd39c7bcee44b6f6432769ca9fdba137e4/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/ConsoleWrite.scala#L33

使用控制台输出格式显示Spark Streaming Batch的完整结果

1 个答案: