Twitter流媒体的Spark工作没有提供任何输出

时间：2016-11-20 11:29:01

标签： apache-spark twitter spark-streaming twitter-streaming-api

我正在尝试使用以下代码运行带有spark作为master的twitter流媒体作业，而不是获得任何输出。工作暂停了很长时间。有人可以帮助解决问题。该程序在本地模式下运行完美。纱线作为主人，给我我想要的确切输出。但在火花大师中，它表现不尽如人意。

bin/spark-submit –class org.test.scala.twitterAnalysis –master spark://localhost.localdomain:7077 –jars <dependency jarlocation> jar name

点击命令

[edureka@localhost spark-1.5.2]$ bin/spark-submit --class org.test.scala.twitterAnalysis --master spark://localhost.localdomain:7077 --jars file:///home/edureka/TransferredJars/twitter-0.0.1-SNAPSHOT-jar-with-dependencies.jar  /home/edureka/TransferredJars/twitter-0.0.1-SNAPSHOT.jar #apple #spark #currency #iphone
16/11/20 15:48:31 INFO spark.SparkContext: Running Spark version 1.5.2
16/11/20 15:48:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/11/20 15:48:32 WARN util.Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.0.107 instead (on interface eth3)
16/11/20 15:48:32 WARN util.Utils: Set SPARK_LOCAL_IP if you need to bind to another address
16/11/20 15:48:32 INFO spark.SecurityManager: Changing view acls to: edureka
16/11/20 15:48:32 INFO spark.SecurityManager: Changing modify acls to: edureka
16/11/20 15:48:32 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(edureka); users with modify permissions: Set(edureka)
16/11/20 15:48:34 INFO slf4j.Slf4jLogger: Slf4jLogger started
16/11/20 15:48:34 INFO Remoting: Starting remoting
16/11/20 15:48:35 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.0.107:38060]
16/11/20 15:48:35 INFO util.Utils: Successfully started service 'sparkDriver' on port 38060.
16/11/20 15:48:35 INFO spark.SparkEnv: Registering MapOutputTracker
16/11/20 15:48:35 INFO spark.SparkEnv: Registering BlockManagerMaster
16/11/20 15:48:35 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-521ff84c-473d-4c91-b675-933a02fc6cf4
16/11/20 15:48:35 INFO storage.MemoryStore: MemoryStore started with capacity 534.5 MB
16/11/20 15:48:35 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-1e434766-f32a-4608-92a0-2207b4a52727/httpd-29639e5c-b536-432a-888a-396eb60a60cb
16/11/20 15:48:35 INFO spark.HttpServer: Starting HTTP Server
16/11/20 15:48:35 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/11/20 15:48:36 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:56724
16/11/20 15:48:36 INFO util.Utils: Successfully started service 'HTTP file server' on port 56724.
16/11/20 15:48:36 INFO spark.SparkEnv: Registering OutputCommitCoordinator
16/11/20 15:48:36 INFO server.Server: jetty-8.y.z-SNAPSHOT
16/11/20 15:48:36 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040
16/11/20 15:48:36 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
16/11/20 15:48:36 INFO ui.SparkUI: Started SparkUI at http://192.168.0.107:4040
16/11/20 15:48:44 INFO spark.SparkContext: Added JAR file:///home/edureka/TransferredJars/twitter-0.0.1-SNAPSHOT-jar-with-dependencies.jar at http://192.168.0.107:56724/jars/twitter-0.0.1-SNAPSHOT-jar-with-dependencies.jar with timestamp 1479637124496
16/11/20 15:48:44 INFO spark.SparkContext: Added JAR file:/home/edureka/TransferredJars/twitter-0.0.1-SNAPSHOT.jar at http://192.168.0.107:56724/jars/twitter-0.0.1-SNAPSHOT.jar with timestamp 1479637124520
16/11/20 15:48:44 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.
16/11/20 15:48:44 INFO client.AppClient$ClientEndpoint: Connecting to master spark://localhost.localdomain:7077...
16/11/20 15:48:45 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20161120154845-0000
16/11/20 15:48:45 INFO client.AppClient$ClientEndpoint: Executor added: app-20161120154845-0000/0 on worker-20161120154741-192.168.0.107-42431 (192.168.0.107:42431) with 1 cores
16/11/20 15:48:45 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20161120154845-0000/0 on hostPort 192.168.0.107:42431 with 1 cores, 1024.0 MB RAM
16/11/20 15:48:45 INFO client.AppClient$ClientEndpoint: Executor updated: app-20161120154845-0000/0 is now RUNNING
16/11/20 15:48:46 INFO client.AppClient$ClientEndpoint: Executor updated: app-20161120154845-0000/0 is now LOADING
16/11/20 15:48:47 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 38621.
16/11/20 15:48:47 INFO netty.NettyBlockTransferService: Server created on 38621
16/11/20 15:48:47 INFO storage.BlockManagerMaster: Trying to register BlockManager
16/11/20 15:48:47 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.0.107:38621 with 534.5 MB RAM, BlockManagerId(driver, 192.168.0.107, 38621)
16/11/20 15:48:47 INFO storage.BlockManagerMaster: Registered BlockManager
16/11/20 15:48:48 INFO cluster.SparkDeploySchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
here are the elements of HashTags
16/11/20 15:48:52 WARN util.SizeEstimator: Failed to check whether UseCompressedOops is set; assuming yes

工作正在抓紧，而不是给我任何输出。  下面是终端输出和STDERR日志

终端输出

16/11/20 15:52:36 INFO receiver.BlockGenerator: Pushed block input-0-1479637356400
16/11/20 15:52:37 INFO storage.MemoryStore: ensureFreeSpace(15589) called with curMem=12488978, maxMem=560497950
16/11/20 15:52:37 INFO storage.MemoryStore: Block input-0-1479637356800 stored as bytes in memory (estimated size 15.2 KB, free 522.6 MB)
16/11/20 15:52:37 WARN storage.BlockManager: Block input-0-1479637356800 replicated to only 0 peer(s) instead of 1 peers
16/11/20 15:52:37 INFO receiver.BlockGenerator: Pushed block input-0-1479637356800
16/11/20 15:52:37 INFO storage.MemoryStore: ensureFreeSpace(7069) called with curMem=12504567, maxMem=560497950
16/11/20 15:52:37 INFO storage.MemoryStore: Block input-0-1479637357000 stored as bytes in memory (estimated size 6.9 KB, free 522.6 MB)
16/11/20 15:52:37 WARN storage.BlockManager: Block input-0-1479637357000 replicated to only 0 peer(s) instead of 1 peers
16/11/20 15:52:37 INFO receiver.BlockGenerator: Pushed block input-0-1479637357000
16/11/20 15:52:37 INFO storage.MemoryStore: ensureFreeSpace(47183) called with curMem=12511636, maxMem=560497950
16/11/20 15:52:37 INFO storage.MemoryStore: Block input-0-1479637357200 stored as bytes in memory (estimated size 46.1 KB, free 522.6 MB)
16/11/20 15:52:37 WARN storage.BlockManager: Block input-0-1479637357200 replicated to only 0 peer(s) instead of 1 peers
16/11/20 15:52:37 INFO receiver.BlockGenerator: Pushed block input-0-1479637357200
16/11/20 15:52:37 INFO storage.MemoryStore: ensureFreeSpace(3424) called with curMem=12558819, maxMem=560497950
16/11/20 15:52:37 INFO storage.MemoryStore: Block input-0-1479637357400 stored as bytes in memory (estimated size 3.3 KB, free 522.6 MB)
16/11/20 15:52:37 WARN storage.BlockManager: Block input-0-1479637357400 replicated to only 0 peer(s) instead of 1 peers
16/11/20 15:52:37 INFO receiver.BlockGenerator: Pushed block input-0-1479637357400
16/11/20 15:52:37 INFO storage.MemoryStore: ensureFreeSpace(4630) called with curMem=12562243, maxMem=560497950
16/11/20 15:52:37 INFO storage.MemoryStore: Block input-0-1479637357600 stored as bytes in memory (estimated size 4.5 KB, free 522.5 MB)
16/11/20 15:52:37 WARN storage.BlockManager: Block input-0-1479637357600 replicated to only 0 peer(s) instead of 1 peers
16/11/20 15:52:37 INFO receiver.BlockGenerator: Pushed block input-0-1479637357600
16/11/20 15:52:38 INFO storage.MemoryStore: ensureFreeSpace(10421) called with curMem=12566873, maxMem=560497950
16/11/20 15:52:38 INFO storage.MemoryStore: Block input-0-1479637357800 stored as bytes in memory (estimated size 10.2 KB, free 522.5 MB)
16/11/20 15:52:38 WARN storage.BlockManager: Block input-0-1479637357800 replicated to only 0 peer(s) instead of 1 peers
16/11/20 15:52:38 INFO receiver.BlockGenerator: Pushed block input-0-1479637357800
16/11/20 15:52:38 INFO storage.MemoryStore: ensureFreeSpace(44965) called with curMem=12577294, maxMem=560497950
16/11/20 15:52:38 INFO storage.MemoryStore: Block input-0-1479637358000 stored as bytes in memory (estimated size 43.9 KB, free 522.5 MB)
16/11/20 15:52:38 WARN storage.BlockManager: Block input-0-1479637358000 replicated to only 0 peer(s) instead of 1 peers
16/11/20 15:52:38 INFO receiver.BlockGenerator: Pushed block input-0-1479637358000
16/11/20 15:52:38 INFO storage.MemoryStore: ensureFreeSpace(6376) called with curMem=12622259, maxMem=560497950
16/11/20 15:52:38 INFO storage.MemoryStore: Block input-0-1479637358200 stored as bytes in memory (estimated size 6.2 KB, free 522.5 MB)
16/11/20 15:52:38 WARN storage.BlockManager: Block input-0-1479637358200 replicated to only 0 peer(s) instead of 1 peers
16/11/20 15:52:38 INFO receiver.BlockGenerator: Pushed block input-0-1479637358200
16/11/20 15:52:39 INFO storage.MemoryStore: ensureFreeSpace(10971) called with curMem=12628635, maxMem=560497950
16/11/20 15:52:39 INFO storage.MemoryStore: Block input-0-1479637358800 stored as bytes in memory (estimated size 10.7 KB, free 522.5 MB)
16/11/20 15:52:39 WARN storage.BlockManager: Block input-0-1479637358800 replicated to only 0 peer(s) instead of 1 peers
16/11/20 15:52:39 INFO receiver.BlockGenerator: Pushed block input-0-1479637358800
16/11/20 15:52:39 INFO storage.MemoryStore: ensureFreeSpace(2507) called with curMem=12639606, maxMem=560497950
16/11/20 15:52:39 INFO storage.MemoryStore: Block input-0-1479637359000 stored as bytes in memory (estimated size 2.4 KB, free 522.5 MB)
16/11/20 15:52:39 WARN storage.BlockManager: Block input-0-1479637359000 replicated to only 0 peer(s) instead of 1 peers
16/11/20 15:52:39 INFO receiver.BlockGenerator: Pushed block input-0-1479637359000
16/11/20 15:52:39 INFO storage.MemoryStore: ensureFreeSpace(45512) called with curMem=12642113, maxMem=560497950
16/11/20 15:52:39 INFO storage.MemoryStore: Block input-0-1479637359200 stored as bytes in memory (estimated size 44.4 KB, free 522.4 MB)
16/11/20 15:52:39 WARN storage.BlockManager: Block input-0-1479637359200 replicated to only 0 peer(s) instead of 1 peers

以下是STDERR登录spark UI的快照

div.problematic_div {
  position:relative;
}

日志不断增长，如上所述。请求您帮我解决问题。

0 个答案:

没有答案
相关问题
最新问题