Spark Streaming应用程序卡在远程计算机上

时间:2018-07-02 14:20:20

标签: scala apache-spark jar apache-kafka

我将我的Spark使用者和Kafka生产者应用程序作为可执行jar运行。当我在本地计算机上都运行它们时,一切正常。当我在另一台计算机上使用ssh运行它们时,Spark流应用程序会停留在“ Registered StateStoreCoordinator端点”消息上,并且不会使用Kafka生成的消息。发生这种行为的原因可能是什么?如何解决此问题?

18/07/02 12:41:48 INFO SparkContext: Running Spark version 2.2.0
    18/07/02 12:41:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    18/07/02 12:41:49 WARN Utils: Your hostname, ip-ofthemachine resolves to a loopback address: 127.0.1.1; using ipofthemachine instead (on interface eth0)
    18/07/02 12:41:49 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
    18/07/02 12:41:49 INFO SparkContext: Submitted application: KafkaSparkStreaming
    18/07/02 12:41:49 INFO SecurityManager: Changing view acls to: admin
    18/07/02 12:41:49 INFO SecurityManager: Changing modify acls to: admin
    18/07/02 12:41:49 INFO SecurityManager: Changing view acls groups to: 
    18/07/02 12:41:49 INFO SecurityManager: Changing modify acls groups to: 
    18/07/02 12:41:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(user); groups with view permissions: Set(); users  with modify permissions: Set(user); groups with modify permissions: Set()
    18/07/02 12:41:49 INFO Utils: Successfully started service 'sparkDriver' on port 35401.
    18/07/02 12:41:49 INFO SparkEnv: Registering MapOutputTracker
    18/07/02 12:41:49 INFO SparkEnv: Registering BlockManagerMaster
    18/07/02 12:41:49 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
    18/07/02 12:41:49 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
    18/07/02 12:41:49 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-be4b044f-3082-4199-a1c4-e9ca95d5a9a9
    18/07/02 12:41:49 INFO MemoryStore: MemoryStore started with capacity 1960.8 MB
    18/07/02 12:41:49 INFO SparkEnv: Registering OutputCommitCoordinator
    18/07/02 12:41:49 INFO Utils: Successfully started service 'SparkUI' on port 4040.
    18/07/02 12:41:49 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://ipofthemachine:4040
    18/07/02 12:41:49 INFO Executor: Starting executor ID driver on host localhost
    18/07/02 12:41:49 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34187.
    18/07/02 12:41:49 INFO NettyBlockTransferService: Server created on ipofthemachine:34187
    18/07/02 12:41:49 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
    18/07/02 12:41:49 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, ipofthemachine, 34187, None)
    18/07/02 12:41:50 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:path').
    18/07/02 12:41:50 INFO SharedState: Warehouse path is 'file:path'.
    18/07/02 12:41:50 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint

工作方式:

18/07/02 15:34:45 INFO SparkContext: Running Spark version 2.2.0
18/07/02 15:34:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/07/02 15:34:46 WARN Utils: Your hostname resolves to a loopback address: 127.0.1.1; using ipofthelocalmachine instead (on interface wlp2s0)
18/07/02 15:34:46 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
18/07/02 15:34:46 INFO SparkContext: Submitted application: KafkaSparkStreaming
18/07/02 15:34:46 INFO SecurityManager: Changing view acls to: roksolana
18/07/02 15:34:46 INFO SecurityManager: Changing modify acls to: roksolana
18/07/02 15:34:46 INFO SecurityManager: Changing view acls groups to: 
18/07/02 15:34:46 INFO SecurityManager: Changing modify acls groups to: 
18/07/02 15:34:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(user); groups with view permissions: Set(); users  with modify permissions: Set(user); groups with modify permissions: Set()
18/07/02 15:34:46 INFO Utils: Successfully started service 'sparkDriver' on port 43701.
18/07/02 15:34:46 INFO SparkEnv: Registering MapOutputTracker
18/07/02 15:34:46 INFO SparkEnv: Registering BlockManagerMaster
18/07/02 15:34:46 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/07/02 15:34:46 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/07/02 15:34:46 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-3327ce5a-640e-4961-88a8-e8c1140e1f73
18/07/02 15:34:46 INFO MemoryStore: MemoryStore started with capacity 1946.1 MB
18/07/02 15:34:46 INFO SparkEnv: Registering OutputCommitCoordinator
18/07/02 15:34:46 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/07/02 15:34:46 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://ipofthelocalmachine:4040
18/07/02 15:34:46 INFO Executor: Starting executor ID driver on host localhost
18/07/02 15:34:46 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42963.
18/07/02 15:34:46 INFO NettyBlockTransferService: Server created on ipofthelocalmachine:42963
18/07/02 15:34:46 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, ipofthelocalmachine, 42963, None)
18/07/02 15:34:46 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver,ipofthelocalmachine, 42963, None)
18/07/02 15:34:46 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:path').
18/07/02 15:34:46 INFO SharedState: Warehouse path is 'file:path'.
18/07/02 15:34:47 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
-------------------------------------------
Time: 1530534888000 ms
-------------------------------------------

-------------------------------------------
Time: 1530534891000 ms
-------------------------------------------

0 个答案:

没有答案