我将我的Spark使用者和Kafka生产者应用程序作为可执行jar运行。当我在本地计算机上都运行它们时,一切正常。当我在另一台计算机上使用ssh运行它们时,Spark流应用程序会停留在“ Registered StateStoreCoordinator端点”消息上,并且不会使用Kafka生成的消息。发生这种行为的原因可能是什么?如何解决此问题?
18/07/02 12:41:48 INFO SparkContext: Running Spark version 2.2.0
18/07/02 12:41:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/07/02 12:41:49 WARN Utils: Your hostname, ip-ofthemachine resolves to a loopback address: 127.0.1.1; using ipofthemachine instead (on interface eth0)
18/07/02 12:41:49 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
18/07/02 12:41:49 INFO SparkContext: Submitted application: KafkaSparkStreaming
18/07/02 12:41:49 INFO SecurityManager: Changing view acls to: admin
18/07/02 12:41:49 INFO SecurityManager: Changing modify acls to: admin
18/07/02 12:41:49 INFO SecurityManager: Changing view acls groups to:
18/07/02 12:41:49 INFO SecurityManager: Changing modify acls groups to:
18/07/02 12:41:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); groups with view permissions: Set(); users with modify permissions: Set(user); groups with modify permissions: Set()
18/07/02 12:41:49 INFO Utils: Successfully started service 'sparkDriver' on port 35401.
18/07/02 12:41:49 INFO SparkEnv: Registering MapOutputTracker
18/07/02 12:41:49 INFO SparkEnv: Registering BlockManagerMaster
18/07/02 12:41:49 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/07/02 12:41:49 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/07/02 12:41:49 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-be4b044f-3082-4199-a1c4-e9ca95d5a9a9
18/07/02 12:41:49 INFO MemoryStore: MemoryStore started with capacity 1960.8 MB
18/07/02 12:41:49 INFO SparkEnv: Registering OutputCommitCoordinator
18/07/02 12:41:49 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/07/02 12:41:49 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://ipofthemachine:4040
18/07/02 12:41:49 INFO Executor: Starting executor ID driver on host localhost
18/07/02 12:41:49 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 34187.
18/07/02 12:41:49 INFO NettyBlockTransferService: Server created on ipofthemachine:34187
18/07/02 12:41:49 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/07/02 12:41:49 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, ipofthemachine, 34187, None)
18/07/02 12:41:50 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:path').
18/07/02 12:41:50 INFO SharedState: Warehouse path is 'file:path'.
18/07/02 12:41:50 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
工作方式:
18/07/02 15:34:45 INFO SparkContext: Running Spark version 2.2.0
18/07/02 15:34:45 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
18/07/02 15:34:46 WARN Utils: Your hostname resolves to a loopback address: 127.0.1.1; using ipofthelocalmachine instead (on interface wlp2s0)
18/07/02 15:34:46 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
18/07/02 15:34:46 INFO SparkContext: Submitted application: KafkaSparkStreaming
18/07/02 15:34:46 INFO SecurityManager: Changing view acls to: roksolana
18/07/02 15:34:46 INFO SecurityManager: Changing modify acls to: roksolana
18/07/02 15:34:46 INFO SecurityManager: Changing view acls groups to:
18/07/02 15:34:46 INFO SecurityManager: Changing modify acls groups to:
18/07/02 15:34:46 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(user); groups with view permissions: Set(); users with modify permissions: Set(user); groups with modify permissions: Set()
18/07/02 15:34:46 INFO Utils: Successfully started service 'sparkDriver' on port 43701.
18/07/02 15:34:46 INFO SparkEnv: Registering MapOutputTracker
18/07/02 15:34:46 INFO SparkEnv: Registering BlockManagerMaster
18/07/02 15:34:46 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/07/02 15:34:46 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/07/02 15:34:46 INFO DiskBlockManager: Created local directory at /tmp/blockmgr-3327ce5a-640e-4961-88a8-e8c1140e1f73
18/07/02 15:34:46 INFO MemoryStore: MemoryStore started with capacity 1946.1 MB
18/07/02 15:34:46 INFO SparkEnv: Registering OutputCommitCoordinator
18/07/02 15:34:46 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/07/02 15:34:46 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://ipofthelocalmachine:4040
18/07/02 15:34:46 INFO Executor: Starting executor ID driver on host localhost
18/07/02 15:34:46 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 42963.
18/07/02 15:34:46 INFO NettyBlockTransferService: Server created on ipofthelocalmachine:42963
18/07/02 15:34:46 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, ipofthelocalmachine, 42963, None)
18/07/02 15:34:46 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver,ipofthelocalmachine, 42963, None)
18/07/02 15:34:46 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:path').
18/07/02 15:34:46 INFO SharedState: Warehouse path is 'file:path'.
18/07/02 15:34:47 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
-------------------------------------------
Time: 1530534888000 ms
-------------------------------------------
-------------------------------------------
Time: 1530534891000 ms
-------------------------------------------