我有一个群集设置,其中有一个驱动程序和两个执行程序以yarn模式运行。每当我提交带有master作为yarn且在部署模式下作为client且动态内存分配为false的spark代码时,它都可以正确执行,但是当在Spark历史记录服务器中查看详细信息时,它表明仅在其中一个执行器中执行了代码/ workers。
我的EC2机器(2个内核和1个主设备):c4.4xlarge,带有30.0 GiB和16vCPU
spark-submit --packages org.apache.spark:spark-streaming-kafka_2.11:1.6.3 --master yarn --driver-memory 24g --executor-memory 20g --num-executors 2 --conf spark.shuffle.service.enabled=false --conf spark.dynamicAllocation.enabled=false --conf spark.dynamicAllocation.maxExecutors=2 --conf spark.dynamicAllocation.minExecutors=2 spark_streaming_job.py
执行者之一的日志:
19/02/10 05:35:47 INFO RMProxy: Connecting to ResourceManager at ip-xx-xx-xx-xx.ap-xxxxx-1.compute.internal/xx.xx.xx.xx:8030
19/02/10 05:35:47 INFO YarnRMClient: Registering the ApplicationMaster
19/02/10 05:35:47 INFO Utils: Using initial executors = 2, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
19/02/10 05:35:47 INFO YarnAllocator: Will request 2 executor container(s), each with 4 core(s) and 22528 MB memory (including 2048 MB of overhead)
19/02/10 05:35:47 INFO YarnAllocator: Submitted 2 unlocalized container requests.
19/02/10 05:35:47 INFO ApplicationMaster: Started progress reporter thread with (heartbeat : 3000, initial allocation : 200) intervals
19/02/10 05:35:49 INFO AMRMClientImpl: Received new token for : ip-xx-xx-xx-xx.ap-southeast-1.compute.internal:8041
19/02/10 05:35:49 INFO YarnAllocator: Launching container container_1549550949694_0005_01_000003 on host ip-xx-xx-xx-xx.ap-xxxx-1.compute.internal for executor with ID 1
19/02/10 05:35:49 INFO YarnAllocator: Received 1 containers from YARN, launching executors on 1 of them.
有没有人可以强迫我获得2名执行人。