Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-1608084c], Releasing partition ownership
Time INFO ConsumerFetcherManager: [ConsumerFetcherManager-1512422151191]
Stopping leader finder thread
Time INFO ConsumerFetcherManager: [ConsumerFetcherManager-1512422151191]
Stopping all fetchers
Time INFO ConsumerFetcherManager: [ConsumerFetcherManager-1512422151191] All
connections stopped
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-6e2f699d], Cleared all relevant queues for this fetcher
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-6e2f699d], Cleared the data chunks in all the consumer message
iterators
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-6e2f699d], Committing all offsets after clearing the fetcher
queues
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-6e2f699d], Releasing partition ownership
Time INFO RangeAssignor: Consumer test-consumer-group_ip-10-114-10-19-
1512422151126-1608084c rebalancing the following partitions: ArrayBuffer(0,
1, 2) for topic JSON_Binder_Compute_Kafka with consumers: List(test-
consumer-group_ip-10-114-10-19-1512422151126-1608084c-0, test-consumer-
group_ip-10-114-10-19-1512422151126-1608084c-1)
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151126-
1608084c-0 attempting to claim partition 0
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151126-
1608084c-0 attempting to claim partition 1
Time INFO RangeAssignor: Consumer test-consumer-group_ip-10-114-10-19-
1512422151123-562f9f99 rebalancing the following partitions: ArrayBuffer(0,
1, 2) for topic JSON_Adj_Balance_Kafka with consumers: List(test-consumer-
group_ip-10-114-10-19-1512422151123-562f9f99-0, test-consumer-group_ip-10-
114-10-19-1512422151123-562f9f99-1)
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151126-
1608084c-1 attempting to claim partition 2
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151123-
562f9f99-0 attempting to claim partition 0
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151123-
562f9f99-0 attempting to claim partition 1
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151123-
562f9f99-1 attempting to claim partition 2
Time INFO RangeAssignor: Consumer test-consumer-group_ip-10-114-10-19-
1512422151129-f890a5cd rebalancing the following partitions: ArrayBuffer(0,
1, 2) for topic JSON_Binder_Compute_LC_Kafka with consumers: List(test-
consumer-group_ip-10-114-10-19-1512422151129-f890a5cd-0, test-consumer-
group_ip-10-114-10-19-1512422151129-f890a5cd-1)
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151129-
f890a5cd-0 attempting to claim partition 0
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151129-
f890a5cd-0 attempting to claim partition 1
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151129-
f890a5cd-1 attempting to claim partition 2
Time INFO RangeAssignor: Consumer test-consumer-group_ip-10-114-10-19-
1512422151126-6e2f699d rebalancing the following partitions: ArrayBuffer(0,
1, 2) for topic JSON_Binder_Status_Kafka with consumers: List(test-
onsumer-group_ip-10-114-10-19-1512422151126-6e2f699d-0, test-consumer-
group_ip-10-114-10-19-1512422151126-6e2f699d-1)
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151126-
6e2f699d-0 attempting to claim partition 0
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151126-
6e2f699d-0 attempting to claim partition 1
Time INFO RangeAssignor: test-consumer-group_ip-10-114-10-19-1512422151126-
6e2f699d-1 attempting to claim partition 2
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-6e2f699d], test-consumer-group_ip-10-114-10-19-1512422151126-
6e2f699d-0 successfully owned partition 0 for topic JSON_Binder_Status_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-1608084c], test-consumer-group_ip-10-114-10-19-1512422151126-
1608084c-0 successfully owned partition 1 for topic
JSON_Binder_Compute_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151129-f890a5cd], test-consumer-group_ip-10-114-10-19-1512422151129-
f890a5cd-0 successfully owned partition 0 for topic
JSON_Binder_Compute_LC_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151123-562f9f99], test-consumer-group_ip-10-114-10-19-1512422151123-
562f9f99-0 successfully owned partition 1 for topic JSON_Adj_Balance_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-1608084c], test-consumer-group_ip-10-114-10-19-1512422151126-
1608084c-1 successfully owned partition 2 for topic
JSON_Binder_Compute_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-6e2f699d], test-consumer-group_ip-10-114-10-19-1512422151126-
6e2f699d-1 successfully owned partition 2 for topic
JSON_Binder_Status_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151129-f890a5cd], test-consumer-group_ip-10-114-10-19-1512422151129-
f890a5cd-1 successfully owned partition 2 for topic
JSON_Binder_Compute_LC_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151123-562f9f99], test-consumer-group_ip-10-114-10-19-1512422151123-
562f9f99-1 successfully owned partition 2 for topic JSON_Adj_Balance_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-1608084c], test-consumer-group_ip-10-114-10-19-1512422151126-
1608084c-0 successfully owned partition 0 for topic
JSON_Binder_Compute_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151129-f890a5cd], test-consumer-group_ip-10-114-10-19-1512422151129-
f890a5cd-0 successfully owned partition 1 for topic
JSON_Binder_Compute_LC_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-6e2f699d], test-consumer-group_ip-10-114-10-19-1512422151126-
6e2f699d-0 successfully owned partition 1 for topic JSON_Binder_Status_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151123-562f9f99], test-consumer-group_ip-10-114-10-19-1512422151123-
562f9f99-0 successfully owned partition 0 for topic JSON_Adj_Balance_Kafka
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-6e2f699d], Consumer test-consumer-group_ip-10-114-10-19-
1512422151126-6e2f699d selected partitions : JSON_Binder_Status_Kafka:0:
fetched offset = 0: consumed offset = 0,JSON_Binder_Status_Kafka:1: fetched
offset = 0: consumed offset = 0,JSON_Binder_Status_Kafka:2: fetched offset =
111: consumed offset = 111
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151129-f890a5cd], Consumer test-consumer-group_ip-10-114-10-19-
1512422151129-f890a5cd selected partitions : JSON_Binder_Compute_LC_Kafka:0:
fetched offset = 0: consumed offset = 0,JSON_Binder_Compute_LC_Kafka:1:
fetched offset = 0: consumed offset = 0,JSON_Binder_Compute_LC_Kafka:2:
fetched offset =
25: consumed offset = 25
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151123-562f9f99], Consumer test-consumer-group_ip-10-114-10-19-
1512422151123-562f9f99 selected partitions : JSON_Adj_Balance_Kafka:0:
fetched offset = 0: consumed offset = 0,JSON_Adj_Balance_Kafka:1: fetched
offset = 1: consumed offset = 1,JSON_Adj_Balance_Kafka:2: fetched offset =
78: consumed offset = 78
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-1608084c], Consumer test-consumer-group_ip-10-114-10-19-
1512422151126-1608084c selected partitions : JSON_Binder_Compute_Kafka:0:
fetched offset = 0: consumed offset = 0,JSON_Binder_Compute_Kafka:1: fetched
offset = 1: consumed offset = 1,JSON_Binder_Compute_Kafka:2: fetched offset
= 137: consumed offset = 137
Time INFO ConsumerFetcherManager$LeaderFinderThread: [test-consumer-
group_ip-10-114-10-19-1512422151123-562f9f99-leader-finder-thread]: Starting
Time INFO ConsumerFetcherManager$LeaderFinderThread: [test-consumer-
group_ip-10-114-10-19-1512422151126-6e2f699d-leader-finder-thread]: Starting
Time INFO ConsumerFetcherManager$LeaderFinderThread: [test-consumer-
group_ip-10-114-10-19-1512422151126-1608084c-leader-finder-thread]: Starting
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151123-562f9f99], end rebalancing consumer test-consumer-group_ip-10-
114-
10-19-1512422151123-562f9f99 try #0
Time INFO ConsumerFetcherManager$LeaderFinderThread: [test-consumer-
group_ip-10-114-10-19-1512422151129-f890a5cd-leader-finder-thread]: Starting
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-6e2f699d], end rebalancing consumer test-consumer-group_ip-10-
114-10-19-1512422151126-6e2f699d try #0
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151126-1608084c], end rebalancing consumer test-consumer-group_ip-10-
114-10-19-1512422151126-1608084c try #0
Time INFO ZookeeperConsumerConnector: [test-consumer-group_ip-10-114-10-19-
1512422151129-f890a5cd], end rebalancing consumer test-consumer-group_ip-10-
114-10-19-1512422151129-f890a5cd try #0
Time INFO KafkaReceiver: Starting MessageHandler.
Time INFO KafkaReceiver: Starting MessageHandler.
Time INFO KafkaReceiver: Starting MessageHandler.
Time INFO KafkaReceiver: Starting MessageHandler.
Time INFO KafkaReceiver: Starting MessageHandler.
Time INFO ReceiverSupervisorImpl: Called receiver 0 onStart
Time INFO KafkaReceiver: Starting MessageHandler.
Time INFO ReceiverSupervisorImpl: Called receiver 1 onStart
Time INFO ReceiverSupervisorImpl: Called receiver 3 onStart
Time INFO ReceiverSupervisorImpl: Waiting for receiver to be stopped
Time INFO ReceiverSupervisorImpl: Waiting for receiver to be stopped
Time INFO ReceiverSupervisorImpl: Called receiver 2 onStart
Time INFO ReceiverSupervisorImpl: Waiting for receiver to be stopped
Time INFO KafkaReceiver: Starting MessageHandler.
Time INFO KafkaReceiver: Starting MessageHandler.
Time INFO ReceiverSupervisorImpl: Waiting for receiver to be stopped
Time INFO VerifiableProperties: Verifying properties
Time INFO VerifiableProperties: Property client.id is overridden to test-
consumer-group
Time INFO VerifiableProperties: Property metadata.broker.list is overridden
to ip-10-114-7-173.ec2.internal:9092,ip-10-114-15-50.ec2.internal:9092,ip-
10-114-18-85.ec2.internal:9092
Time INFO VerifiableProperties: Property request. Timeout.ms is
overridden to 30000
Time INFO VerifiableProperties: Verifying properties
Time INFO VerifiableProperties: Property client.id is overridden to test-
consumer-group
Time INFO VerifiableProperties: Property metadata.broker.list is overridden
to ip-10-114-7-173.ec2.internal:9092,ip-10-114-15-50.ec2.internal:9092,ip-
10-114-18-85.ec2.internal:9092
Time INFO VerifiableProperties: Property request. Timeout.ms is
overridden to 30000
Time INFO VerifiableProperties: Verifying properties
Time INFO VerifiableProperties: Verifying properties
Time INFO VerifiableProperties: Property client.id is overridden to test-
consumer-group
Time INFO VerifiableProperties: Property metadata.broker.list is overridden
to ip-10-114-7-173.ec2.internal:9092,ip-10-114-15-50.ec2.internal:9092,ip-
10-114-18-85.ec2.internal:9092
Time INFO VerifiableProperties: Property request. Timeout.ms is
overridden to 30000
Time INFO VerifiableProperties: Property client.id is overridden to test-
consumer-group
Time INFO VerifiableProperties: Property metadata.broker.list is overridden
to ip-10-114-7-173.ec2.internal:9092,ip-10-114-15-50.ec2.internal:9092,ip-
10-114-18-85.ec2.internal:9092
Time INFO VerifiableProperties: Property request. Timeout.ms is
overridden to 30000
Time INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,ip-
10-114-15-50.ec2.internal,9092) with correlation id 0 for 1 topic(s)
Set(JSON_Adj_Balance_Kafka)
Time INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(2,ip-
10-114-15-50.ec2.internal,9092) with correlation id 0 for 1 topic(s)
Set(JSON_Binder_Compute_Kafka)
Time INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,ip-
10-114-7-173.ec2.internal,9092) with correlation id 0 for 1 topic(s)
Set(JSON_Binder_Compute_LC_Kafka)
Time INFO ClientUtils$: Fetching metadata from broker BrokerEndPoint(1,ip-
10-114-7-173.ec2.internal,9092) with correlation id 0 for 1 topic(s)
Set(JSON_Binder_Status_Kafka)
Time INFO SyncProducer: Connected to ip-10-114-15-50.ec2.internal:9092 for
producing
Time INFO SyncProducer: Connected to ip-10-114-7-173.ec2.internal:9092 for
roducing
Time INFO SyncProducer: Connected to ip-10-114-7-173.ec2.internal:9092 for
producing
Time INFO SyncProducer: Connected to ip-10-114-15-50.ec2.internal:9092 for
producing
Time INFO SyncProducer: Disconnecting from ip-10-114-15-50.ec2.internal:9092
Time INFO SyncProducer: Disconnecting from ip-10-114-7-173.ec2.internal:9092
Time INFO SyncProducer: Disconnecting from ip-10-114-15-50.ec2.internal:9092
Time INFO SyncProducer: Disconnecting from ip-10-114-7-173.ec2.internal:9092
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151126-1608084c-0-1]: Starting
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151126-6e2f699d-0-1]: Starting
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151129-f890a5cd-0-1]: Starting
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151123-562f9f99-0-1]: Starting
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151129-f890a5cd-0-2]: Starting
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151126-6e2f699d-0-2]: Starting
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151129-f890a5cd-0-3]: Starting
Time INFO ConsumerFetcherManager: [ConsumerFetcherManager-1512422151191]
Added fetcher for partitions ArrayBuffer([JSON_Binder_Compute_LC_Kafka-0,
initOffset 0 to broker BrokerEndPoint(1,ip-10-114-7-173.ec2.internal,9092)] ,
[JSON_Binder_Compute_LC_Kafka-2, initOffset 25 to broker
BrokerEndPoint(3,ip-10-114-18-85.ec2.internal,9092)] ,
[JSON_Binder_Compute_LC_Kafka-1, initOffset 0 to broker BrokerEndPoint(2,ip-
10-114-15-50.ec2.internal,9092)] )
Time INFO ConsumerFetcherManager: [ConsumerFetcherManager-1512422151191]
Added fetcher for partitions ArrayBuffer([JSON_Binder_Compute_Kafka-1,
initOffset 1 to broker BrokerEndPoint(3,ip-10-114-18-85.ec2.internal,9092)]
, [JSON_Binder_Compute_Kafka-0, initOffset 0 to broker BrokerEndPoint(2,ip-
10-114-15-50.ec2.internal,9092)] , [JSON_Binder_Compute_Kafka-2, initOffset
137 to broker BrokerEndPoint(1,ip-10-114-7-173.ec2.internal,9092)] )
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151123-562f9f99-0-2]: Starting
Time INFO ConsumerFetcherManager: [ConsumerFetcherManager-1512422151191]
Added fetcher for partitions ArrayBuffer([JSON_Binder_Status_Kafka-2,
initOffset 111 to broker BrokerEndPoint(1,ip-10-114-7-
173.ec2.internal,9092)] , [JSON_Binder_Status_Kafka-1, initOffset 0 to
broker BrokerEndPoint(3,ip-10-114-18-85.ec2.internal,9092)] ,
[JSON_Binder_Status_Kafka-0, initOffset 0 to broker BrokerEndPoint(2,ip-10-
114-15-50.ec2.internal,9092)] )
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151126-1608084c-0-3]: Starting
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151126-6e2f699d-0-3]: Starting
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151126-1608084c-0-2]: Starting
Time INFO ConsumerFetcherManager: [ConsumerFetcherManager-1512422151191]
Added fetcher for partitions ArrayBuffer([JSON_Adj_Balance_Kafka-0,
initOffset 0 to broker BrokerEndPoint(1,ip-10-114-7-173.ec2.internal,9092)]
JSON_Adj_Balance_Kafka-2, initOffset 78 to broker BrokerEndPoint(3,ip-10-
114-18-85.ec2.internal,9092)] , [JSON_Adj_Balance_Kafka-1, initOffset 1 to
roker BrokerEndPoint(2,ip-10-114-15-50.ec2.internal,9092)] )
Time INFO ConsumerFetcherThread: [ConsumerFetcherThread-test-consumer-
group_ip-10-114-10-19-1512422151123-562f9f99-0-3]: Starting
Time INFO MemoryStore: Block input-3-1512422156800 stored as bytes in memory
(estimated size 208.0 B, free 911.9 MB)
Time INFO BlockManagerInfo: Added input-3-1512422156800 in memory on
10.114.10.19:43101 (size: 208.0 B, free: 912.2 MB)
Time INFO BlockGenerator: Pushed block input-3-1512422156800
Time INFO JobScheduler: Added jobs for Time 1512422160000 ms
Time INFO JobScheduler: Starting job streaming job 1512422160000 ms.0 from
job set of Time 1512422160000 ms
Time INFO JobScheduler: Starting job streaming job 1512422160000 ms.2 from
job set of Time 1512422160000 ms
Time INFO JobScheduler: Starting job streaming job 1512422160000 ms.1 from
job set of Time 1512422160000 ms
Time INFO JobScheduler: Starting job streaming job 1512422160000 ms.3 from
job set of Time 1512422160000 ms
Time INFO SparkContext: Starting job: count at Odq_WTB_Streaming.scala:81
Time INFO SparkContext: Starting job: count at Odq_WTB_Streaming.scala:148
Time INFO DAGScheduler: Got job 5 (count at Odq_WTB_Streaming.scala:148)
with 1 output partitions
Time INFO DAGScheduler: Final stage: ResultStage 4 (count at
Odq_WTB_Streaming.scala:148)
Time INFO DAGScheduler: Parents of final stage: List()
Time INFO DAGScheduler: Missing parents: List()
Time INFO MapPartitionsRDD: Removing RDD 5 from persistence list
Time INFO DAGScheduler: Submitting ResultStage 4 (MapPartitionsRDD[11] at
map at Odq_WTB_Streaming.scala:54), which has no missing parents
Time INFO SparkContext: Starting job: count at Odq_WTB_Streaming.scala:103
Time INFO DAGScheduler: Job 6 finished: count at
Odq_WTB_Streaming.scala:103, took 0.000064 s
Time INFO DAGScheduler: Job 4 finished: count at Odq_WTB_Streaming.scala:81,
took 0.006275 s
Time INFO SparkContext: Starting job: count at Odq_WTB_Streaming.scala:105
Time INFO DAGScheduler: Job 7 finished: count at
Odq_WTB_Streaming.scala:105, took 0.000039 s
stage3_out Time INFO MapPartitionsRDD: Removing RDD 9 from persistence
list
Time INFO SparkContext: Starting job: count at
Odq_WTB_Streaming.scala:82
Time INFO DAGScheduler: Job 8 finished: count at
Odq_WTB_Streaming.scala:82, took 0.000024 s
stage2_out Time INFO MapPartitionsRDD: Removing RDD 7 from
persistence list
Time INFO BlockManager: Removing RDD 9
Time INFO BlockManager: Removing RDD 5
Time INFO BlockManager: Removing RDD 7
Time INFO JobScheduler: Finished job streaming job 1512422160000 ms.1 from
job set of Time 1512422160000 ms
Time INFO MemoryStore: Block broadcast_4 stored as values in memory
(estimated size 1560.0 B, free 911.9 MB)
Time INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory
(estimated size 1087.0 B, free 911.9 MB)
Time INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on
10.114.10.19:43101 (size: 1087.0 B, free: 912.2 MB)
Time INFO SparkContext: Created broadcast 4 from broadcast at
DAGScheduler.scala:1047
Time INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 4
(MapPartitionsRDD[11] at map at Odq_WTB_Streaming.scala:54) (first 15 tasks
are for partitions Vector(0))
Time INFO TaskSchedulerImpl: Adding task set 4.0 with 1 tasks
Time INFO JobScheduler: Finished job streaming job 1512422160000 ms.0 from
job set of Time 1512422160000 ms
Time INFO JobScheduler: Finished job streaming job 1512422160000 ms.2 from
job set of Time 1512422160000 ms
17/12/04 21:18:00 INFO JobScheduler: Added jobs for Time 1512422280000
ms
17/12/04 21:18:00 INFO MapPartitionsRDD: Removing RDD 13 from persistence
list
17/12/04 21:18:00 INFO JobScheduler: Starting job streaming job
1512422280000 ms.0 from job set of Time 1512422280000 ms
17/12/04 21:18:00 INFO JobScheduler: Starting job streaming job
1512422280000 ms.2 from job set of Time 1512422280000 ms
17/12/04 21:18:00 INFO BlockManager: Removing RDD 13
17/12/04 21:18:00 INFO SparkContext: Starting job: count at
Odq_WTB_Streaming.scala:103
17/12/04 21:18:00 INFO DAGScheduler: Job 9 finished: count at
Odq_WTB_Streaming.scala:103, took 0.000022 s
7/12/04 21:18:00 INFO JobScheduler: Starting job streaming job 1512422280000
ms.1 from job set of Time 1512422280000 ms
17/12/04 21:18:00 INFO SparkContext: Starting job: count at
Odq_WTB_Streaming.scala:81
17/12/04 21:18:00 INFO DAGScheduler: Job 10 finished: count at
Odq_WTB_Streaming.scala:81, took 0.000016 s
17/12/04 21:18:00 INFO SparkContext: Starting job: count at
Odq_WTB_Streaming.scala:82
17/12/04 21:18:00 INFO DAGScheduler: Job 11 finished: count at
Odq_WTB_Streaming.scala:82, took 0.000016 s
17/12/04 21:18:00 INFO SparkContext: Starting job: count at
Odq_WTB_Streaming.scala:105
17/12/04 21:18:00 INFO DAGScheduler: Job 12 finished: count at
Odq_WTB_Streaming.scala:105, took 0.000020 s
17/12/04 21:18:00 INFO MapPartitionsRDD: Removing RDD 15 from persistence
list
17/12/04 21:18:00 INFO MapPartitionsRDD: Removing RDD 17 from persistence
list
17/12/04 21:18:00 INFO BlockManager: Removing RDD 15
17/12/04 21:18:00 INFO BlockManager: Removing RDD 17
17/12/04 21:18:00 INFO JobScheduler: Finished job streaming job
1512422280000 ms.0 from job set of Time 1512422280000 ms
17/12/04 21:18:00 INFO JobScheduler: Finished job streaming job
1512422280000 ms.1 from job set of Time 1512422280000 ms
17/12/04 21:18:00 INFO JobScheduler: Starting job streaming job
1512422280000 ms.3 from job set of Time 1512422280000 ms
17/12/04 21:18:00 INFO JobScheduler: Finished job streaming job
1512422280000 ms.2 from job set of Time 1512422280000 ms
17/12/04 21:18:00 INFO SparkContext: Starting job: count at
Odq_WTB_Streaming.scala:148
17/12/04 21:18:00 INFO DAGScheduler: Job 13 finished: count at
Odq_WTB_Streaming.scala:148, took 0.000084 s
17/12/04 21:18:00 INFO SparkContext: Starting job: count at
Odq_WTB_Streaming.scala:149
17/12/04 21:18:00 INFO DAGScheduler: Job 14 finished: count at
Odq_WTB_Streaming.scala:149, took 0.000020 s
17/12/04 21:18:00 INFO JobScheduler: Finished job streaming job
1512422280000 ms.3 from job set of Time 1512422280000 ms
17/12/04 21:18:00 INFO JobScheduler: Total delay: 0.031 s for Time
1512422280000 ms (execution: 0.021 s)
17/12/04 21:18:00 INFO MapPartitionsRDD: Removing RDD 5 from persistence
list
17/12/04 21:18:00 INFO BlockManager: Removing RDD 5
17/12/04 21:18:00 INFO BlockRDD: Removing RDD 4 from persistence list
17/12/04 21:18:00 INFO BlockManager: Removing RDD 4
17/12/04 21:18:00 INFO KafkaInputDStream: Removing blocks of RDD BlockRDD[4]
at createStream at Odq_WTB_Streaming.scala:50 of Time 1512422280000 ms
17/12/04 21:18:00 INFO MapPartitionsRDD: Removing RDD 7 from persistence
list
17/12/04 21:18:00 INFO BlockManager: Removing RDD 7
17/12/04 21:18:00 INFO BlockRDD: Removing RDD 6 from persistence list
17/12/04 21:18:00 INFO BlockManager: Removing RDD 6
17/12/04 21:18:00 INFO KafkaInputDStream: Removing blocks of RDD BlockRDD[6]
at createStream at Odq_WTB_Streaming.scala:51 of Time 1512422280000 ms
17/12/04 21:18:00 INFO MapPartitionsRDD: Removing RDD 9 from persistence
list
17/12/04 21:18:00 INFO BlockManager: Removing RDD 9
17/12/04 21:18:00 INFO BlockRDD: Removing RDD 8 from persistence list
17/12/04 21:18:00 INFO BlockManager: Removing RDD 8
17/12/04 21:18:00 INFO KafkaInputDStream: Removing blocks of RDD BlockRDD[8]
at createStream at Odq_WTB_Streaming.scala:52 of Time 1512422280000 ms
17/12/04 21:18:00 INFO MapPartitionsRDD: Removing RDD 11 from persistence
list
17/12/04 21:18:00 INFO BlockManager: Removing RDD 11
17/12/04 21:18:00 INFO BlockRDD: Removing RDD 10 from persistence list
17/12/04 21:18:00 INFO BlockManager: Removing RDD 10
17/12/04 21:18:00 INFO KafkaInputDStream: Removing blocks of RDD
BlockRDD[10] at createStream at Odq_WTB_Streaming.scala:54 of Time
1512422280000 ms
17/12/04 21:18:00 INFO ReceivedBlockTracker: Deleting batches:
17/12/04 21:18:00 INFO BlockManagerInfo: Removed input-3-1512422156800 on
10.114.10.19:43101 in memory (size: 208.0 B, free: 912.2 MB)
17/12/04 21:18:00 INFO InputInfoTracker: remove old batch metadata:
context：我正在尝试使用Spark从Kafka的5个主题中流式传输
流。我有
流媒体3主题没问题。当我添加我的第四个主题时，我确实看到了Spark
消耗消息并存储在内存中。我的逻辑是消费
来自Kafka的消息和存储在临时区域中并且每个都运行一个Spark作业
两分钟到Pics从分段和处理它们。因为我必须做一些
加入这4条数据流。
我有3个Zookeepers，3个Kafka经纪人，Kafka的每个主题都有3个
分区。我将所有这些接收器作为单个消费者群体的一部分。
我已经厌倦了createstream API和CreateDirectStream API。