Spark工作需要约36分钟才能获得约3200万条记录,以便将oracle数据加载到cassandra中。 带有Scala的Spark 1.6.2。 在写入cassandra时,有人可以帮助正确设置。
spark.cassandra.output.batch.size.rows
spark.cassandra.output.concurrent.writes
spark.cassandra.output.batch.size.bytes
我在Master下面看到一些INFO消息。
INFO 2018-03-01 07:55:34,911 com.datastax.spark.connector.cql.CassandraConnector:已连接到 Cassandra集群:OCP-Cluster INFO 2018-03-01 07:55:37,494 com.datastax.spark.connector.cql.CassandraConnector:断开连接 Cassandra集群:OCP-Cluster INFO 2018-03-01 07:55:37,502 com.datastax.spark.connector.cql.CassandraConnector:断开连接 Cassandra集群:OCP-Cluster INFO 2018-03-01 07:55:37,511 com.datastax.spark.connector.cql.CassandraConnector:断开连接 Cassandra集群:OCP-Cluster INFO 2018-03-01 07:55:37,525 com.datastax.spark.connector.cql.CassandraConnector:断开连接 Cassandra集群:OCP-Cluster INFO 2018-03-01 07:55:37,528 com.datastax.spark.connector.cql.CassandraConnector:断开连接 Cassandra集群:OCP-Cluster INFO 2018-03-01 07:55:37,533 com.datastax.spark.connector.cql.CassandraConnector:断开连接 Cassandra集群:OCP-Cluster INFO 2018-03-01 07:55:37,543 com.datastax.spark.connector.cql.CassandraConnector:断开连接 Cassandra集群:OCP-Cluster INFO 2018-03-01 07:55:37,543 com.datastax.spark.connector.cql.CassandraConnector:断开连接 Cassandra集群:OCP-Cluster INFO 2018-03-01 07:55:37,546 com.datastax.spark.connector.cql.CassandraConnector:断开连接 Cassandra集群:OCP-Cluster INFO 2018-03-01 07:55:37,547 com.datastax.spark.connector.cql.CassandraConnector:断开连接 Cassandra集群:OCP-Cluster