我正在从连接到远程Cassandra群集的Windows 10中运行Spark。
我正在使用Datastax Cassandra 5.0，它是Apache Cassandra 3.11.1。 我的Spark版本是spark-2.1.0-bin-hadoop2.7。
在cassandra.yaml中，我有：
client_encryption_options:
enabled: true
optional: false
keystore: /etc/ssl/jks/production_client.jks
keystore_password: keystore_pass
require_client_auth: true
truststore: /etc/ssl/jks/production_client_truststore.jks
truststore_password: truststore_pass
# More advanced defaults below:
protocol: TLS
algorithm: SunX509
store_type: JKS
cipher_suites:
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
- TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
- TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
当我启动spark-shell并运行以下命令时，我可以看到ssl没有使用。
import org.apache.spark.{SparkConf, SparkContext}
import org.apache.spark.sql.SQLContext
import com.datastax.spark.connector._
val conf = new SparkConf(true)
conf.set("spark.cassandra.connection.host", "10.0.0.1")
conf.set("spark.cassandra.auth.username","myuser")
conf.set("spark.cassandra.auth.password","mypassword")
conf.set("spark.cassandra.connection.ssl.enabled","true")
conf.set("spark.cassandra.connection.ssl.keyStore.password","keystore_pass")
conf.set("spark.cassandra.connection.ssl.keyStore.path","C:\\works\\tmp\\mykeys\\production_client.jks")
conf.set("spark.cassandra.connection.ssl.trustStore.path","C:\\works\\tmp\\mykeys\\production_client_truststore.jks")
conf.set("spark.cassandra.connection.ssl.trustStore.password","truststore_pass")
conf.set("spark.cassandra.connection.ssl.enabledAlgorithms","Set(TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256)")
val sc = new SparkContext("local", "Cassandra Connector Test", conf)
我启用了调试模式并获得了以下日志：
18/06/29 16:16:29 INFO SparkContext: Running Spark version 2.1.0
18/06/29 16:16:29 INFO SecurityManager: Changing view acls to: myuser
18/06/29 16:16:29 INFO SecurityManager: Changing modify acls to: myuser
18/06/29 16:16:29 INFO SecurityManager: Changing view acls groups to:
18/06/29 16:16:29 INFO SecurityManager: Changing modify acls groups to:
18/06/29 16:16:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(myuser); groups with view permissions: Set(); users with modify permissions: Set(myuser); groups with modify permissions: Set()
18/06/29 16:16:29 DEBUG SecurityManager: Created SSL options for fs: SSLOptions{enabled=false, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
18/06/29 16:16:29 DEBUG TransportServer: Shuffle server started on port: 65427
18/06/29 16:16:29 INFO Utils: Successfully started service 'sparkDriver' on port 65427.
18/06/29 16:16:29 DEBUG SparkEnv: Using serializer: class org.apache.spark.serializer.JavaSerializer
18/06/29 16:16:29 INFO SparkEnv: Registering MapOutputTracker
18/06/29 16:16:29 DEBUG MapOutputTrackerMasterEndpoint: init
18/06/29 16:16:29 INFO SparkEnv: Registering BlockManagerMaster
18/06/29 16:16:29 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/06/29 16:16:29 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/06/29 16:16:29 INFO DiskBlockManager: Created local directory at C:\Users\myuser\AppData\Local\Temp\blockmgr-eb8dc0c1-93f2-499b-939b-eb6e6d5f118e
18/06/29 16:16:29 DEBUG DiskBlockManager: Adding shutdown hook
18/06/29 16:16:29 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
18/06/29 16:16:29 INFO SparkEnv: Registering OutputCommitCoordinator
18/06/29 16:16:29 DEBUG OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: init
18/06/29 16:16:29 DEBUG SecurityManager: Created SSL options for ui: SSLOptions{enabled=false, keyStore=None, keyStorePassword=None, trustStore=None, trustStorePassword=None, protocol=None, enabledAlgorithms=Set()}
18/06/29 16:16:29 INFO Utils: Successfully started service 'SparkUI' on port 4040.
18/06/29 16:16:29 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://172.16.175.4:4040
18/06/29 16:16:29 INFO Executor: Starting executor ID driver on host localhost
18/06/29 16:16:29 DEBUG TransportServer: Shuffle server started on port: 65440
18/06/29 16:16:29 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 65440.
18/06/29 16:16:29 INFO NettyBlockTransferService: Server created on 172.16.175.4:65440
18/06/29 16:16:29 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/06/29 16:16:29 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 172.16.175.4, 65440, None)
18/06/29 16:16:29 DEBUG DefaultTopologyMapper: Got a request for 172.16.175.4
18/06/29 16:16:29 INFO BlockManagerMasterEndpoint: Registering block manager 172.16.175.4:65440 with 366.3 MB RAM, BlockManagerId(driver, 172.16.175.4, 65440, None)
18/06/29 16:16:29 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 172.16.175.4, 65440, None)
18/06/29 16:16:29 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 172.16.175.4, 65440, None)
18/06/29 16:16:29 DEBUG SparkContext: Adding shutdown hook
sc: org.apache.spark.SparkContext = org.apache.spark.SparkContext@57dee2b9
现在，当我运行查询时，在以下命令中出现传输错误。
val df = sqlContext.read.format("org.apache.spark.sql.cassandra").options(Map("keyspace" -> "mykeyspace", "table" -> "mytable")).load.cache()
显示错误：
com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.0.0.1:9042 (com.datastax.driver.core.exceptions.TransportException: [/10.0.0.1:9042] Channel has been closed))
有人可以建议我在这里我想念的吗？
谢谢。