Question

我试图从Spark连接到我当地的MongoDB 我已经阅读了很少有关它的主题，但没有一个有效。我的设置有什么问题？

  val spark = SparkSession.builder()
    .master("local")
    .appName("MongoSparkConnectorIntro")
    .config("spark.mongodb.input.uri", "mongodb://user:pass@localhost:27017/dbName.collectionName")
    .getOrCreate()

  it should "connect" in {
    val rdd = MongoSpark.load(spark)
    println(rdd.count())
  }

验证设置：

mongo dbName --eval "db.createUser({ user: 'user', pwd: 'pass', roles: [ { role: '-dbOwner', db: 'dbName' } ] });"

错误：

14:13:04.538 [ScalaTest-run-running-MongoConnector] DEBUG org.mongodb.driver.cluster - Updating cluster description to  {type=UNKNOWN, servers=[{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING}]
14:13:04.540 [ScalaTest-run-running-MongoConnector] INFO  org.mongodb.driver.cluster - Cluster description not yet available. Waiting for 30000 ms before timing out
14:13:04.563 [cluster-ClusterId{value='59f5d3e0ee179e015809a115', description='null'}-127.0.0.1:27017] DEBUG org.mongodb.driver.connection - Closing connection connectionId{localValue:1}
14:13:04.564 [cluster-ClusterId{value='59f5d3e0ee179e015809a115', description='null'}-127.0.0.1:27017] INFO  org.mongodb.driver.cluster - Exception in monitor thread while connecting to server 127.0.0.1:27017
com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=null, userName='user', source='dbName', password=<hidden>, mechanismProperties={}}
(...)
Caused by: com.mongodb.MongoCommandException: Command failed with error 18: 'Authentication failed.' on server 127.0.0.1:27017. The full response is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18, "codeName" : "AuthenticationFailed" }
    at com.mongodb.connection.CommandHelper.createCommandFailureException(CommandHelper.java:170)
    at com.mongodb.connection.CommandHelper.receiveCommandResult(CommandHelper.java:123)
    at com.mongodb.connection.CommandHelper.executeCommand(CommandHelper.java:32)
    at com.mongodb.connection.SaslAuthenticator.sendSaslStart(SaslAuthenticator.java:117)
14:17:28.944 [cluster-ClusterId{value='59f5d4e8ee179e027923cfdf', description='null'}-127.0.0.1:27017] DEBUG org.mongodb.driver.cluster - Updating cluster description to  {type=UNKNOWN, servers=[{address=127.0.0.1:27017, type=UNKNOWN, state=CONNECTING, exception={com.mongodb.MongoSecurityException: Exception authenticating MongoCredential{mechanism=null, userName='user', source='dbName', password=<hidden>, mechanismProperties={}}}, caused by {com.mongodb.MongoCommandException: Command failed with error 18: 'Authentication failed.' on server 127.0.0.1:27017. The full response is { "ok" : 0.0, "errmsg" : "Authentication failed.", "code" : 18, "codeName" : "AuthenticationFailed" }}}]
14:17:29.451 [cluster-ClusterId{value='59f5d4e8ee179e027923cfdf', description='null'}-127.0.0.1:27017] DEBUG org.mongodb.driver.connection - Closing connection connectionId{localValue:2}

版本：
1. Spark 2.2.0
2. Mongo 3.4
3. Mongo-spark-connector_2.11，版本：2.2.0
4. Scala 2.11.8

修改
我试图加入

.config("spark.mongodb.input.uri", "mongodb://user:pass@localhost:27017/dbName.collectionName")

标志：

？authSource = admin
？authSource = dbOwner
？authSource = dbName
？authenticationDatabase = DBNAME

但它们都不起作用

Answer 1

我遇到了同样的问题，过了一会儿，这个设置可以与authSource一起使用。

首先，请确保您的mongo设置正确（如果只有一台计算机，请跳过此步骤），如果有副本集，请确保它们之间通过日志的通信正常。

在我的设置中，我想从admin数据库中拥有一个admin用户，而在自定义数据库中具有一个自定义用户。此自定义用户具有的唯一权限是dbOwner。我在spark中使用的命令是：

df = sqlContext.read
  .format("com.mongodb.spark.sql")
  .options(uri="mongodb://CUSTOM_USERNAME:PASSWORD@ip.address:27017/?authSource=CUSTOM_DB_NAME", database="CUSTOM_DB_NAME", collection="COLLECTION_NAME")
  .load()

它对我来说很完美。注意事项：

authSource的默认值为admin，因此，如果您根本无法连接，请在admin数据库上以具有权限{role:'root',db:'admin'}的管理员用户身份。那么默认配置应该足够好。
请注意，URI中问号前的斜杠-否则URI无效。
我的mongo连接器版本为2.4.0-也许以前被破坏了。

祝你好运！

更多文档：

Mongo Spark连接器，auth错误

1 个答案: