使用Spark SQL从Cou​​chbase 5.x中读取文档

时间:2019-03-13 08:46:17

标签: apache-spark apache-spark-sql couchbase

我正在尝试通过spark-shell使用Spark SQL读取存储桶的文档。

spark-shell --packages com.couchbase.client:spark-connector_2.11:2.2.0

import org.apache.spark.sql.SparkSession
import com.couchbase.spark.sql._
import com.couchbase.client.java.document.JsonDocument 
import com.couchbase.client.java.query.N1qlQuery
import com.couchbase.client.java.view.ViewQuery
import org.apache.spark.sql.sources.EqualTo 

// Configure Spark
val sparkConf = SparkSession.
      builder().
      appName("KeyValueExample").
      master("local[*]").
      config("spark.couchbase.nodes", "135.x.x.x").
      config("spark.couchbase.username", "Administrator").
      config("spark.couchbase.password", "password").
      config("spark.couchbase.bucket.transaction-datastore", "transaction-datastore").
      getOrCreate()

import sparkConf.implicits._
val sc = sparkConf.sparkContext
val sql = sparkConf.sqlContext

// Create a DataFrame with Schema Inference
val cc = sql.read.couchbase(EqualTo("type", "Credit Card"))

sql.read.couchbase()引发如下错误:

  

19/03/13 13:04:19 WARN端点:[null] [KeyValueEndpoint]:   验证失败。 19/03/13 13:04:19 WARN端点:   [null] [KeyValueEndpoint]:身份验证失败。 19/03/13 13:04:19   WARN端点:重新连接期间出错:   com.couchbase.client.core.endpoint.kv.AuthenticationException:   验证失败           在com.couchbase.client.core.endpoint.kv.KeyValueAuthHandler.checkIsAuthed(KeyValueAuthHandler.java:288)           在com.couchbase.client.core.endpoint.kv.KeyValueAuthHandler.channelRead0(KeyValueAuthHandler.java:173)           在com.couchbase.client.core.endpoint.kv.KeyValueAuthHandler.channelRead0(KeyValueAuthHandler.java:52)           在com.couchbase.client.deps.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)           在com.couchbase.client.deps.io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)           在com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler $ DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:438)           在com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)           在com.couchbase.client.deps.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)           在com.couchbase.client.deps.io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:253)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)           在com.couchbase.client.deps.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)           在com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline $ HeadContext.channelRead(DefaultChannelPipeline.java:1302)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)           在com.couchbase.client.deps.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)           在com.couchbase.client.deps.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)           在com.couchbase.client.deps.io.netty.channel.nio.AbstractNioByteChannel $ NioByteUnsafe.read(AbstractNioByteChannel.java:131)           在com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:646)           在com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:581)           在com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498)           在com.couchbase.client.deps.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:460)           在com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor $ 2.run(SingleThreadEventExecutor.java:131)           在com.couchbase.client.deps.io.netty.util.concurrent.DefaultThreadFactory $ DefaultRunnableDecorator.run(DefaultThreadFactory.java:138)           在java.lang.Thread.run(Thread.java:748)

请帮忙!提前致谢。 :)

1 个答案:

答案 0 :(得分:1)

1-通过“安全性”->“添加用户”创建新的用户/密码

2-为该用户添加访问存储区的权限。 (如果您正在测试环境中,只需添加“完全管理员”权限)

3-在您的代码中,执行以下操作:

val sparkConf = SparkSession.
  builder().
  appName("KeyValueExample").
  master("local[*]").
  config("spark.couchbase.nodes", "135.x.x.x").
  config("spark.couchbase.username", "myUser").
  config("spark.couchbase.password", "myPassword").
  config("spark.couchbase.bucket.myBucketName", ""). //it must have a empty string as a parameter (backward compatibility)
  getOrCreate()