Question

我正在尝试为融合的kafka平台设置mongo db源连接器[https://www.confluent.io/hub/mongodb/kafka-connect-mongodb]。使用Mongo DB Atlas的连接URI时，我能够成功建立mongo db和kafka之间的流。当我在Azure kubernetes集群上运行mongo db时出现问题。我创建了具有3个副本的mongo db statefulset，并且已通过负载均衡器将mongo db服务公开到了Internet。我可以使用robo 3T连接到公开IP上的mongo数据库，并执行CRUD操作。现在，当我使用在kubernetes中运行的mongo db的连接URI看起来像“ mongodb：// load-balance-ip：27017 / test？ssl = false＆authSource = admin＆retryWrites = true＆w = majority”时，出现以下错误kafka连接日志

INFO Failed to resume change stream: The $changeStream stage is only supported on replica sets 40573 (com.mongodb.kafka.connect.source.MongoSourceTask:253)

Mongo数据库有状态集yml看起来像这样

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: eic-mongo-mongodb
spec:
  serviceName: eic-mongo-mongodb
  replicas: 3
  selector:
    matchLabels:
      app: eic-mongo-mongodb
  template:
    metadata:
      labels:
        app: eic-mongo-mongodb
        selector: eic-mongo-mongodb
    spec:
      affinity:
        # Try to put each ES data node on a different node in the K8s cluster
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - eic-mongo-mongodb
                topologyKey: kubernetes.io/hostname
      containers:
        - name: eic-mongo-mongodb
          image: mongo:4.0.8
          resources:
            limits:
              cpu: 500m
              memory: "1Gi"
            requests:
              cpu: 500m
              memory: "1Gi"
          volumeMounts:
            - name: mongo-volume
              mountPath: /data/db
  volumeClaimTemplates:
    - metadata:
        name: mongo-volume
      spec:
        accessModes:
          - ReadWriteOnce
        volumeMode: Filesystem
        resources:
          requests:
            storage: 3Gi

而mongo DB服务yml看起来像这样

apiVersion: v1
kind: Service
metadata:
  name: eic-mongo-mongodb
 ports:
  - name: "27017"
    nodePort: 31683
    port: 27017
    protocol: TCP
    targetPort: 27017
  selector:
    app: eic-mongo-mongodb
  type: LoadBalancer

有人可以让我知道我在哪里犯错吗？

Answer 1

我认为您需要以正确的方式设置mongodb集群。请遵循this，使用运算符在kuberntes上部署mongodb副本集。如果要使用statefulset进行设置，请仅遵循this

Answer 2

我正在尝试使用 Mongo DB atlas URI 和融合的 kafka 平台连接器设置 mongo db 源连接器和接收器连接器。
Sink 连接器已成功建立。
但是当我尝试制作 Source 连接器时，出现了问题。
你能找到我的问题吗？

这是 http://localhost:8083/connectors

的 POST json 数据

{"name": "mongo-source-cashRecord",
   "config": {
     "tasks.max":"1",
     "connector.class":"com.mongodb.kafka.connect.MongoSourceConnector",
     "topic.prefix":"mymongo_joy_",
     "key.converter":"org.apache.kafka.connect.json.JsonConverter",
     "key.converter.schemas.enable":false,
     "value.converter":"org.apache.kafka.connect.json.JsonConverter",
     "value.converter.schemas.enable":false,
     "connection.uri":"mongodb+srv://userid:password@clusterjoy.iz9ag.mongodb.net/testdb?retryWrites=true&w=majority",
     "database":"testdb",
     "collection":"cashRecord",
     "pipeline": "[{\"$match\":{\"operationType\":{\"$in\":[\"insert\",\"update\",\"replace\"]}}}]"
}}

kafka mongo db源连接器，在kubernetes上运行mongo db

2 个答案: