Question

我正在尝试将消息从发布/订阅订户发送到队列中的Elasticsearch索引（如SQL中的行插入）。

我的Elasticsearch索引映射如下：

"properties" : {
    "field1" : {
      "type" : "float"
    },
    "field2" : {
      "type" : "keyword"
    },
    "field3" : {
      "type" : "keyword"
    },
    "field4" : {
      "type" : "keyword"
    },
    "field5" : {
      "type" : "date"
    }

我正在使用以下代码将消息从发布/订阅发送到Elasticsearch：

def collect_data(data):
    es = Elasticsearch( [host], port = port)
    data = data.decode('utf-8')
    data = data.replace("\'", "\"")
    twraw = json.loads(data)
    response = requests.post('http://{}:{}/my_index/_doc/'.format(host,port), data=twraw, headers={"Content-Type": "application/json"})
    print(response.text)

def receive_data(project, subscription_name):
    subscriber = pubsub.SubscriberClient()
    subscription_path = subscriber.subscription_path(project, subscription_name)

    def callback(message):
       print('Received message: {}'.format(message))
       collect_data(message.data)
       message.ack()

    subscription = subscriber.subscribe(subscription_path, callback=callback)
    print('Listening for messages on {}'.format(subscription_path))

    future = subscriber.subscribe(subscription_path, callback=callback)
    try:
        future.result(timeout=10)
    except Exception as e:
       print(
        'Listening for messages on {} threw an Exception: {}'.format(
            subscription_name, e))
       raise


    while True:
       time.sleep(60)

对于每条我无法解决的消息，我都会遇到错误。

{
  "error": {
    "root_cause": [
      {
        "type": "mapper_parsing_exception",
        "reason": "failed to parse"
      }
    ],
    "type": "mapper_parsing_exception",
    "reason": "failed to parse",
    "caused_by": {
      "type": "not_x_content_exception",
      "reason": "Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes"
    }
  },
  "status": 400
}

Answer 1

这对我来说可以将邮件发送到elasticsearch：

def collect_data(data):
    es = Elasticsearch( [host], port = port)
    data = data.decode('utf-8')
    data = data.replace("\'", "\"")
    twraw = json.loads(data)
    es.index(index = 'my_index', body = twraw)

将消息从发布/订阅发送到Elasticsearch

1 个答案: