Question

这是我的代码：

  es_write_conf = {
     ¦   "es.nodes" : ES_IP,
     ¦   "es.port" : ES_PORT,
     ¦   "es.resource" : "%s/%s" % (index, doc_type),
     ¦   "es.input.json": "true",
     ¦   # "es.mapping.rich.date": "true"
     ¦   # "es.mapping.id": "guid"
     }

     dstream.foreachRDD(lambda es_rdd: es_rdd.saveAsNewAPIHadoopFile(
     ¦   path="-",
     ¦   outputFormatClass="org.elasticsearch.hadoop.mr.EsOutputFormat",
     ¦   keyClass="org.apache.hadoop.io.NullWritable",
     ¦   valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
     ¦   conf=es_write_conf))

我得到这个警告

WARN EsOutputFormat: Speculative execution enabled for reducer - consider disabling it to prevent data corruption

如何解决此警告？

Answer 1

我以此解决问题

     es_write_conf = {
     ¦   "es.nodes" : ES_IP,
     ¦   "es.port" : ES_PORT,
     ¦   "es.resource" : "%s/%s" % (index, doc_type),
     ¦   "es.input.json": "true",
     ¦   "mapred.reduce.tasks.speculative.execution": "false",
     ¦   "mapred.map.tasks.speculative.execution": "false",
     ¦   # "es.mapping.rich.date": "true"
     ¦   # "es.mapping.id": "guid"
     }

     dstream.foreachRDD(lambda es_rdd: es_rdd.saveAsNewAPIHadoopFile(
     ¦   path="-",
     ¦   outputFormatClass="org.elasticsearch.hadoop.mr.EsOutputFormat",
     ¦   keyClass="org.apache.hadoop.io.NullWritable",
     ¦   valueClass="org.elasticsearch.hadoop.mr.LinkedMapWritable",
     ¦   conf=es_write_conf))

pyspark使用saveAsNewAPIHadoopFile将dstream数据写入es发出警告

1 个答案: