当“数据流速率超过物源记录速率”时,HandleHttpRequest失败并发生SERVICE_UNAVAILABLE

时间:2016-05-11 10:46:18

标签: hortonworks-data-platform apache-nifi httpcontent

我使用jMeter进行批量测试,他使用HandleHttpRequest向处理器NIFI发送多个HTTP请求(GET)并发送到Topic Kafka。

问题是StandardHTTPContextMap返回SERVICE_UNAVAILABLE错误,当数据流的速率超过出处记录率但似乎不确定时会发生这种情况。

任何人都有任何想法?我删除了部分日志:

2016-05-05 15:12:14,064 WARN [Timer-Driven Process Thread-7] 
o.a.n.p.PersistentProvenanceRepository The rate of the dataflow is exceeding the provenance recording rate. Slowing down flow to accommodate. Currently, there are 96 journal files (533328812 bytes) and threshold for blocking is 80 (1181116006 bytes)

2016-05-05 15:12:20,310 INFO [Provenance Repository Rollover Thread-2] 
o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (46096 records) into single Provenance Log File ./provenance_repository/8913710.prov in 43254 milliseconds

2016-05-05 15:12:20,314 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 65422 records

2016-05-05 15:12:20,398 INFO [Timer-Driven Process Thread-7]   
o.a.n.p.PersistentProvenanceRepository Provenance Repository has now caught up with rolling over journal files. Current number of journal files to be rolled over is 80

2016-05-05 15:12:20,399 INFO [Timer-Driven Process Thread-7] 
o.a.n.p.PersistentProvenanceRepository Created new Provenance Event Writers for events starting with ID 9190418

2016-05-05 15:12:21,422 INFO [qtp1693512967-121] 
o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=3858f0ad-b165-427b-a460-67fbf7cff0d8] Sending back a SERVICE_UNAVAILABLE response to 172.26.60.27; request was GET 172.26.60.27

1 个答案:

答案 0 :(得分:3)

在分析中,您看到的HTTP响应来自HttpContextMap [1]是正确的。特别是“请求到期”'属性。当请求已在Map中超过配置的金额时,它将自动回复SERVICE_UNAVAILABLE。

我对你的问题的猜测是,NiFi花费太长时间来处理你提交的所有请求,导致Provenance Repo强制翻身,这是一个"停止世界"事件。所以你停止处理任何数据6秒(导致请求到期)。

假设你不想接受随机的6秒"停止世界"事件并且不了解您的流量或配置,您基本上需要缩放或调整流量。有两种选择:

  • 扩展到更大的节点或更多节点
  • 处理更大的FlowFiles而不是许多FlowFiles(有助于加快Provenance的速度)
  • 将originty repo推送到自己的磁盘/将其推送到多个磁盘

[1] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.http.StandardHttpContextMap/index.html