需要帮助调试kafka延迟以获取高频数据

Question

我有一个IoT设备，每15毫秒将数据流传输到python服务器。 python服务器将数据上传到kafka，另一台服务器使用它。

我有一个基于传感器ID的数据分区键。前几条消息的延迟不到30毫秒，但随后迅速飙升至500毫秒，然后缓慢下降然后重复（下图）。

我在这里的假设是生产者在发送数据之前先对数据进行批处理。我似乎找不到要关闭此设置的设置，以使延迟保持一致。即使我有空白消息，问题似乎也会发生。

这是我的制作人的代码

Jul 09, 2019 3:32:30 PM org.springframework.cloud.localconfig.PropertiesFileResolver findCloudPropertiesFileFromClasspath
WARNING: the properties file spring-cloud-bootstrap.properties contained properties besides spring.cloud.propertiesFile; ignoring
Jul 09, 2019 3:32:30 PM org.springframework.cloud.localconfig.PropertiesFileResolver findCloudPropertiesFileFromClasspath
INFO: derived configuration file name: src/test/resources/local-service-info.properties
Jul 09, 2019 3:32:30 PM org.springframework.cloud.localconfig.PropertiesFileResolver findCloudPropertiesFile
INFO: using configuration file derived from spring-cloud-bootstrap.properties
Jul 09, 2019 3:32:30 PM org.springframework.cloud.localconfig.LocalConfigConnector readFileProperties
INFO: loading service definitions from properties file src/test/resources/local-service-info.properties
Jul 09, 2019 3:32:30 PM org.springframework.cloud.localconfig.LocalConfigConnector readFileProperties
INFO: properties loaded successfully
15:32:30,476 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
15:32:30,476 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
15:32:30,476 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [file:/Users/jjoyce797/git/VOICE-Blocking-Service/app/target/classes/logback.xml]
15:32:30,535 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
15:32:30,536 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
15:32:30,542 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
15:32:30,568 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - This appender no longer admits a layout as a sub-component, set an encoder instead.
15:32:30,568 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - To ensure compatibility, wrapping your layout in LayoutWrappingEncoder.
15:32:30,568 |-WARN in ch.qos.logback.core.ConsoleAppender[STDOUT] - See also http://logback.qos.ch/codes.html#layoutInsteadOfEncoder for details
15:32:30,569 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [net.logstash.logback.appender.LogstashTcpSocketAppender]
15:32:30,577 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [LOGSTASH]
15:32:30,586 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.core.net.ssl.SSLConfiguration] for [ssl] property
15:32:30,586 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.core.net.ssl.KeyStoreFactoryBean] for [trustStore] property
15:32:30,944 |-INFO in net.logstash.logback.appender.LogstashTcpSocketAppender[LOGSTASH] - SSL protocol 'SSL' provider 'SunJSSE version 1.8'
15:32:30,952 |-INFO in net.logstash.logback.appender.LogstashTcpSocketAppender[LOGSTASH] - trust store of type 'JKS' provider 'SUN version 1.8': classpath:cacerts.truststore
15:32:30,952 |-INFO in net.logstash.logback.appender.LogstashTcpSocketAppender[LOGSTASH] - trust manager algorithm 'PKIX' provider 'SunJSSE version 1.8'
15:32:30,954 |-INFO in net.logstash.logback.appender.LogstashTcpSocketAppender[LOGSTASH] - secure random algorithm 'SHA1PRNG' provider 'SUN version 1.8'
15:32:30,976 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
15:32:30,976 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
15:32:30,977 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [LOGSTASH] to Logger[ROOT]
15:32:30,977 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [com.comcast] to INFO
15:32:30,977 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [com.comcast] to false
15:32:30,977 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[com.comcast]
15:32:30,977 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [LOGSTASH] to Logger[com.comcast]
15:32:30,977 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.cxf.interceptor.security.JAASLoginInterceptor] to DEBUG
15:32:30,977 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.apache.cxf.interceptor.security.JAASLoginInterceptor] to false
15:32:30,977 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.apache.cxf.interceptor.security.JAASLoginInterceptor]
15:32:30,977 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [LOGSTASH] to Logger[org.apache.cxf.interceptor.security.JAASLoginInterceptor]
15:32:30,977 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework] to TRACE
15:32:30,977 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.springframework] to false
15:32:30,977 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.springframework]
15:32:30,977 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [LOGSTASH] to Logger[org.springframework]
15:32:30,978 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.cxf] to INFO
15:32:30,978 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.apache.cxf] to false
15:32:30,978 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.apache.cxf]
15:32:30,978 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [LOGSTASH] to Logger[org.apache.cxf]
15:32:30,978 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.springframework.security] to DEBUG
15:32:30,978 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting additivity of logger [org.springframework.security] to false
15:32:30,978 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.springframework.security]
15:32:30,978 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [LOGSTASH] to Logger[org.springframework.security]
15:32:30,978 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
15:32:30,978 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@79e4c792 - Registering current configuration as safe fallback point


  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.1.3.RELEASE)

我一直在上下阅读documentation，并尝试了几种不同的配置，但没有任何帮助。我的服务器和kafka实例在heroku上运行。

感谢您的帮助。