Datadog-agent在docker容器输出中出现JMX RMI连接错误

时间:2017-07-05 18:31:00

标签: docker jmx datadog jmxmp

我的datadog-agent正在尝试从其自己的docker容器中存在的内部应用程序中收集JMX(通过RMI)指标,从而拒绝连接。但是,jconsole能够从其自己的docker容器中存在的应用程序中收集度量标准。 datadog-agent存在于自己的容器中。两个容器都存在于同一主机上的同一网络中。有任何想法吗?我查看了其他堆栈溢出问题。

  • 已在自定义jmx.yaml文件/etc/dd-agent/conf.d/jmx.yaml
  • 中尝试了IP地址0.0.0.0以及特定主机地址

Docker Container 0:
*运行输出kafka流指标的my_streams_app *执行途径:

`docker run -d --name my_streams_app  
      -v /var/run/docker.sock:/var/run/docker.sock:ro  
      -v /proc/:/host/proc/:ro  
      -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro  
      -e API_KEY=someapikeyhere  
      -e SD_JMX_ENABLE=yes -p 9998:9998 --network=my_streams_default quay.io/temp/my_streams`
  • jconsole能够获取已发布的指标。

Docker Container 1:
*在容器中运行datadog-agent * Datadog-agent使用JMX默认值(RMI)从上面的容器0中的my_streams_app获取指标。
*两个容器在同一主机(我的笔记本电脑MAC OSX)的同一网络上运行 *能够从docker容器中的datadog-agent内的netcat到另一个容器中的my_streams_app ip和port。使用0.0.0.0和9998,也可以使用特定的IP地址
*命令从容器中运行datadog代理

docker run -v /var/run/docker.sock:/var/run/docker.sock:ro  -v /proc/:/host/proc/:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e LOG_LEVEL=DEBUG -e SD_BACKEND=docker --network=mystreams_default 4b1488e74733
  • 用于通过容器内的datadog jmx收集指标的jmx配置:

    实例:

    • 主机:0.0.0.0    港口:9998    标签:         newTag:my_streams    jmx_url:“service:jmx:rmi:/// jndi / rmi://0.0.0.0:9998 / jmxrmi”    名称:jmx_instance

    docker_images:          - my_streams_app

    init_config:     is_jmx:是的     CONF:          - 包括:             域名:'“kafka.streams”'             bean:'“kafka.streams”:type =“stream-metrics”,client-id =“my_test-1-StreamThread-1”'             属性:                 提交通话费率:                     metric_type:规范                 按提交时间平均:                     metric_type:规范                 提交时间-MAX:                     metric_type:规范                 调查通话费率:                     metric_type:gauge

的JConsole:
*通过以下方式从docker容器0中的my_streams_app收集指标:

jconsole 0.0.0.0:9998  

错误输出:

2017-07-05 20:48:20,236 | ERROR | App | Cannot connect to instance service:jmx:rmi:///jndi/rmi://0.0.0.0:9998/jmxrmi. java.io.IOException: 

Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 0.0.0.0; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)]
java.io.IOException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 0.0.0.0; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)]
    at org.datadog.jmxfetch.Connection.connectWithTimeout(Connection.java:117)
    at org.datadog.jmxfetch.Connection.createConnection(Connection.java:61)
    at org.datadog.jmxfetch.RemoteConnection.<init>(RemoteConnection.java:56)
    at org.datadog.jmxfetch.ConnectionFactory.createConnection(ConnectionFactory.java:29)
    at org.datadog.jmxfetch.Instance.getConnection(Instance.java:162)
    at org.datadog.jmxfetch.Instance.init(Instance.java:173)
    at org.datadog.jmxfetch.App.init(App.java:511)
    at org.datadog.jmxfetch.App.main(App.java:115)
Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 0.0.0.0; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)]
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:370)
    at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:268)
    at org.datadog.jmxfetch.Connection$1.run(Connection.java:86)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:473)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 0.0.0.0; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)]
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:142)
    at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:204)
    at javax.naming.InitialContext.lookup(InitialContext.java:415)
    at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1928)
    at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1895)
    at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
    ... 7 more
Caused by: java.rmi.ConnectException: Connection refused to host: 0.0.0.0; nested exception is: 
    java.net.ConnectException: Connection refused (Connection refused)
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
    at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
    at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
    at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:341)
    at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
    at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:138)
    ... 12 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:580)
    at java.net.Socket.connect(Socket.java:529)
    at java.net.Socket.<init>(Socket.java:429)
    at java.net.Socket.<init>(Socket.java:209)
    at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
    at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:147)
    at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)

rmiregistry已根据Failed to retrieve RMIServer stub

启动

1 个答案:

答案 0 :(得分:0)

解决方案:

  • docker container 0

    • 运行正在输出指标的应用程序
    • 在输出指标的应用程序中创建一个bash脚本。
    • 脚本中的
    • 使用docker容器$ HOSTNAME环境变量的值设置为jmxremote.host和rmi.server.hostname。

    #!/ bin / sh的
        java -Djava.util.logging.config.file = logging.properties -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate = false -Dcom.sun.management.jmxremote.ssl = false -Dcom。 sun.management.jmxremote.rmi.port = 9998 -Dcom.sun.management.jmxremote.port = 9998 -Djava.rmi.server.hostname = $ HOSTNAME -Dcom.sun.management.jmxremote.host = $ HOSTNAME -Dcom。 sun.management.jmxremote.local.only = false -jar /app/my-streams.jar

    • 记得设置chmod + x
    • 设置dockerfile CMD以运行上面的脚本,如下所示:
      CMD [” ./"]
  • docker container 1

    • 运行datadog代理的容器
    • 如上所述在问题中配置jmx.yaml文件。只需将主机设置为应用程序名称
    • 即可
  • 从堆栈溢出帖子中可以获得更多的东西。但上面修复了从datadog-agent发现错误的指标。

以下是运行每个组件的方法:

docker container 0
* my-streams
*在标签中启动相关服务
** mvn clean package docker:build
** docker-compose up

  • 另一个标签旋转了my-streams-app
    ** docker kill my-streams-app
    ** docker rm my-streams-app
    ** docker run -d --name my-streams-app -p 9998:9998 - network = mystreams_default quay.io/myimage/my-streams

码头工人集装箱1
* docker build -t dd-agent-my-streams。
* docker run -v /var/run/docker.sock:/var/run/docker.sock:ro -v / proc /:/ host / proc /:ro -v / sys / fs / cgroup /:/ host / sys / fs / cgroup:ro -e LOG_LEVEL = DEBUG -e SD_BACKEND = docker --network = mystreams_default

将ssh插入docker容器1以验证指标是否有效 * docker ps //找到要登录的容器名称
* docker exec -it / bin / bash
root @ 904e6561cc97:/ #service datadog-agent configcheck
root @ 904e6561cc97:/ #service datadog-agent jmx list_everything
root @ 904e6561cc97:/#service datadog-agent jmx collect