最近几天,Debian(Google Compute Engine)服务器每天都面临套接字不可用的问题,需要重启tomcat。
Dec 30, 2015 1:16:53 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:220)
at java.lang.Thread.run(Thread.java:745)
在使用netstat -ap
检查的情况下使用时发现GOOGLE :(正在充斥服务器。
tcp 1 0 backend-tomcat:53976 wk-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42251 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45929 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42159 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:55348 wo-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:44973 wa-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42148 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45729 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45721 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42146 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:36557 wq-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45723 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45915 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45295 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:53819 wk-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 0 0 backend-tomcat:45968 wn-in-f95.1e100.n:https ESTABLISHED 5081/java
tcp 1 0 backend-tomcat:41737 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:55132 wm-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:53969 wk-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:40883 wj-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45734 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:40889 wj-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:41834 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
使用lsof -p 5081
检查了流程,下面有大量的行列表
java 5081 tomcat7 1181u IPv4 1959257 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48570->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1182u IPv4 1959420 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48575->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1183u IPv4 1959526 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48577->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1184u IPv4 1959676 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48583->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1185u IPv4 1960099 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48591->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1186u IPv4 1959857 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48589->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1187u IPv4 1960209 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48596->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1188u IPv4 1960352 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49524->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1189u IPv4 1960555 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49526->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1190u IPv4 1960721 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49531->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1191u IPv4 1960962 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49537->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1192u IPv4 1961151 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49539->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1193u IPv4 1961318 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49544->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1194u IPv4 1961557 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49550->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1195u IPv4 1961828 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46380->wl-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1196u IPv4 1962155 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46386->wl-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1197u IPv4 1962356 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46392->wl-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1198u IPv4 1962461 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46393->wl-in-f95.1e100.net:https (CLOSE_WAIT)
从ulimit发现打开文件限制设置为max
open files (-n) 65536
是否归功于GCE的云调试器功能?有没有办法要求Google不要每天让服务器流血?
答案 0 :(得分:2)
感谢您报告。此错误现已修复:https://github.com/GoogleCloudPlatform/cloud-debug-java/commit/057d2f181ebfb5b670e31ecc9c155b38cc495150。
请在format_env_gce.sh
脚本中删除应用程序版本,以确保VM正在获取最新版本的调试器代理。例如:
...format_env_gce.sh --version=2 ...
如果您使用Managed VM,则下次重新部署时它将自动更新为最新版本的调试器代理。
项目的GitHub page提供了最新的二进制文件。