Elastic Beanstalk根据4xx响应禁用健康状况更改

时间:2016-04-04 09:04:05

标签: rest amazon-web-services elastic-beanstalk

我在Elastic Beanstalk上运行了一个休息api,效果很好。应用方面的一切都运行良好,并按预期工作。

该应用程序是一个rest api,用于查找不同的用户。

example url: http://service.com/user?uid=xxxx&anotherid=xxxx

如果找到具有任何ID的用户,则api会以200 OK回复,否则,会根据404 Not Found回复。 HTTP/1.1状态码保护。

我们的api在很多请求上回答404 Not Found并且弹性beanstalk将我们的环境从OK转移到Warning甚至转移到Degraded的情况并不少见因为这。由于这种降级状态,nginx似乎拒绝了与应用程序的连接。 (看起来它有{30}的阈值进入warning和50%进入degraded状态。这是一个问题,因为应用程序实际上按预期工作,但Elastic Beanstalks默认设置认为这是一个问题,当它真的没有。

有没有人知道在EB中编辑4xx警告和状态转换阈值的方法,还是完全禁用它们?

或者我应该真的做一个症状治疗并停止在这样的电话上使用404 Not Found吗? (我真的不喜欢这个选项)

7 个答案:

答案 0 :(得分:56)

更新:AWS EB最终包含一个内置设置: https://stackoverflow.com/a/51556599/1123355

旧解决方案:潜入EB实例并花费几个小时寻找EB的健康检查守护程序实际上将状态代码报告给EB进行评估的地方,我终于找到了,并提出了一个补丁,可以作为一个完美的解决方法,防止4xx响应代码将环境转变为Degraded环境健康状态,并毫无意义地通过此电子邮件通知您:

Environment health has transitioned from Ok to Degraded. 59.2 % of the requests are erroring with HTTP 4xx.

状态代码报告逻辑位于healthd-appstat内,这是由EB团队开发的Ruby脚本,它不断监视/var/log/nginx/access.log并将状态代码报告给EB,具体在以下路径中:

/opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/healthd-appstat-1.0.1/lib/healthd-appstat/plugin.rb

以下.ebextensions文件将修补此Ruby脚本,以避免将4xx响应代码报告回EB。这意味着EB不会因4xx错误而降低环境健康状况,因为它只是不知道它们正在发生。这也意味着"健康"您的EB环境中的页面将始终显示0响应代码计数{。}}。

4xx

是的,它有点难看,但它完成了工作,至少在EB团队通过某些配置参数提供忽略container_commands: 01-patch-healthd: command: "sudo /bin/sed -i 's/\\# normalize units to seconds with millisecond resolution/if status \\&\\& status.index(\"4\") == 0 then next end/g' /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/healthd-appstat-1.0.1/lib/healthd-appstat/plugin.rb" 02-restart-healthd: command: "sudo /usr/bin/kill $(/bin/ps aux | /bin/grep -e '/bin/bash -c healthd' | /usr/bin/awk '{ print $2 }')" ignoreErrors: true 错误的方法之前。在部署时,将其与您的应用程序一起包含在相对于项目根目录的以下路径中:

4xx

祝你好运,如果这有帮助,请告诉我!

答案 1 :(得分:21)

有专门的健康监控规则定制,称为 忽略HTTP 4xx(附加屏幕快照) 只需启用它,EB不会因4xx错误而降低实例运行状况。 enter image description here

答案 2 :(得分:14)

感谢您的回答Elad Nava,我遇到了同样的问题,您的解决方案对我来说非常合适!

但是,在AWS Support Center中打开故障单后,他们建议我修改nginx配置以忽略健康检查上的4xx,而不是修改ruby脚本。为此,我还必须将配置文件添加到.ebextensions目录,以覆盖默认的nginx.conf文件:

files:
  "/tmp/nginx.conf":
    content: |

      # Elastic Beanstalk Managed

      # Elastic Beanstalk managed configuration file
      # Some configuration of nginx can be by placing files in /etc/nginx/conf.d
      # using Configuration Files.
      # http://docs.amazonwebservices.com/elasticbeanstalk/latest/dg/customize-containers.html
      #
      # Modifications of nginx.conf can be performed using container_commands to modify the staged version
      # located in /tmp/deployment/config/etc#nginx#nginx.conf

      # Elastic_Beanstalk
      # For more information on configuration, see:
      #   * Official English Documentation: http://nginx.org/en/docs/
      #   * Official Russian Documentation: http://nginx.org/ru/docs/

      user  nginx;
      worker_processes  auto;

      error_log  /var/log/nginx/error.log;

      pid        /var/run/nginx.pid;


      worker_rlimit_nofile 1024;

      events {
          worker_connections  1024;
      }

      http {

          ###############################
          # CUSTOM CONFIG TO IGNORE 4xx #
          ###############################

          map $status $loggable {
            ~^[4]  0;
            default 1;
          }

          map $status $modstatus {
            ~^[4]  200;
            default $status;
          }

          #####################
          # END CUSTOM CONFIG #
          #####################

          port_in_redirect off;
          include       /etc/nginx/mime.types;
          default_type  application/octet-stream;


          # This log format was modified to ignore 4xx status codes!
          log_format  main   '$remote_addr - $remote_user [$time_local] "$request" '
                             '$status $body_bytes_sent "$http_referer" '
                             '"$http_user_agent" "$http_x_forwarded_for"';

          access_log  /var/log/nginx/access.log  main;

          log_format healthd '$msec"$uri"'
                             '$modstatus"$request_time"$upstream_response_time"'
                             '$http_x_forwarded_for' if=$loggable;

          sendfile        on;
          include /etc/nginx/conf.d/*.conf;

          keepalive_timeout  1200;

      }

container_commands:
  01_modify_nginx:
    command: cp /tmp/nginx.conf /tmp/deployment/config/#etc#nginx#nginx.conf

虽然这个解决方案非常详细,但我个人认为实施更安全,只要它不依赖于任何AWS专有脚本。我的意思是,如果出于某种原因,AWS决定删除或修改他们的ruby脚本(不管我信不信,他们喜欢在没有事先通知的情况下更改脚本),sed的解决方法很可能会不再工作了。

答案 3 :(得分:2)

以下是基于Adriano Valente's answer的解决方案。我无法使$loggable位工作,虽然跳过404的日志记录似乎是一个很好的解决方案。我只创建了一个定义.conf变量的新$modstatus文件,然后覆盖healthd日志格式以使用$modstatus代替$status。此更改还需要重新启动nginx。这适用于运行Ruby 2.3(Puma)的Elastic Beanstalk的64位Amazon Linux 2016.09 v2.3.1。

# .ebextensions/nginx.conf

files:
  "/tmp/nginx.conf":
    content: |

      # Custom config to ignore 4xx in the health file only
      map $status $modstatus {
        ~^[4]  200;
        default $status;
      }

container_commands:
  modify_nginx_1:
    command: "cp /tmp/nginx.conf /etc/nginx/conf.d/custom_status.conf"
  modify_nginx_2:
    command: sudo sed -r -i 's@\$status@$modstatus@' /opt/elasticbeanstalk/support/conf/webapp_healthd.conf
  modify_nginx_3:
    command: sudo /etc/init.d/nginx restart

答案 4 :(得分:1)

基于Elad Nava's Answer,我认为最好直接使用elasticbeanstalk healthd的控制脚本而不是kill:

container_commands:
    01-patch-healthd:
        command: "sudo /bin/sed -i 's/\\# normalize units to seconds with millisecond resolution/if status \\&\\& status.index(\"4\") == 0 then next end/g' /opt/elasticbeanstalk/lib/ruby/lib/ruby/gems/2.2.0/gems/healthd-appstat-1.0.1/lib/healthd-appstat/plugin.rb"
    02-restart-healthd:
        command: "sudo /opt/elasticbeanstalk/bin/healthd-restart"

最后,在调查此问题时,我注意到健康和apache日志状态代码与使用%s的前者不同,而后者%> s导致它们之间存在差异。我也用这个补丁:

    03-healthd-logs:
        command: sed -i 's/^LogFormat.*/LogFormat "%{%s}t\\"%U\\"%>s\\"%D\\"%D\\"%{X-Forwarded-For}i" healthd/g' /etc/httpd/conf.d/healthd.conf

答案 5 :(得分:1)

我最近遇到了同样被4xx错误轰炸的问题。我尝试了上面列出的建议,但没有任何对我有用。我联系了AWS Support,这是他们的建议,它解决了我的问题。我有一个运行2个实例的Elastic Beanstalk应用程序。

  1. 创建名为.ebextensions
  2. 的文件夹
  3. 在此文件夹中,创建一个名为nginx.config的文件(确保它具有.config扩展名。" .conf"赢了!)
  4. 如果要使用Docker容器部署应用程序,请确保部署包中包含此.ebextensions文件夹。对我来说,捆绑包包括文件夹以及Dockerrun.aws.json
  5. 以下是nginix.config文件的全部内容:

    files:
      "/etc/nginx/nginx.conf":
        content: |
          # Elastic Beanstalk Nginx Configuration File
          user  nginx;
          worker_processes  auto;
    
          error_log  /var/log/nginx/error.log;
    
          pid        /var/run/nginx.pid;
    
          events {
              worker_connections  1024;
          }
    
          http {
    
              # Custom config
              # HTTP 4xx ignored.
              map $status $loggable {
                ~^[4]  0;
                default 1;
              }
    
              # Custom config
              # HTTP 4xx ignored.
              map $status $modstatus {
                ~^[4]  200;
                default $status;
              }
    
              include       /etc/nginx/mime.types;
              default_type  application/octet-stream;
    
              access_log    /var/log/nginx/access.log;
    
              log_format  healthd '$msec"$uri"$modstatus"$request_time"$upstream_response_time"$http_x_forwarded_for';
    
              include       /etc/nginx/conf.d/*.conf;
              include       /etc/nginx/sites-enabled/*;
          }
    

答案 6 :(得分:1)

AWS支持自2018年4月起提供的解决方案:

 files:
  "/tmp/custom-site-nginx.conf":
    mode: "000664"
    owner: root
    group: root
    content: |
       map $http_upgrade $connection_upgrade {
           default        "upgrade";
           ""            "";
       }
       # Elastic Beanstalk Modification(EB_INCLUDE)
       # Custom config
       # HTTP 4xx ignored.
       map $status $loggable {
                ~^[4]  0;
                default 1;
       }


       server {
           listen 80;

         gzip on;
         gzip_comp_level 4;
         gzip_types text/html text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

           if ($time_iso8601 ~ "^(\d{4})-(\d{2})-(\d{2})T(\d{2})") {
               set $year $1;
               set $month $2;
               set $day $3;
               set $hour $4;
           }
           access_log /var/log/nginx/healthd/application.log.$year-$month-$day-$hour healthd if=$loggable;

           access_log    /var/log/nginx/access.log;

           location / {
               proxy_pass            http://docker;
               proxy_http_version    1.1;

               proxy_set_header    Connection            $connection_upgrade;
               proxy_set_header    Upgrade                $http_upgrade;
               proxy_set_header    Host                $host;
               proxy_set_header    X-Real-IP            $remote_addr;
               proxy_set_header    X-Forwarded-For        $proxy_add_x_forwarded_for;
           }
       }

 container_commands:
   override_beanstalk_nginx:
     command: "mv -f /tmp/custom-site-nginx.conf /etc/nginx/sites-available/elasticbeanstalk-nginx-docker-proxy.conf"