openshift上的tensorflow部署,gunicorn和mod_wsgi的错误

时间:2017-05-21 22:41:32

标签: python openshift redhat gunicorn

我实现了一个基于这个git的小应用程序: gitlab.com/osevg/python-flask-modwsgi

所有我改变的是执行分析上传文件的脚本的wsgi.py(该文件通过单独的基于PHP的应用程序上传,通过http post发送到此Python应用程序)。脚本本身非常简单,在我的本地Windows设备上运行没有任何问题(运行5-6秒,然后通过JSON发回结果)。我正在努力让这个工具在Openshift v3上运行。

import os
from flask import Flask, request, redirect, url_for
from werkzeug.utils import secure_filename

os.environ['TF_CPP_MIN_LOG_LEVEL']='2'
import tensorflow as tf, sys

import json

UPLOAD_FOLDER = 'uploads'
ALLOWED_EXTENSIONS = set(['jpeg'])

application = Flask(__name__)
application.config['UPLOAD_FOLDER'] = UPLOAD_FOLDER
application.config['PROPAGATE_EXCEPTIONS'] = True

@application.route('/', methods=['GET', 'POST'])
def upload_file():
    if request.method == 'POST':
            filename = secure_filename(file.filename)
            file.save(os.path.join(application.config['UPLOAD_FOLDER'], filename))
            # change this as you see fit
            #image_path = sys.argv[1]
            image_path = os.path.join(application.config['UPLOAD_FOLDER'], filename)

            # Read in the image_data
            image_data = tf.gfile.FastGFile(image_path, 'rb').read()

            # Loads label file, strips off carriage return
            label_lines = [line.rstrip() for line
                               in tf.gfile.GFile("retrained_labels.txt")]

            # Unpersists graph from file
            with tf.gfile.FastGFile("retrained_graph.pb", 'rb') as f:
                graph_def = tf.GraphDef()
                graph_def.ParseFromString(f.read())
                _ = tf.import_graph_def(graph_def, name='')

            with tf.Session() as sess:
                # Feed the image_data as input to the graph and get first prediction
                softmax_tensor = sess.graph.get_tensor_by_name('final_result:0')

                predictions = sess.run(softmax_tensor, \
                         {'DecodeJpeg/contents:0': image_data})

                # Sort to show labels of first prediction in order of confidence
                top_k = predictions[0].argsort()[-len(predictions[0]):][::-1]

                results = []

                for node_id in top_k:
                    human_string = label_lines[node_id]
                    score = predictions[0][node_id] * 100
                    # print('%s (score = %.5f)' % (human_string, score))
                    results.append({"guess" : human_string, "stat" : score.tolist()})

                return json.dumps(results)

if __name__ == "__main__":
    application.run(debug=True)

============== 更新: 正好50%的时间工作。成功第一次;第二次失败;交替成功/失败。

失败时,日志显示:"从守护程序进程收到的截断或超大响应头。"来自"由信号9终止。"

---> Running application from Python script (app.py) ...
Server URL         : http://localhost:8080/
Server Root        : /tmp/mod_wsgi-localhost:8080:1030200000
Server Conf        : /tmp/mod_wsgi-localhost:8080:1030200000/httpd.conf
Error Log File     : /dev/stderr (debug)
Request Capacity   : 5 (1 process * 5 threads)
Request Timeout    : 60 (seconds)
Startup Timeout    : 15 (seconds)
Queue Backlog      : 100 (connections)
Queue Timeout      : 45 (seconds)
Server Capacity    : 20 (event/worker), 20 (prefork)
Server Backlog     : 500 (connections)
Locale Setting     : en_US.UTF-8
[Tue May 23 22:06:46.439793 2017] [wsgi:debug] [pid 1:tid 139903948601472] src/server/mod_wsgi.c(8247): mod_wsgi (pid=1): Socket for 'localhost:8080' is '/tmp/mod_wsgi-localhost:8080:1030200000/wsgi.1.0.1.sock'.
[Tue May 23 22:06:46.440087 2017] [wsgi:debug] [pid 1:tid 139903948601472] src/server/mod_wsgi.c(8305): mod_wsgi (pid=1): Listen backlog for socket '/tmp/mod_wsgi-localhost:8080:1030200000/wsgi.1.0.1.sock' is '100'.
[Tue May 23 22:06:46.440465 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24): Starting process 'localhost:8080' with threads=5.
[Tue May 23 22:06:46.440605 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9947): mod_wsgi (pid=24): Setting lang to en_US.UTF-8 for daemon process group localhost:8080.
[Tue May 23 22:06:46.440611 2017] [mpm_event:notice] [pid 1:tid 139903948601472] AH00489: Apache/2.4.18 (Red Hat) mod_wsgi/4.5.15 Python/3.5 configured -- resuming normal operations
[Tue May 23 22:06:46.440623 2017] [mpm_event:info] [pid 1:tid 139903948601472] AH00490: Server built: Jul 13 2016 08:48:18
[Tue May 23 22:06:46.440635 2017] [core:notice] [pid 1:tid 139903948601472] AH00094: Command line: 'httpd (mod_wsgi-express)         -f /tmp/mod_wsgi-localhost:8080:1030200000/httpd.conf -D MOD_WSGI_MULTIPROCESS -D MOD_WSGI_WITH_PROXY_HEADERS -D MOD_WSGI_MPM_ENABLE_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_EVENT_MODULE -D MOD_WSGI_MPM_EXISTS_WORKER_MODULE -D MOD_WSGI_MPM_EXISTS_PREFORK_MODULE -D FOREGROUND'
[Tue May 23 22:06:46.440639 2017] [core:debug] [pid 1:tid 139903948601472] log.c(1546): AH02639: Using SO_REUSEPORT: yes (1)
[Tue May 23 22:06:46.440645 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9960): mod_wsgi (pid=24): Setting locale to en_US.UTF-8 for daemon process group localhost:8080.
[Tue May 23 22:06:46.440745 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24): Python home /opt/app-root.
[Tue May 23 22:06:46.440811 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24): Initializing Python.
[Tue May 23 22:06:46.440993 2017] [mpm_event:debug] [pid 25:tid 139903948596992] event.c(2094): AH02471: start_threads: Using epoll
[Tue May 23 22:06:46.465067 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(10116): mod_wsgi (pid=24): Process 'localhost:8080' logging to 'localhost'.
[Tue May 23 22:06:46.465193 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24): Attach interpreter ''.
[Tue May 23 22:06:46.473206 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24): Imported 'mod_wsgi'.
[Tue May 23 22:06:46.473277 2017] [wsgi:info] [pid 24:tid 139903948601472] mod_wsgi (pid=24, process='localhost:8080', application=''): Loading WSGI script '/tmp/mod_wsgi-localhost:8080:1030200000/handler.wsgi'.
[Tue May 23 22:06:48.916242 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9366): mod_wsgi (pid=24): Starting 5 threads in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916238 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8929): mod_wsgi (pid=24): Enable monitor thread in process 'localhost:8080'.
[Tue May 23 22:06:48.916251 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=24): Starting thread 1 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916252 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8933): mod_wsgi (pid=24): Startup timeout is 15.
[Tue May 23 22:06:48.916255 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8936): mod_wsgi (pid=24): Deadlock timeout is 60.
[Tue May 23 22:06:48.916257 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8939): mod_wsgi (pid=24): Idle inactivity timeout is 0.
[Tue May 23 22:06:48.916259 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8942): mod_wsgi (pid=24): Request time limit is 60.
[Tue May 23 22:06:48.916261 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8945): mod_wsgi (pid=24): Graceful timeout is 15.
[Tue May 23 22:06:48.916263 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8948): mod_wsgi (pid=24): Eviction timeout is 0.
[Tue May 23 22:06:48.916266 2017] [wsgi:debug] [pid 24:tid 139903498942208] src/server/mod_wsgi.c(8951): mod_wsgi (pid=24): Restart interval is 0.
[Tue May 23 22:06:48.916328 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=24): Starting thread 2 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916375 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=24): Starting thread 3 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916414 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=24): Starting thread 4 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916481 2017] [wsgi:info] [pid 24:tid 139903482156800] mod_wsgi (pid=24): Started thread 0 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916496 2017] [wsgi:debug] [pid 24:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=24): Starting thread 5 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916448 2017] [wsgi:info] [pid 24:tid 139903473764096] mod_wsgi (pid=24): Started thread 1 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916510 2017] [wsgi:info] [pid 24:tid 139903465371392] mod_wsgi (pid=24): Started thread 2 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916611 2017] [wsgi:info] [pid 24:tid 139903456978688] mod_wsgi (pid=24): Started thread 3 in daemon process 'localhost:8080'.
[Tue May 23 22:06:48.916514 2017] [wsgi:debug] [pid 24:tid 139903490549504] src/server/mod_wsgi.c(8894): mod_wsgi (pid=24): Enable deadlock thread in process 'localhost:8080'.
[Tue May 23 22:06:48.916628 2017] [wsgi:info] [pid 24:tid 139903448585984] mod_wsgi (pid=24): Started thread 4 in daemon process 'localhost:8080'.
[Tue May 23 23:00:24.575348 2017] [authz_core:debug] [pid 25:tid 139903948330752] mod_authz_core.c(809): [client 10.129.0.1:57546] AH01626: authorization result of Require all granted: granted
[Tue May 23 23:00:24.575385 2017] [authz_core:debug] [pid 25:tid 139903948330752] mod_authz_core.c(809): [client 10.129.0.1:57546] AH01626: authorization result of <RequireAny>: granted
[Tue May 23 23:00:24.575496 2017] [authz_core:debug] [pid 25:tid 139903948330752] mod_authz_core.c(809): [client 10.129.0.1:57546] AH01626: authorization result of Require all granted: granted
[Tue May 23 23:00:24.575507 2017] [authz_core:debug] [pid 25:tid 139903948330752] mod_authz_core.c(809): [client 10.129.0.1:57546] AH01626: authorization result of <RequireAny>: granted
[Tue May 23 23:00:24.576463 2017] [wsgi:debug] [pid 25:tid 139903948330752] src/server/mod_wsgi.c(11650): mod_wsgi (pid=25): Request server was 'localhost|8080'.
[Tue May 23 23:00:24.576578 2017] [wsgi:debug] [pid 24:tid 139903482156800] src/server/mod_wsgi.c(12648): mod_wsgi (pid=24): Server listener address '|8080'.
[Tue May 23 23:00:24.576591 2017] [wsgi:debug] [pid 24:tid 139903482156800] src/server/mod_wsgi.c(12657): mod_wsgi (pid=24): Server listener address '|8080' was found.
[Tue May 23 23:00:24.576610 2017] [wsgi:debug] [pid 24:tid 139903482156800] src/server/mod_wsgi.c(12669): mod_wsgi (pid=24): Connection server matched was 'localhost|8080'.
[Tue May 23 23:00:24.576617 2017] [wsgi:debug] [pid 24:tid 139903482156800] src/server/mod_wsgi.c(12685): mod_wsgi (pid=24): Request server matched was 'localhost|8080'.
[Tue May 23 23:00:24.838739 2017] [mpm_event:debug] [pid 49:tid 139903948596992] event.c(2094): AH02471: start_threads: Using epoll
[Tue May 23 23:00:43.265971 2017] [authz_core:debug] [pid 25:tid 139903948064512] mod_authz_core.c(809): [client 10.131.0.1:47450] AH01626: authorization result of Require all granted: granted
[Tue May 23 23:00:43.266000 2017] [authz_core:debug] [pid 25:tid 139903948064512] mod_authz_core.c(809): [client 10.131.0.1:47450] AH01626: authorization result of <RequireAny>: granted
[Tue May 23 23:00:43.269467 2017] [authz_core:debug] [pid 25:tid 139903948064512] mod_authz_core.c(809): [client 10.131.0.1:47450] AH01626: authorization result of Require all granted: granted
[Tue May 23 23:00:43.269492 2017] [authz_core:debug] [pid 25:tid 139903948064512] mod_authz_core.c(809): [client 10.131.0.1:47450] AH01626: authorization result of <RequireAny>: granted
[Tue May 23 23:00:43.270535 2017] [wsgi:debug] [pid 25:tid 139903948064512] src/server/mod_wsgi.c(11650): mod_wsgi (pid=25): Request server was 'localhost|8080'.
[Tue May 23 23:00:43.270648 2017] [wsgi:debug] [pid 24:tid 139903448585984] src/server/mod_wsgi.c(12648): mod_wsgi (pid=24): Server listener address '|8080'.
[Tue May 23 23:00:43.270669 2017] [wsgi:debug] [pid 24:tid 139903448585984] src/server/mod_wsgi.c(12657): mod_wsgi (pid=24): Server listener address '|8080' was found.
[Tue May 23 23:00:43.270675 2017] [wsgi:debug] [pid 24:tid 139903448585984] src/server/mod_wsgi.c(12669): mod_wsgi (pid=24): Connection server matched was 'localhost|8080'.
[Tue May 23 23:00:43.270680 2017] [wsgi:debug] [pid 24:tid 139903448585984] src/server/mod_wsgi.c(12685): mod_wsgi (pid=24): Request server matched was 'localhost|8080'.
[Tue May 23 23:00:43.792919 2017] [wsgi:error] [pid 25:tid 139903948064512] [client 10.131.0.1:47450] Truncated or oversized response headers received from daemon process 'localhost:8080': /tmp/mod_wsgi-localhost:8080:1030200000/htdocs/
[Tue May 23 23:00:43.856204 2017] [wsgi:info] [pid 1:tid 139903948601472] mod_wsgi (pid=24): Process 'localhost:8080' has died, deregister and restart it.
[Tue May 23 23:00:43.856264 2017] [wsgi:info] [pid 1:tid 139903948601472] mod_wsgi (pid=24): Process 'localhost:8080' terminated by signal 9
[Tue May 23 23:00:43.856270 2017] [wsgi:info] [pid 1:tid 139903948601472] mod_wsgi (pid=24): Process 'localhost:8080' has been deregistered and will no longer be monitored.
[Tue May 23 23:00:43.868568 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70): Starting process 'localhost:8080' with threads=5.
[Tue May 23 23:00:43.868708 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9947): mod_wsgi (pid=70): Setting lang to en_US.UTF-8 for daemon process group localhost:8080.
[Tue May 23 23:00:43.868749 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9960): mod_wsgi (pid=70): Setting locale to en_US.UTF-8 for daemon process group localhost:8080.
[Tue May 23 23:00:43.876257 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70): Python home /opt/app-root.
[Tue May 23 23:00:43.877029 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70): Initializing Python.
[Tue May 23 23:00:43.972394 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(10116): mod_wsgi (pid=70): Process 'localhost:8080' logging to 'localhost'.
[Tue May 23 23:00:43.972537 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70): Attach interpreter ''.
[Tue May 23 23:00:43.986453 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70): Imported 'mod_wsgi'.
[Tue May 23 23:00:43.986556 2017] [wsgi:info] [pid 70:tid 139903948601472] mod_wsgi (pid=70, process='localhost:8080', application=''): Loading WSGI script '/tmp/mod_wsgi-localhost:8080:1030200000/handler.wsgi'.
[Tue May 23 23:00:46.405597 2017] [wsgi:error] [pid 70:tid 139903948601472] test1
[Tue May 23 23:00:46.405883 2017] [wsgi:error] [pid 70:tid 139903948601472] test5
[Tue May 23 23:00:46.406112 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8929): mod_wsgi (pid=70): Enable monitor thread in process 'localhost:8080'.
[Tue May 23 23:00:46.406126 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8933): mod_wsgi (pid=70): Startup timeout is 15.
[Tue May 23 23:00:46.406130 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8936): mod_wsgi (pid=70): Deadlock timeout is 60.
[Tue May 23 23:00:46.406132 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8939): mod_wsgi (pid=70): Idle inactivity timeout is 0.
[Tue May 23 23:00:46.406134 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8942): mod_wsgi (pid=70): Request time limit is 60.
[Tue May 23 23:00:46.406142 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9366): mod_wsgi (pid=70): Starting 5 threads in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406145 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8945): mod_wsgi (pid=70): Graceful timeout is 15.
[Tue May 23 23:00:46.406148 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=70): Starting thread 1 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406148 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8948): mod_wsgi (pid=70): Eviction timeout is 0.
[Tue May 23 23:00:46.406151 2017] [wsgi:debug] [pid 70:tid 139903498942208] src/server/mod_wsgi.c(8951): mod_wsgi (pid=70): Restart interval is 0.
[Tue May 23 23:00:46.406193 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=70): Starting thread 2 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406235 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=70): Starting thread 3 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406283 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=70): Starting thread 4 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406305 2017] [wsgi:info] [pid 70:tid 139903465371392] mod_wsgi (pid=70): Started thread 2 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406330 2017] [wsgi:debug] [pid 70:tid 139903948601472] src/server/mod_wsgi.c(9376): mod_wsgi (pid=70): Starting thread 5 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406382 2017] [wsgi:debug] [pid 70:tid 139903490549504] src/server/mod_wsgi.c(8894): mod_wsgi (pid=70): Enable deadlock thread in process 'localhost:8080'.
[Tue May 23 23:00:46.406428 2017] [wsgi:info] [pid 70:tid 139903482156800] mod_wsgi (pid=70): Started thread 0 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406442 2017] [wsgi:info] [pid 70:tid 139903473764096] mod_wsgi (pid=70): Started thread 1 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406460 2017] [wsgi:info] [pid 70:tid 139903456978688] mod_wsgi (pid=70): Started thread 3 in daemon process 'localhost:8080'.
[Tue May 23 23:00:46.406554 2017] [wsgi:info] [pid 70:tid 139903448585984] mod_wsgi (pid=70): Started thread 4 in daemon process 'localhost:8080'.

更新#2: 第二次,它似乎在以下一行失败:

        predictions = sess.run(softmax_tensor, \
                 {'DecodeJpeg/contents:0': image_data})

SEMI-UPDATE: 作为对任何人阅读的一个FYI,我可以切换回gunicorn并且问题仍然存在(每次使用&#34;启动工作人员使用pid&#34;在TF端并且502错误在发送方)。

最终(可能)更新: 回到mod_wsgi。拥抱疯狂。作为所有hack-ey变通方法中最棘手的,我只是在呼叫端实现了一个虚拟的第二个卷曲,以便第一个调用工作,第二个调用导致重新启动服务器的错误。荒谬,是的;然而,它确实有效,你真的无法区分前端(只是一个起始演示)。

0 个答案:

没有答案