Question

监视Keras（及其生成器）的python进程，以检查是否挂断或无提示失败的最佳方法是什么？

到目前为止，我能够提出2个回调，一个打印进程数（我手动检查日志中的查阅），另一个打印日期时间（这样我可以看到最后一个历元何时结束并手动比较）与当前时间和纪元时间）。不过，这也不是理想的解决方案。

class ProcNumPrinter(Callback):
    """
    Keras callback, that automatically monitors the number of processes (in one line at stdout).
    Useful in cases, when we suspect, that some process did hang.

    """
    def __init__(self):
        super(ProcNumPrinter, self).__init__()

    def on_epoch_end(self, epoch, logs=None):
        print("Multiprocessing reporting {} processes".format(len(multiprocessing.active_children())))


class DatetimePrinter(Callback):
    """
    Keras callback printing the date and time of the last callback activity.
    Useful for checking for hanged processes when keras output used without progressbar.
    """
    def __init__(self):
        super(DatetimePrinter, self).__init__()

    def on_epoch_end(self, epoch, logs=None):
        print(time.strftime("%Y-%m-%d %H:%M"))

Keras挂进程看门狗

0 个答案: