Question

我基于存储库[https://github.com/matterport/Mask_RCNN]开发了CNN模型。当我运行程序时（使用cmd：coco.py train --dataset = / DATASET / COCO / 2017 --model = None，我推荐加载语句跳过模型权重加载），该过程经历了模型构建，coco数据集加载，然后开始调用model.train（）。

    # Create model
    if args.command == "train":
        model = modellib.MeshMask_RCNN(mode="training", config=config,
                                  model_dir=args.logs)
    else:
        model = modellib.MeshMask_RCNN(mode="inference", config=config,
                                  model_dir=args.logs)

    # Select weights file to load
    if args.model.lower() == "coco":
        model_path = COCO_MODEL_PATH
    elif args.model.lower() == "last":
        # Find last trained weights
        model_path = model.find_last()
    elif args.model.lower() == "imagenet":
        # Start from ImageNet trained weights
        model_path = IMAGENET_MODEL_PATH()
    else:
        model_path = args.model

    # Load weights
    print("Loading weights ", model_path)
    # model.load_weights(model_path, by_name=True)

    # Train or evaluate
    if args.command == "train":
        # Training dataset. Use the training set and 35K from the
        # validation set, as as in the Mask RCNN paper.
        dataset_train = CocoDataset()
        dataset_train.load_coco(args.dataset, "train", year=args.year, auto_download=args.download)
        if args.year in '2014':
            dataset_train.load_coco(args.dataset, "valminusminival", year=args.year, auto_download=args.download)
        dataset_train.prepare()

        # Validation dataset
        dataset_val = CocoDataset()
        val_type = "val" if args.year in '2017' else "minival"
        dataset_val.load_coco(args.dataset, val_type, year=args.year, auto_download=args.download)
        dataset_val.prepare()

        # Image Augmentation
        # Right/Left flip 50% of the time
        augmentation = imgaug.augmenters.Fliplr(0.5)

        # *** This training schedule is an example. Update to your needs ***
        # Training - Stage 0
        print("Fine tune all layers")

        #  get stuck when invoking this function #
>         model.train(dataset_train, dataset_val,
>                     learning_rate=config.LEARNING_RATE,
>                     epochs=160,
>                     layers='all',
>                     augmentation=augmentation)

在model.train（）中，它开始从磁盘加载图像，并且内存使用开始增加到约80GB，然后所有进度都卡住了（没有培训消息，并且Cpu / Gpu使用率很低）。我暂停了一下，发现程序在multiprocessing / pool.py中的404〜406行之间循环。

    @staticmethod
    def _handle_workers(pool):
        thread = threading.current_thread()

        # Keep maintaining workers until the cache gets drained, unless the pool
        # is terminated.
404     while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
405         pool._maintain_pool()
406         time.sleep(0.1)
        # send sentinel to stop workers
        pool._taskqueue.put(None)
        util.debug('worker handler exiting')

这是否意味着有些资源无法满足需求，所以卡住了？我是keras和tensorflow的新手。有人可以帮忙吗？谢谢。

修改：当我追踪时，我发现了程序卡住的确切语句。

# tensorflow_core/python/client/session.py
class _Callable(object):

  def __init__(self, session, callable_options):
    self._session = session
    self._handle = None
    options_ptr = tf_session.TF_NewBufferFromString(
        compat.as_bytes(callable_options.SerializeToString()))
    try:
>     slef._handle = tf_session.TF_SessionMakeCallable(
>         session._session, options_ptr)

    finally:
      tf_session.TF_DeleteBuffer(options_ptr)

Answer 1

确保您正在使用tenorflow gpu：

import tensorflow-gpu

另外，为张量流会话添加设备

with tf.device('/gpu:0'):

Answer 2

实际上，它没有卡住，只是消耗了太多时间。我没有意识到我正在构建的模型有多大。我以为它卡住了，因为在打印“ epoch 1/160”之后，tf花了将近一个小时的时间才能准备好进行操作（我意识到在将其运行一整夜之后）。

模型本身绝对不能训练，并且之后会引发OOM错误，因此我需要重新设计模型。对不起，我的错。

（Keras）我的CNN模型训练进度陷入困境

2 个答案: