如何使用多处理加载大型视频数据集？

Question

我正在处理以帧为单位预处理和转换的大型视频数据集。所以我有以下文件夹树：

• 数据

  • 训练集

    • class1
      • 视频1
        • Frame0.png, ..., FrameN.png
      • videoN
        • Frame0.png, ..., FrameN.png
    • class2
      • 视频1
        • Frame0.png, ..., FrameN.png
      • videoN
        • Frame0.png, ..., FrameN.png

  • validation_set

    • class1
      • 视频1
        • Frame0.png, ..., FrameN.png
      • videoN
        • Frame0.png, ..., FrameN.png
    • class2
      • 视频1
        • Frame0.png, ..., FrameN.png
      • videoN
        • Frame0.png, ..., FrameN.png

我需要加载所有这些数据，每次从 class1 文件夹读取视频时，我都会在标签列表中添加 0，每次从 class2 文件夹中读取视频时，我都会在标签中添加 1。

由于视频数以千计，我希望进行多重处理，但我不明白如何做到这一点，以保持视频和标签之间的一致性。

sets = [TRAINING_DIR, VALIDATION_DIR]

for dataset in sets:
    for folder in classes:
        subfolder = os.path.join(dataset, folder)
        if max_videos > -1:
            bar_length = min(max_videos, len(os.listdir(subfolder)))
        else:
            bar_length = len(os.listdir(subfolder))

        bar = ChargingBar('Loading ' + subfolder, max=bar_length)
        bar_interval = int(min(max_videos, len(os.listdir(subfolder)))/bar_length)
        if bar_interval == 0:
            bar_interval = 1

        for index, video_folder_name in enumerate(os.listdir(subfolder)):
            if index == max_videos:
                break
            video_path = os.path.join(subfolder, video_folder_name)
            frames_number = len(os.listdir(video_path))
            frames_interval = int(frames_number / min_video_frames)
            frames_paths = os.listdir(video_path)
            if frames_interval > 0:
                frames_paths = frames_paths[::frames_interval]
            frames = []
            for index, frame_image in enumerate(frames_paths):
                if len(frames) >= min_video_frames:
                    break
                image = np.asarray(cv2.imread(os.path.join(video_path, frame_image)))
                image = cv2.resize(image, (IMAGE_SIZE, IMAGE_SIZE))
                frames.append(image)

            while len(frames) < min_video_frames:
                frames.append(frames[-1])

            if dataset == TRAINING_DIR:
                videos.append(frames)
                if "class1" in folder:
                    labels.append(0)
                else:
                    labels.append(1)
            else:
                validation_videos.append(frames)
                if "class1" in folder:
                    validation_labels.append(0)
                else:
                    validation_labels.append(1)
            
            if index % bar_interval == 0:
                bar.next()
            
        bar.finish()