下载内容时出现问题 - 某些数据集

时间:2018-01-30 03:18:01

标签: python-3.x computer-vision dataset

我现在正在学习视频识别,并希望使用数据集 - 某些东西 - 由二十亿(https://www.twentybn.com/datasets/something-something)发布。

我按照https://github.com/TwentyBN/twentybn-dl中的下载和安装提示,告诉我在虚拟环境中下载数据集。

当进程达到'extract-chunks'步骤时,出现了错误,如

Using: '/home/hanzp/sth_sth' as storage.
Will now extract chunks for: 'something-something'
100%|████████████████████████████████▉| 5.33M/5.33M [19:19<00:00, 19.4Kimages/s]Exception in thread background thread for pid 30664:
Traceback (most recent call last):
  File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
  File "/home/hanzp/twentybn-dl/lib/python3.5/site-packages/sh.py", line 1540, in wrap
    fn(*args, **kwargs)
  File "/home/hanzp/twentybn-dl/lib/python3.5/site-packages/sh.py", line 2459, in background_thread
    handle_exit_code(exit_code)
  File "/home/hanzp/twentybn-dl/lib/python3.5/site-packages/sh.py", line 2157, in fn
    return self.command.handle_command_exit_code(exit_code)
  File "/home/hanzp/twentybn-dl/lib/python3.5/site-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_2: 

  RAN: /bin/tar xvz

  STDOUT:
20bn-something-something-v1/16296/00045.jpg
20bn-something-something-v1/16296/00019.jpg
20bn-something-something-v1/16296/00003.jpg
20bn-something-something-v1/16296/00062.jpg
20bn-something-something-v1/16296/00057.jpg
20bn-something-something-v1/16296/00014.jpg
20bn-something-something-v1/16296/00040.jpg
20bn-something-something-v1/16296/00011.jpg
20bn-something-something-v1/16296/00021.jpg
20bn-something-something-v1/16296/00008.jpg
20bn-something-something-v1/16296/00027.jpg
20bn-something-something-v1/16296/00028.jpg
20bn-something-something-v1/16296/00050.jpg
20bn-something-something-v1/16296/00034.jpg
20bn-something-something-v1/16296/00063.jpg
20bn-something-something-v1/16296/00007.jpg
20bn-something-something-v1/16296/00009.jpg
20... (137762091 more, please see e.stdout)

  STDERR:
/bin/tar: 20bn-something-something-v1/27730:无法 mkdir: 过多的连接
/bin/tar: 20bn-something-something-v1/27730/00043.jpg:无法 open: 没有那个文件或目录
/bin/tar: 20bn-something-something-v1/27730:无法 mkdir: 过多的连接
/bin/tar: 20bn-something-something-v1/27730/00066.jpg:无法 open: 没有那个文件或目录
/bin/tar: 20bn-something-something-v1/27730:无法 mkdir: 过多的连接
/bin/tar: 20bn-something-something-v1/27730/00049.jpg:无法 open: 没有那个文件或目录
/bin/tar: 20bn-something-something-v1/27730:无法 mkdir: 过多的连接
/bin/tar: 20bn-something-something-v1/27730/00001.jpg:无法 open: 没有那个文件或目录
/bin/tar: 20bn-something-something-v1/27730:无法 mkdir... (271684964 more, please see e.stderr)

100%|█████████████████████████████████| 5.33M/5.33M [19:26<00:00, 4.57Kimages/s]
Traceback (most recent call last):
  File "/home/hanzp/twentybn-dl/bin/twentybn-dl", line 6, in <module>
    main()
  File "/home/hanzp/twentybn-dl/lib/python3.5/site-packages/twentybn_dl/cli.py", line 108, in main
    extract_chunks(d)
  File "/home/hanzp/twentybn-dl/lib/python3.5/site-packages/twentybn_dl/cli.py", line 49, in extract_chunks
    d.extract_chunks()
  File "/home/hanzp/twentybn-dl/lib/python3.5/site-packages/twentybn_dl/schema.py", line 100, in extract_chunks
    extract_chunks(self.chunk_paths, self.jpegs, self.storage)
  File "/home/hanzp/twentybn-dl/lib/python3.5/site-packages/twentybn_dl/extract.py", line 31, in extract_chunks
    for line in process:
  File "/home/hanzp/twentybn-dl/lib/python3.5/site-packages/sh.py", line 863, in next
    self.wait()
  File "/home/hanzp/twentybn-dl/lib/python3.5/site-packages/sh.py", line 792, in wait
    self.handle_command_exit_code(exit_code)
  File "/home/hanzp/twentybn-dl/lib/python3.5/site-packages/sh.py", line 815, in handle_command_exit_code
    raise exc
sh.ErrorReturnCode_2: 

  RAN: /bin/tar xvz

  STDOUT:
20bn-something-something-v1/16296/00045.jpg
20bn-something-something-v1/16296/00019.jpg
20bn-something-something-v1/16296/00003.jpg
20bn-something-something-v1/16296/00062.jpg
20bn-something-something-v1/16296/00057.jpg
20bn-something-something-v1/16296/00014.jpg
20bn-something-something-v1/16296/00040.jpg
20bn-something-something-v1/16296/00011.jpg
20bn-something-something-v1/16296/00021.jpg
20bn-something-something-v1/16296/00008.jpg
20bn-something-something-v1/16296/00027.jpg
20bn-something-something-v1/16296/00028.jpg
20bn-something-something-v1/16296/00050.jpg
20bn-something-something-v1/16296/00034.jpg
20bn-something-something-v1/16296/00063.jpg
20bn-something-something-v1/16296/00007.jpg
20bn-something-something-v1/16296/00009.jpg
20... (137762091 more, please see e.stdout)

  STDERR:
/bin/tar: 20bn-something-something-v1/27730:无法 mkdir: 过多的连接
/bin/tar: 20bn-something-something-v1/27730/00043.jpg:无法 open: 没有那个文件或目录
/bin/tar: 20bn-something-something-v1/27730:无法 mkdir: 过多的连接
/bin/tar: 20bn-something-something-v1/27730/00066.jpg:无法 open: 没有那个文件或目录
/bin/tar: 20bn-something-something-v1/27730:无法 mkdir: 过多的连接
/bin/tar: 20bn-something-something-v1/27730/00049.jpg:无法 open: 没有那个文件或目录
/bin/tar: 20bn-something-something-v1/27730:无法 mkdir: 过多的连接
/bin/tar: 20bn-something-something-v1/27730/00001.jpg:无法 open: 没有那个文件或目录
/bin/tar: 20bn-something-something-v1/27730:无法 mkdir... (271684964 more, please see e.stderr)

也许我在我的ubuntu中使用中文环境。中文部分是

不能mkdir:连接太多

无法打开:没有这样的文件或目录

实际上我已成功安装了6万个视频,但视频总数为10万。

有人知道这个数据集遇到了同样的问题吗?或者任何人都知道如何处理这个错误?

非常感谢

1 个答案:

答案 0 :(得分:0)

我自己已经解决了这个问题。我的翻译可能是错的,实际的中文部分是'不能mkdir:太多的链接'。这是我的ubuntu系统中每个文件夹中只允许60k子文件夹的关键,所以发生了这个错误。我使用另一台计算机并成功下载并提取了数据集。