Question

我能够从Sagemaker笔记本中读取和写入S3存储桶中的csv文件，但是当尝试使用csv文件中使用的path方法读取bz2文件时，出现无文件或目录的错误

IOErrorTraceback (most recent call last)
<ipython-input-19-d14d47a702e1> in <module>()
      2 # Create corpus
      3 #%time wiki = WikiCorpus("resources/articles1.xml.bz2", tokenizer_func=spacy_tokenize)
----> 4 wiki = WikiCorpus("s3://sagemakerq/enwiki.xml.bz2", tokenizer_func=spacy_tokenize)

/home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages/gensim/corpora/wikicorpus.pyc in __init__(self, fname, processes, lemmatize, dictionary, filter_namespaces, tokenizer_func, article_min_tokens, token_min_len, token_max_len, lower, filter_articles)
    634 
    635         if dictionary is None:
--> 636             self.dictionary = Dictionary(self.get_texts())
    637         else:
    638             self.dictionary = dictionary

/home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages/gensim/corpora/dictionary.pyc in __init__(self, documents, prune_at)
     82 
     83         if documents is not None:
---> 84             self.add_documents(documents, prune_at=prune_at)
     85 
     86     def __getitem__(self, tokenid):

/home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages/gensim/corpora/dictionary.pyc in add_documents(self, documents, prune_at)
    195 
    196         """
--> 197         for docno, document in enumerate(documents):
    198             # log progress & run a regular check for pruning, once every 10k docs
    199             if docno % 10000 == 0:

/home/ec2-user/anaconda3/envs/amazonei_mxnet_p27/lib/python2.7/site-packages/gensim/corpora/wikicorpus.pyc in get_texts(self)
    676             ((text, self.lemmatize, title, pageid, tokenization_params)
    677              for title, text, pageid
--> 678              in extract_pages(bz2.BZ2File(self.fname), self.filter_namespaces, self.filter_articles))
    679         pool = multiprocessing.Pool(self.processes, init_to_ignore_interrupt)
    680 

IOError: [Errno 2] No such file or directory: 's3://sagemakerq/enwiki.xml.bz2'

Answer 1

好像您正在使用Python gensim包从S3的基于Wiki的数据库转储中构建语料库。 package不支持直接从S3读取。相反，您可以下载文件并使用它。

import boto3
from gensim.corpora.wikicorpus import WikiCorpus

s3 = boto3.client('s3')
s3.download_file('BUCKET_NAME', 'OBJECT_NAME', 'FILE_NAME')
wiki = WikiCorpus('FILE_NAME')

从Sagemaker笔记本中访问S3中的bz2文件

1 个答案: