脚本

Question

我有一个在lambda服务上使用NLTK的脚本。我使用管道来自动化所有开发步骤。当在GitHub存储库上发生新的提交时，AWS CodeBuild会处理项目并在我的Lambda函数上实现它。

脚本

环境：Python 3.6.5
将nltk与停用词和wordnet软件包一起使用

我将以下解决方案用于我的代码：Installing NLTK/WORDNET on AWS Lambda via CodeBuild

version: 0.2
phases:
 install:
   commands:
     - echo "install step"
     - apt-get update
     - apt-get install zip -y
     - apt-get install python3-pip -y
     - pip install --upgrade pip
     - pip install --upgrade awscli
     # Define directories
     - export HOME_DIR=`pwd`
     - export NLTK_DATA=$HOME_DIR/nltk_data
 pre_build:
   commands:
     - echo "pre_build step"
     - cd $HOME_DIR
     - virtualenv venv
     - . venv/bin/activate
     # Install modules
     - pip install -U requests
     # NLTK download
     - pip install -U nltk
     - python -m nltk.downloader -d $NLTK_DATA wordnet stopwords
     - pip freeze > requirements.txt
 build:
   commands:
     - echo 'build step'
     - cd $HOME_DIR
     - mv $VIRTUAL_ENV/lib/python3.6/site-packages/* .
     - sudo zip -r9 algo.zip .
     - aws s3 cp --recursive --acl public-read ./ s3://hilightalgo/
     # Put the zip on the lambda function
     - aws lambda update-function-code --function-name arn:aws:lambda:eu-west-3:671560023774:function:LaunchHilight --zip-file fileb://algo.zip
 post_build:
   commands:
     - echo "Build: end"

不同的步骤效果很好。没有错误，但是当我尝试使用Lambda函数时，似乎没有nltk数据。参见下面的lambda执行结果：

{"errorMessage":"\n**********************************************************************\n Resource \u001b[93mstopwords\u001b[0m not found.\n Please use the NLTK Downloader to obtain the resource:\n\n \u001b[31m>>> import nltk\n >>> nltk.download('stopwords')\n \u001b[0m\n Attempted to load \u001b[93mcorpora/stopwords\u001b[0m\n\n Searched in:\n - '/home/sbx_user1060/nltk_data'\n - '/var/lang/nltk_data'\n - '/var/lang/share/nltk_data'\n - '/var/lang/lib/nltk_data'\n - '/usr/share/nltk_data'\n - '/usr/local/share/nltk_data'\n - '/usr/lib/nltk_data'\n - '/usr/local/lib/nltk_data'\n**********************************************************************\n","errorType":"LookupError","stackTrace":[" File \"/var/task/lambda_function.py\", line 13, in lambda_handler\n return preprocessing.find_sentences('twitter.txt', 'english')\n"," File \"./hilight_aglo_v2/preprocessing.py\", line 100, in find_sentences\n (data, data_stopwords) = sentence_tokenize(file, language)\n"," File \"./hilight_aglo_v2/preprocessing.py\", line 52, in sentence_tokenize\n stop_words = set(stopwords.words(language))\n"," File \"/var/task/nltk/corpus/util.py\", line 123, in __getattr__\n self.__load()\n"," File \"/var/task/nltk/corpus/util.py\", line 88, in __load\n raise e\n"," File \"/var/task/nltk/corpus/util.py\", line 83, in __load\n root = nltk.data.find('{}/{}'.format(self.subdir, self.__name))\n"," File \"/var/task/nltk/data.py\", line 699, in find\n raise LookupError(resource_not_found)\n"]}

我不知道为什么lambda找不到nltk数据。有人有解决我问题的想法吗？

Answer 1

根据错误消息，NLTK在这些目录中搜索语料库：

Searched in:
 - '/home/sbx_user1060/nltk_data'
 - '/var/lang/nltk_data'
 - '/var/lang/share/nltk_data'
 - '/var/lang/lib/nltk_data'
 - '/usr/share/nltk_data'
 - '/usr/local/share/nltk_data'
 - '/usr/lib/nltk_data'
 - '/usr/local/lib/nltk_data'

但是，在Lambda执行环境中，对文件系统的访问受到一定限制；这些甚至可能不存在，更不用说代码可读了。此外，您的代码（您创建的.zip存档）被提取到/var/task中。基本上就是主目录。

幸运的是，it seems可以通过设置环境变量让nltk知道在哪里寻找语料库。如果我正确理解了构建过程，则可以将NLTK语料库捆绑到子目录nltk_data中，该子目录位于python代码和所需的库旁边。因此，在Lambda执行环境中，可以在/var/task/nltk_data上找到它。

因此，请尝试在CodeBuild过程结束时为函数设置NLTK_DATA环境变量：

aws lambda update-function-configuration \
--function-name arn:aws:lambda:eu-west-3:671560023774:function:LaunchHilight \
--environment 'Variables={NLTK_DATA=/var/task/nltk_data}'

lambda找不到通过AWS CodeBuild下载的NLTK数据

脚本

1 个答案: