Question

When I try to run the following pickle program in aws ubuntu instance, I am getting 'killed' message and it is not generating pickle. However the same works when I try it in my local machine (Mac). I am using python3 to run the program:

import nltk
import random
from nltk.classify.scikitlearn import SklearnClassifier
import pickle
from nltk.tokenize import word_tokenize

documents_f = open("documents.pickle", "rb")
documents = pickle.load(documents_f)
documents_f.close()

word_features5k_f = open("word_features5k.pickle", "rb")
word_features = pickle.load(word_features5k_f)
word_features5k_f.close()


def find_features(document):
    words = word_tokenize(document)
    features = {}
    for w in word_features:
        features[w] = (w in words)
    #print (features)
    return features

featuresets = [(find_features(rev), category) for (rev, category) in documents]
save_featuresets = open("featuresets.pickle","wb")
pickle.dump(featuresets, save_featuresets)
save_featuresets.close()

I believe it might be due to memory issue, because I am using aws free tire. Someone please let me know how to fix this?

Issue:

$ python3 my_pickle.py
Killed

Answer 1

Your EC2 instance is running out of memory, and your process is being killed by the OOM killer.

Upgrade to a larger instance type, or find a way to make your process use less memory. (For example, you may want to consider processing one document at a time instead of loading them all at once.)

Getting Killed error in aws but works in Mac

1 个答案: