我在同一个文件夹 utils.py
和 preprocessing.py
中有2个python文件。
我在 utils.py 文件中定义了两个函数: preprocess_text, text2seq_generator
我需要在functions
中使用这些preprocessing.py
。但我无法导入和警告说从第19行重新定义未使用的preprocess_text
代码:
utils.py
from __future__ import print_function
from collections import defaultdict
from collections import Counter
from string import punctuation
#import pdb
def filter_text(text):
return ''.join([c for c in text if c not in punctuation]).lower()
def preprocess_text(file_path_src, file_path_tar, max_feats):
f_src = open(file_path_src)
f_tar = open(file_path_tar)
vocab = defaultdict(int)
freq_src = defaultdict(int)
freq_tar = defaultdict(int)
sents_src = [line.rstrip() for line in f_src.readlines()]
sents_tar = [line.rstrip() for line in f_tar.readlines()]
for sent in sents_src:
sent = filter_text(sent)
for word in sent.split():
freq_src[word] += 1
for sent in sents_tar:
for word in sent.split():
freq_tar[word] += 1
freq_sorted_src = Counter(freq_src)
freq_sorted_src = freq_sorted_src.most_common(max_feats)
freq_sorted_tar = Counter(freq_tar)
freq_sorted_tar = freq_sorted_tar.most_common(max_feats)
freq_words_src = map(lambda x:x[0], freq_sorted_src)
freq_words_tar = map(lambda x:x[0], freq_sorted_tar)
vocab_src = dict({v:k for k, v in enumerate(freq_words_src)})
vocab_tar = dict({v:k for k, v in enumerate(freq_words_tar)})
vocab_src["UNK"] = max_feats
vocab_src["<s>"] = max_feats + 1
vocab_src["</s>"] = max_feats + 2
vocab_tar["UNK"] = max_feats
vocab_tar["<s>"] = max_feats + 1
vocab_tar["</s>"] = max_feats + 2
#pdb.set_trace()
return vocab_src, vocab_tar, sents_src, sents_tar
def text2seq_generator(vocab_src, vocab_tar, sents_src, sents_tar):
unk_key = vocab_src["UNK"]
for sent_src, sent_tar in zip(sents_src, sents_tar):
seq_src = map(lambda x:vocab_src.get(x, unk_key), filter_text(sent_src).split())
seq_tar = map(lambda x:vocab_tar.get(x, unk_key), sent_tar.split())
#pdb.set_trace()
yield seq_src, seq_tar
#vs, vt, ss, st = preprocess_text('../data/training.hi-en.en', '../data/training.hi-en.hi', 5000)
#for x,y in text2seq_generator(vs, vt, ss, st):
#print(y)
preprocessing.py文件:
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
import codecs
import pdb
import numpy as np
from utils import preprocess_text,text2seq_generator
错误:
from keras.preprocessing.text import Tokenizer
from keras.preprocessing.sequence import pad_sequences
import codecs
import pdb
import numpy as np
from utils import preprocess_text
Traceback (most recent call last):
File "<ipython-input-8-335a30ab7871>", line 6, in <module>
from utils import preprocess_text
ImportError: cannot import name 'preprocess_text'
警告:
redefinition of unused preprocess_text from line 19
答案 0 :(得分:1)
选项1
您从python IDE运行它(而不是从命令行运行它)。然后IDE(至少在spyder中就是这种情况)将在T.shape = (68690, 4, 4)
环境变量中查找path
脚本。哪个最有可能不存在。
解决方案1
在脚本开头添加以下行:
utils.py
或者只是更改您的工作目录:
import sys
sys.path.insert(0, 'directory_of_utils.py_script')
from utils import preprocess_text, text2seq_generator
选项2
您已安装模块import os
os.chdir('directory_of_utils.py_script')
from utils import preprocess_text, text2seq_generator
。因此,脚本将尝试在此模块中找到函数utils
(BTW:如果通过命令行运行它,则不会发生这种情况)。
解决方案2
重命名preprocess_text
(如果从IDE运行,请添加解决方案1中的行!)。
答案 1 :(得分:1)
您utils
中的某个地方似乎有另一个名为PYTHONPATH
的程序包与您当前的文件冲突。
您可以使用importlib
模块从其路径显式加载模块。
将这些行添加到preprocessing.py
(python 3.3和3.4):
from importlib.machinery import SourceFileLoader
utils = SourceFileLoader("utils", "/path/to/utils.py").load_module()
utils.preprocess_text(...)
对于python3.5 +,您应该使用:
import importlib.util
spec = importlib.util.spec_from_file_location("utils", "/path/to/utils.py")
utils = importlib.util.module_from_spec(spec)
spec.loader.exec_module(utils)
utils.preprocess_text(...)