Question

我需要使用python3和nltk对来自意大利维基的文本进行规范化，我遇到了一个问题。大多数单词都没问题，但有些单词映射不正确，更确切地说 - 一些符号。

例如：

'fruibilit \ xe3'，'n \ xe2 \ xba'，'citt \ xe3'

我确信这个问题出现在像à，è这样的符号中。

代码：

# coding: utf8
import os

from nltk import corpus, word_tokenize, ConditionalFreqDist


it_sw_plus = corpus.stopwords.words('italian') + ['doc', 'https']
#it_folder_names = ['AA', 'AB', 'AC', 'AD', 'AE', 'AF']
it_path = os.listdir('C:\\Users\\1\\projects\\i')
it_corpora = []

def normalize(raw_text):
    tokens = word_tokenize(raw_text)
    norm_tokens = []
    for token in tokens:
        if token not in it_sw_plus and token.isalpha():
            token = token.lower().encode('utf8')
            norm_tokens.append(token)
    return norm_tokens

for folder_name in it_path:
    path_to_files = 'C:\\Users\\1\\projects\\i\\%s' % (folder_name)
    files_list = os.listdir(path_to_files)
    for file_name in files_list:
        file_path = path_to_files + '\\' + file_name
        text_file = open(file_path)
        raw_text = text_file.read().decode('utf8')
        norm_tokens = normalize(raw_text)
        it_corpora.append(norm_tokens)
    print(it_corpora)

如何解决此问题？我正在使用Win7（rus）。

当我尝试这段代码时：

import io

with open('C:\\Users\\1\\projects\\i\\AA\\wiki_00', 'r', encoding='utf8') as fin:
    for line in fin:
        print (line)

在PowerShell中：

    <doc id="2" url="https://it.wikipedia.org/wiki?curid=2" title="Armonium">

Armonium



Traceback (most recent call last):
  File "i.py", line 5, in <module>
    print (line)
  File "C:\Python35-32\lib\encodings\cp866.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position 3: character maps to <undefined>

在Python命令行中：

<doc id="2" url="https://it.wikipedia.org/wiki?curid=2" title="Armonium">

Armonium



Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\1\projects\i.py", line 5, in <module>
    print (line)
  File "C:\Python35-32\lib\encodings\cp866.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position
3: character maps to <undefined>

当我尝试请求时：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python35-32\lib\encodings\cp866.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_map)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\u2013' in position
90: character maps to <undefined>

Answer 1

如果您知道编码，请在python2

中尝试指定编码时的编码

import io
with io.open(filename, 'r', encoding='latin-1') as fin:
    for line in fin:
        print line # line should be encoded as latin-1

但在您的情况下，您发布的文件不是latin1文件，而是utf8文件，python3：

>>> import urllib.request
>>> url = 'https://raw.githubusercontent.com/GiteItAwayNow/TrueTry/master/it'
>>> response = urllib.request.urlopen(url)
>>> data = response.read()
>>> text = data.decode('utf8')
>>> print (text) # this prints the file perfectly.

阅读＆＃39; utf8＆＃39; python2中的文件：

import io
with io.open(filename, 'r', encoding='utf8') as fin:
    for line in fin:
        print (line) # line should be encoded as utf8

阅读＆＃39; utf8＆＃39;文件，在python3中：

with open(filename, 'r', encoding='utf8') as fin:
    for line in fin:
        print (line) # line should be encoded as utf8

作为一种好的做法，在处理文本数据时，尽可能尝试使用unicode和python3。请看看

此外，如果您没有安装此模块以在Windows控制台上打印utf8，您应该尝试一下：

pip install win-unicode-console

或者下载：https://pypi.python.org/packages/source/w/win_unicode_console/win_unicode_console-0.4.zip然后python setup.py install

不能正确表达一些意大利语单词

1 个答案: