Question

我正在制作网络刮刀我访问谷歌搜索，我得到网页的链接，然后我得到Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_65) scala> val rawblocks=sc.textFile("linkage") rawblocks: org.apache.spark.rdd.RDD[String] = linkage MapPartitionsRDD[3] at textFile at <console>:27 scala> rawblocks.partitions.length res1: Int = 10标签的内容。
问题是，例如，字符串<title>应为"P\xe1gina N\xe3o Encontrada!"。我尝试解码为latin-1，然后编码为utf-8，但它不起作用。

"Página Não Encontrada!"

你能帮助我吗？谢谢！

Answer 1

您可以将检索到的文本变量更改为：

string = u'P\xe1gina N\xe3o Encontrada!'.encode('utf-8')

打印string后，它对我来说似乎没问题。

修改

您是否尝试使用.encode('utf8')而不是添加empresa_str.decode('latin1')？

如：

string = empresa_str.decode('latin_1')

Answer 2

不是最优雅的解决方案，但为我工作：

def remove_all(substr, str):
 index = 0
 length = len(substr)
 while string.find(str, substr) != -1:
    index = string.find(str, substr)
    str = str[0:index] + str[index+length:]
 return str

 def latin1_to_ascii (unicrap):
    xlate={ 'xc3cb3':'o' , 'xc3xa7':'c','xc3xb5':'o',  'xc3xa3':'a',  'xc3xa9':'e',
    'xc0':'A', 'xc1':'A', 'xc2':'A', 'xc3':'A', 'xc4':'A', 'xc5':'A',
    'xc6':'Ae', 'xc7':'C',
    'xc8':'E', 'xc9':'E', 'xca':'E', 'xcb':'E',
    'xcc':'I', 'xcd':'I', 'xce':'I', 'xcf':'I',
    'xd0':'Th', 'xd1':'N',
    'xd2':'O', 'xd3':'O', 'xd4':'O', 'xd5':'O', 'xd6':'O', 'xd8':'O',
    'xd9':'U', 'xda':'U', 'xdb':'U', 'xdc':'U',
    'xdd':'Y', 'xde':'th', 'xdf':'ss',
    'xe0':'a', 'xe1':'a', 'xe2':'a', 'xe3':'a', 'xe4':'a', 'xe5':'a',
    'xe6':'ae', 'xe7':'c',
    'xe8':'e', 'xe9':'e', 'xea':'e', 'xeb':'e',
    'xec':'i', 'xed':'i', 'xee':'i', 'xef':'i',
    'xf0':'th', 'xf1':'n',
    'xf2':'o', 'xf3':'o', 'xf4':'o', 'xf5':'o', 'xf6':'o', 'xf8':'o',
    'xf9':'u', 'xfa':'u', 'xfb':'u', 'xfc':'u',
    'xfd':'y', 'xfe':'th', 'xff':'y',
    'xa1':'!', 'xa2':'{cent}', 'xa3':'{pound}', 'xa4':'{currency}',
    'xa5':'{yen}', 'xa6':'|', 'xa7':'{section}', 'xa8':'{umlaut}',
    'xa9':'{C}', 'xaa':'{^a}', 'xab':'<<', 'xac':'{not}',
    'xad':'-', 'xae':'{R}', 'xaf':'_', 'xb0':'{degrees}',
    'xb1':'{+/-}', 'xb2':'{^2}', 'xb3':'{^3}', 'xb4':'',
    'xb5':'{micro}', 'xb6':'{paragraph}', 'xb7':'*', 'xb8':'{cedilla}',
    'xb9':'{^1}', 'xba':'{^o}', 'xbb':'>>', 
    'xbc':'{1/4}', 'xbd':'{1/2}', 'xbe':'{3/4}', 'xbf':'?',
    'xd7':'*', 'xf7':'/'
    }
    unicrap = remove_all ('\\', unicrap)
    unicrap = remove_all('&amp;', unicrap)
    unicrap = remove_all('u2013', unicrap)

    r = unicrap
    for item,valor in xlate.items():
        #print item, unicrap.find(item)
        r = r.replace(item,valor)
    return r

Python和字符串重音

2 个答案: