Question

我正在尝试学习如何使用Twitter数据进行一些文本分析。我在创建Term Frequency Matrix时遇到了问题。我用西班牙语文本（带有特殊字符）创建语料库，没有任何问题。

然而，当我创建Term Frequency Matrix（使用quanteda或tm库）时，西班牙语字符不会按预期显示（而不是看canción，我看到canciÃ³n）。

有关如何让Term Frequency Matrix以正确的字符存储文字的任何建议？

感谢您的帮助。

作为一个注释：我更喜欢使用quanteda库，因为最终我将创建一个wordcloud，我想我更好地理解这个库的方法。我也在使用Windows机器。

我尝试过编码（tw2）＆lt; - “UTF-8”没有运气。

library(dplyr)
library(tm)
library(quanteda)

#' Creating a character with special Spanish characters:
tw2 <- "RT @None: Enmascarados, si masduro chingán a tarek. Si quieres ahora, la aguantas canción  . https://t."


#Cleaning the tweet, removing special punctuation, numbers http links, 
extra spaces:
clean_tw2 <- tolower(tw2)
clean_tw2 = gsub("&amp", "", clean_tw2)
clean_tw2 = gsub("(rt|via)((?:\\b\\W*@\\w+)+)", "", clean_tw2)
clean_tw2 = gsub("@\\w+", "", clean_tw2)
clean_tw2 = gsub("[[:punct:]]", "", clean_tw2)
clean_tw2 = gsub("http\\w+", "", clean_tw2)
clean_tw2 = gsub("[ \t]{2,}", "", clean_tw2)
clean_tw2 = gsub("^\\s+|\\s+$", "", clean_tw2) 

# creates a vector with common stopwords, and other words which I want removed.
myStopwords <- c(stopwords("spanish"),"tarek","vez","ser","ahora")
clean_tw2 <- (removeWords(clean_tw2,myStopwords))

# If we print clean_tw2 we see that all the characters are displayed as expected.
clean_tw2

#'Create Corpus Using quanteda library
corp_quan<-corpus(clean_tw2)
# The corpus created via quanteda, displays the characters as expected.
corp_quan$documents$texts

#'Create Corpus Using TD library
corp_td<-Corpus(VectorSource(clean_tw2))
#' Remove common words from spanish from the Corpus.
#' If we inspect the corp_td, we see that the characters and words are displayed correctly
inspect(corp_td)

# Create the DFM with quanteda library.
tdm_quan<-dfm(corp_quan)
# Here we see that the spanish characters are displayed incorrectly for Example: canción = canciÃ³n
tdm_quan

# Create the TDM with TD library
tdm_td<-TermDocumentMatrix(corp_td)

# Here we see that the Spanish characters are displayed incorrectly (e.g. canción = canciÃ), and "si" is missing.
tdm_td$dimnames$Terms

Answer 1

让我猜一下......你在使用Windows吗？在macOS上它工作正常：

clean_tw2
## [1] "enmascarados si masduro chingán   si quieres   aguantas canción"
Encoding(clean_tw2)
## [1] "UTF-8"
dfm(clean_tw2)
## Document-feature matrix of: 1 document, 7 features (0% sparse).
## 1 x 7 sparse Matrix of class "dfm"
##        features
## docs    enmascarados si masduro chingán quieres aguantas canción
##   text1            1  2       1       1       1        1       1

我的系统信息：

sessionInfo()
# R version 3.4.4 (2018-03-15)
# Platform: x86_64-apple-darwin15.6.0 (64-bit)
# Running under: macOS High Sierra 10.13.4
# 
# Matrix products: default
# BLAS: /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
# LAPACK: /Library/Frameworks/R.framework/Versions/3.4/Resources/lib/libRlapack.dylib
# 
# locale:
# [1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8
# 
# attached base packages:
# [1] stats     graphics  grDevices utils     datasets  methods   base     
# 
# other attached packages:
# [1] tm_0.7-3       NLP_0.1-11     dplyr_0.7.4    quanteda_1.1.6

Answer 2

在Windows平台上创建DFM时，看起来quanteda（和tm）正在丢失编码。在this tidytext问题中，使用取消令牌时会出现同样的问题。现在哪个工作正常，quanteda的{{1}}工作正常。如果我对tokens的{{1}}强制执行UTF-8或latin1编码，您将获得正确的结果。

@Dimnames$features

如果您执行以下操作：

dfm

R西班牙语术语频率矩阵与TD和Quanteda西班牙语字符

2 个答案: