Question

我正在尝试读取多个csv文件的目录，每个文件大约有7K +行和~1800列。我有一个数据字典，我可以读入数据框，其中数据字典的每一行都标识变量（列）名称，以及数据类型。

在查看?read_csv包中找到的readr时，可以指定列类型。但是，鉴于我有近1800列要指定，我希望使用可用数据字典中的信息以函数所需的正确格式指定列/类型对。

另一种不太理想的选择是将每一列作为一个字符读入，然后根据需要手工修改。

非常感谢您提供有关如何指定列类型的任何帮助。

如果有帮助的话，这是我的代码，可以将数据字典转换成我所指的格式。

## Get the data dictionary
URL = "https://collegescorecard.ed.gov/assets/CollegeScorecardDataDictionary.xlsx"
download.file(URL, destfile="raw-data/dictionary.xlsx")

## read in the dictionary to get the variables
dict = read_excel("raw-data/dictionary.xlsx", sheet = "data_dictionary")
colnames(dict) = tolower(gsub(" ", "_", colnames(dict)))
dict = dict %>% filter(!is.na(variable_name))

## create a data dictionary
## https://stackoverflow.com/questions/46738968/specify-column-types-in-read-csv-by-using-values-in-a-dataframe/46742411#46742411
dict <- dict %>% mutate(variable_type = case_when(api_data_type == "integer" ~ "i", 
                                                  api_data_type == "autocomplete" ~ "c", #assumption that this is a string
                                                  api_data_type == "string" ~ "c",
                                                  api_data_type == "float" ~ "d"))

返回：

> ## read in the dictionary to get the variables
> dict = read_excel("raw-data/dictionary.xlsx", sheet = "data_dictionary")
> colnames(dict) = tolower(gsub(" ", "_", colnames(dict)))
> dict = dict %>% filter(!is.na(variable_name))
> dict <- dict %>% mutate(variable_type = case_when(api_data_type == "integer" ~ "i", 
+                                                   api_data_type == "autocomplete" ~ "c", #assumption that this is a string
+                                                   api_data_type == "string" ~ "c",
+                                                   api_data_type == "float" ~ "d"))
Error: object 'api_data_type' not found

和我的sessionInfo

> sessionInfo()
R version 3.3.1 (2016-06-21)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.11.6 (El Capitan)

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] stringr_1.2.0 readxl_0.1.1  readr_1.1.0   dplyr_0.5.0  

loaded via a namespace (and not attached):
 [1] rjson_0.2.15   lazyeval_0.2.0 magrittr_1.5   R6_2.2.2       assertthat_0.1 hms_0.2        DBI_0.7        tools_3.3.1   
 [9] tibble_1.2     yaml_2.1.14    Rcpp_0.12.11   stringi_1.1.5  jsonlite_1.5

Answer 1

您可以使用mutate和case_when的组合来使用紧凑字符串表示来映射api_data_type列。这是每个列类型由单个字母表示的位置：c =字符，i =整数，n =数字，d = double，l =逻辑等。现在，此字符向量可用于col_types参数read_csv。

## Load libraries
library(dplyr)
library(readxl)

## Get the data dictionary
URL = "https://collegescorecard.ed.gov/assets/CollegeScorecardDataDictionary.xlsx"
download.file(URL, destfile="raw-data/dictionary.xlsx")

## read in the dictionary to get the variables
dict = read_excel("raw-data/dictionary.xlsx", sheet = "data_dictionary")
colnames(dict) = tolower(gsub(" ", "_", colnames(dict)))
dict = dict %>% filter(!is.na(variable_name))

unique(dict$api_data_type)
#> [1] "integer"      "autocomplete" "string"       "float"

dict <- dict %>% mutate(variable_type = case_when(api_data_type == "integer" ~ "i", 
                                                  api_data_type == "autocomplete" ~ "c", #assumption that this is a string
                                                  api_data_type == "string" ~ "c",
                                                  api_data_type == "float" ~ "d"
                                                  ) 
                        )  
cnames <- dict %>% select(variable_name) %>% pull
head(cnames)
#> [1] "UNITID" "OPEID"  "OPEID6" "INSTNM" "CITY"   "STABBR"
ctypes <- dict %>% select(variable_type) %>% pull
head(ctypes)
#> [1] "i" "i" "i" "c" "c" "c"

使用数据框

1 个答案: