读取多个文件并根据用户输入计算均值

时间:2014-05-13 20:04:46

标签: r function subset mean missing-data

我正在尝试在R中编写一个带3个输入的函数:

  1. 目录
  2. 污染物
  3. ID
  4. 我的计算机上有一个目录,里面装满了CSV文件,超过300个。这个功能的功能如下所示:

    pollutantmean <- function(directory, pollutant, id = 1:332) {
            ## 'directory' is a character vector of length 1 indicating
            ## the location of the CSV files
    
            ## 'pollutant' is a character vector of length 1 indicating
            ## the name of the pollutant for which we will calculate the
            ## mean; either "sulfate" or "nitrate".
    
            ## 'id' is an integer vector indicating the monitor ID numbers
            ## to be used
    
            ## Return the mean of the pollutant across all monitors list
            ## in the 'id' vector (ignoring NA values)
            }
    

    此函数的示例输出如下所示:

    source("pollutantmean.R")
    pollutantmean("specdata", "sulfate", 1:10)
    
    ## [1] 4.064
    
    pollutantmean("specdata", "nitrate", 70:72)
    
    ## [1] 1.706
    
    pollutantmean("specdata", "nitrate", 23)
    
    ## [1] 1.281
    

    我可以一次性阅读整篇文章:

    path = "C:/Users/Sean/Documents/R Projects/Data/specdata"
    fileList = list.files(path=path,pattern="\\.csv$",full.names=T)
    all.files.data = lapply(fileList,read.csv,header=TRUE)
    DATA = do.call("rbind",all.files.data)
    

    我的问题是:

    1. 用户输入原子或范围内的id,例如假设用户输入1,但文件名为001.csv,或者如果用户输入范围1:10,则文件名为001.csv ... 010.csv
    2. 用户会输入色谱柱,即&#34;硫酸盐&#34;或者&#34;硝酸盐&#34;他/她有兴趣获得...的平均值这些列中有很多缺失值(在计算平均值之前我需要从列中省略。
    3. 来自所有文件的整个数据如下所示:

      summary(DATA)
               Date           sulfate          nitrate             ID       
       2004-01-01:   250   Min.   : 0.0     Min.   : 0.0     Min.   :  1.0  
       2004-01-02:   250   1st Qu.: 1.3     1st Qu.: 0.4     1st Qu.: 79.0  
       2004-01-03:   250   Median : 2.4     Median : 0.8     Median :168.0  
       2004-01-04:   250   Mean   : 3.2     Mean   : 1.7     Mean   :164.5  
       2004-01-05:   250   3rd Qu.: 4.0     3rd Qu.: 2.0     3rd Qu.:247.0  
       2004-01-06:   250   Max.   :35.9     Max.   :53.9     Max.   :332.0  
       (Other)   :770587   NA's   :653304   NA's   :657738
      

      任何想法如何制定这一点都将受到高度赞赏...

      干杯

8 个答案:

答案 0 :(得分:4)

所以,你可以像这样模拟你的情况;

# Simulate some data:
# Create 332 data frames
set.seed(1)
df.list<-replicate(332,data.frame(sulfate=rnorm(100),nitrate=rnorm(100)),simplify=FALSE)
# Generate names like 001.csv and 010.csv
file.names<-paste0('specdata/',sprintf('%03d',1:332),'.csv')
# Write them to disk
invisible(mapply(write.csv,df.list,file.names))

这是一个可以读取这些文件的函数:

pollutantmean <- function(directory, pollutant, id = 1:332) {
  file.names <- list.files(directory)
  file.numbers <- as.numeric(sub('\\.csv$','', file.names))
  selected.files <- na.omit(file.names[match(id, file.numbers)])
  selected.dfs <- lapply(file.path(directory,selected.files), read.csv)
  mean(c(sapply(selected.dfs, function(x) x[ ,pollutant])), na.rm=TRUE)
}

pollutantmean('specdata','nitrate',c(1:100,141))
# [1] -0.005450574

答案 1 :(得分:2)

User enters id either atomic or in a range e.g. 
     

假设用户输入1,但文件名为001.csv,或者如果用户输入范围,该怎么办   1:10然后文件名是001.csv ... 010.csv

您可以使用正则表达式和gsub函数从文件名中删除前导零,然后创建一个字典(在r中,一个命名向量),将修改后的/ gsub'd文件名转换为实际的文件名。 例如:如果您的文件名是字符向量,fnames

fnames = c("001.csv","002.csv")
names(fnames) <- gsub(pattern="^[0]*", replacement="", x=fnames)

通过这个,矢量fnames被转换为字典,让你用001.csv的行调用名为fnames["1.csv"]的文件。您还可以使用gsub()删除文件名的.csv部分。

  

用户使用的是“硫酸盐”或“硝酸盐”   他/她有兴趣获得...的意思。有很多   这些列中缺少值(我需要从列中省略)   在计算平均值之前。

许多R函数都有一个忽略指示缺失值的特殊字符的选项。尝试在R命令提示符处输入help(mean)以查找有关此功能的信息。

答案 2 :(得分:2)

这是一个甚至你的祖母都能理解的解决方案:

pollutantmean <- function(directory, pollutant, id = 1:332) {

  # Break this function up into a series of smaller functions
  # that do exactly what you expect them to. Your friends
  # will love you for it.

  csvFiles = getFilesById(id, directory)

  dataFrames = readMultipleCsvFiles(csvFiles)

  dataFrame = bindMultipleDataFrames(dataFrames)

  getColumnMean(dataFrame, column = pollutant)
}


getFilesById <- function(id, directory = getwd()) {
  allFiles = list.files(directory)
  file.path(directory, allFiles[id])
}

readMultipleCsvFiles <- function(csvFiles) {
  lapply(csvFiles, read.csv)
}

bindMultipleDataFrames <- function(dataFrames) {
  Reduce(function(x, y) rbind(x, y), dataFrames)
}

getColumnMean <- function(dataFrame, column, ignoreNA = TRUE) {
  mean(dataFrame[ , column], na.rm = ignoreNA)
}

答案 3 :(得分:1)

这就是我修复它的方式:

pollutantmean <- function(directory, pollutant, id = 1:332) {
    #set the path
    path = directory

    #get the file List in that directory
    fileList = list.files(path)

    #extract the file names and store as numeric for comparison
    file.names = as.numeric(sub("\\.csv$","",fileList))

    #select files to be imported based on the user input or default
    selected.files = fileList[match(id,file.names)]

    #import data
    Data = lapply(file.path(path,selected.files),read.csv)

    #convert into data frame
    Data = do.call(rbind.data.frame,Data)

    #calculate mean
    mean(Data[,pollutant],na.rm=TRUE)

    }

最后一个问题是我的功能应该调用&#34; specdata&#34; (所有csv所在的目录名称)作为目录,r中是否有目录类型对象?

假设我将该函数称为:

pollutantmean(specdata, "niterate", 1:10)

它应该在我的工作目录中找到specdata目录的路径......我该怎么做?

答案 4 :(得分:1)

这是一个用于计算文件列表中特定列的平均值的一般功能。不确定应该如何设置id,但现在它充当索引向量(即id = 1:3计算文件列表中前三个文件的平均值。)

multifile.means <- function(directory = getwd(), pollutant, id = NULL)
{
    d <- match.arg(directory, list.files())
    cn <- match.arg(pollutant,  c('sulfate', 'nitrate'))
    ## get a vector of complete file paths in the given 'directory'
    p <- dir(d, full.names = TRUE)
    ## subset 'p' based on 'id' values
    if(!is.null(id)){
        id <- id[!id > length(p)]
        p <- p[id]
    }
    ## read, store, and name the relevant columns
    cl <- sapply(p, function(x){ read.csv(x)[,cn] }, USE.NAMES = FALSE)
    colnames(cl) <- basename(p)
    ## return a named list of some results
    list(values = cl, 
         mean = mean(cl, na.rm = TRUE), 
         colMeans = colMeans(cl, na.rm = TRUE))
}

将其用于试驾:

> multifile.means('testDir', 'sulfate')
# $values
#      001.csv 057.csv 146.csv 213.csv
# [1,]       5      10      NA       9
# [2,]       1       1      10       3
# [3,]      10       4      10       2
# [4,]       3      10       9      NA
# [5,]       4       1       5       5

# $mean
# [1] 5.666667

# $colMeans
# 001.csv 057.csv 146.csv 213.csv 
#    4.60    5.20    8.50    4.75 

答案 5 :(得分:1)

选择的答案看起来不错,但这是另一种选择。这个答案适用于JHU课程所涵盖的基础知识。

pollutantmean <- function(directory, pollutant, id = 1:332) {
    csvfiles <- dir(directory, "*\\.csv$", full.names = TRUE)
    data <- lapply(csvfiles[id], read.csv)
    numDataPoints <- 0L
    total <- 0L
    for (filedata in data) {
        d <- filedata[[pollutant]] # relevant column data
        d <- d[complete.cases(d)] # remove NA values
        numDataPoints <- numDataPoints + length(d)
        total <- total + sum(d)
    }
    total / numDataPoints
}

答案 6 :(得分:1)

我花了几个小时来解决这个问题,但这是我的(较短的)版本

pollutmean<- function(dir, pollutant, id=1:332) {
  dir<- list.files(dir, full.names = T)     #list files
  dat<- data.frame()                        #make empty df
  for (i in id) {
    dat <- rbind(dat, read.csv(dir[i]))     #rbind all files
  }
  mean(dat[,pollutant], na.rm = TRUE)       #calculate mean of given column
}

pollutmean("assign/specdata", "sulfate", id=1:60)

答案 7 :(得分:-1)

我也在阅读课程,并提出了以下解决方案:

pollutantmean <- function(directory="d:/dev/r/documents/specdata",       pollutant, 
                      id)   {
myfilename = paste(directory,"/",formatC(id, width=3, flag="0"),".csv",
                   sep="")
master = lapply(myfilename, read.table, header=TRUE, sep=",")
masterfile = do.call("rbind", master)
head(masterfile[[2]], 100)

if (pollutant == "sulfate") {
    #result=lapply(masterfile[[2]], mean, na.rm=TRUE)
    result=mean(masterfile[[2]], na.rm=TRUE)  

}
if (pollutant == "nitrate") {
    result=mean(masterfile[[3]], na.rm=TRUE)

}
result
}