Question

我有一个MySQL表，我试图用R使用RMySQL访问。

应该从

返回1690004行

dbGetQuery(con, "SELECT * FROM tablename WHERE export_date ='2015-01-29'")

不幸的是，我收到以下警告消息：

In is(object, Cl) : error while fetching row
In dbGetQuery(con, "SELECT * FROM tablename WHERE export_date ='2015-01-29'",  : pending rows

只能接收~400K行。

如果我将查询分解为几个＆＃34;提取＆＃34;使用dbSendQuery，警告消息在收到~400K行后开始出现。

任何帮助都将不胜感激。

Answer 1

所以，看起来这是因为我的托管服务提供商施加了60秒的超时（该死的Arvixe！）。我通过“分页/分块”输出来解决这个问题。因为我的数据有一个自动递增的主键，所以返回的每一行都是有序的，这样我就可以在每次迭代后获取下一行X行。

要获得1.6M行，我执行了以下操作：

library(RMySQL)
con <- MySQLConnect() # mysql connection function
day <- '2015-01-29' # date of interest
numofids <- 50000 # number of rows to include in each 'chunk'
count <- dbGetQuery(con, paste0("SELECT COUNT(*) as count FROM tablename WHERE export_date = '",day,"'"))$count # get the number of rows returned from the table.
dbDisconnect(con)
ns <- seq(1, count, numofids) # get sequence of rows to work over
tosave <- data.frame() # data frame to bind results to
# iterate through table to get data in 50k row chunks
for(nextseries in ns){ # for each row
  print(nextseries) # print the row it's on
  con <- MySQLConnect()
  d1 <- dbGetQuery(con, paste0("SELECT * FROM tablename WHERE export_date = '",day,"' LIMIT ", nextseries,",",numofids)) # extract data in chunks of 50k rows
  dbDisconnect(con)
  # bind data to tosave dataframe. (the ifelse is avoid an error when it tries to rbind d1 to an empty dataframe on the first pass).
  if(nrow(tosave)>0){
      tosave <- rbind(tosave, d1)
  }else{
      tosave <- d1
  }
}

In（object，Cl）：获取行R时出错

1 个答案: