我正在使用活动记录来获取我的故事,然后生成CSV,这是在rails cast中完成的标准方式。但我有很多行,需要几分钟。我想如果我能让posgresql做csv渲染,那么我可以节省一些时间。
我现在所拥有的:
query = "COPY stories TO STDOUT WITH CSV HEADER;"
results = ActiveRecord::Base.connection.execute(query);
但此查询的结果为空:
=> #<PG::Result:0x00000006ea0488 @connection=#<PG::Connection:0x00000006c62fb8 @socket_io=nil, @notice_receiver=nil, @notice_processor=nil>>
2.0.0-p247 :053 > result.count
=> 0
更好的了解方式:
2.0.0-p247 :059 > result.to_json
=> "[]"
我怀疑我的控制器看起来像这样:
format.csv { send_data raw_results }
这适用于普通查询,我只是无法弄清楚将CSV结果返回到rails的SQL语法。
更新
将CSV从120000毫秒导出到290毫秒
我的模特:
def self.to_csv(story_ids)
csv = []
conn = ActiveRecord::Base.connection.raw_connection
conn.copy_data("COPY (SELECT * FROM stories WHERE stories.id IN (#{story_ids.join(',')})) TO STDOUT WITH (FORMAT CSV, HEADER TRUE, FORCE_QUOTE *, ESCAPE E'\\\\');") do
while row = conn.get_copy_data
csv.push(row)
end
end
csv.join("\r\n")
end
我的控制器:
send_data Story.to_csv(Story.order(:created_at).pluck(:id))
答案 0 :(得分:13)
AFAIK您需要在基础PostgreSQL数据库连接上使用copy_data
方法:
- (对象)copy_data(sql)
呼叫-SEQ:
conn.copy_data( sql ) {|sql_result| ... } -> PG::Result
执行复制过程以将[sic]数据传输到服务器或从服务器传输。
这将通过
COPY
发出SQL#exec
命令。对此的响应(如果命令中没有错误)是传递给块的PG::Result
对象,其状态代码为PGRES_COPY_OUT或PGRES_COPY_IN(取决于指定的复制方向)。然后,应用程序应使用#put_copy_data
或#get_copy_data
来接收或传输数据行,并在完成后从块中返回。
还有一个例子:
conn.copy_data "COPY my_table TO STDOUT CSV" do
while row=conn.get_copy_data
p row
end
end
ActiveRecord的原始数据库连接包装器不知道copy_data
是什么,但您可以使用raw_connection
来解包它:
conn = ActiveRecord::Base.connection.raw_connection
csv = [ ]
conn.copy_data('copy stories to stdout with csv header') do
while row = conn.get_copy_data
csv.push(row)
end
end
这将为您留下csv
中的一系列CSV字符串(每个数组条目一个CSV行),您可以csv.join("\r\n")
获取最终的CSV数据。
答案 1 :(得分:0)
This answer builds up on the answer provided by @mu-is-too-short, but without a temporary object using streaming instead.
headers['X-Accel-Buffering'] = 'no'
headers["Cache-Control"] = 'no-cache'
headers["Transfer-Encoding"] = 'chunked'
headers['Content-Type'] = 'text/csv; charset=utf-8'
headers['Content-Disposition'] = 'inline; filename="data.csv"'
headers.delete('Content-Length')
sql = "SELECT * FROM stories WHERE stories.id IN (#{story_ids.join(',')})"
self.response_body = Enumerator.new do |chunk|
conn = ActiveRecord::Base.connection.raw_connection
conn.copy_data("COPY (#{sql.chomp(';')}) TO STDOUT WITH (FORMAT CSV, HEADER TRUE, RCE_QUOTE *, ESCAPE E'\\\\');") do
while row = conn.get_copy_data
chunk << "#{row.length.to_s(16)}\r\n"
chunk << row
chunk << "\r\n"
end
chunk << "0\r\n\r\n"
end
end
You can also use gz = Zlib::GzipWriter.new(Stream.new(chunk))
and gz.write row
with a class akin to
class Stream
def initialize(block)
@block = block
end
def write(row)
@block << "#{row.length.to_s(16)}\r\n"
@block << row
@block << "\r\n"
end
end
And remember headers['Content-Encoding'] = 'gzip'
. See also this gist.