Ruby处理大型XML会泄漏内存

时间:2015-12-01 14:32:04

标签: ruby xml memory sinatra nokogiri

我有一个类通过公开的Web服务接收大型(4Mb)XML文件,然后继续将文件保存到系统,另一个方法将文件加载到内存并处理它并保存它和结果进入数据库的过程。

据我所知,将完整文件加载到内存应该使用大量内存,但问题是在处理文件(关闭和删除)后,应用程序内存未释放。

最初我们把这个任务作为sidekiq工作,并且sidekiq使用的内存没有被释放,担心sidekiq中的内存泄漏我们在sidekiq之外运行进程并且整个应用程序的内存遭受同样的泄漏&# 39;

我们使用Nokogiri加载文件并通过xpath。

要尝试分析泄漏,我们正在运行此脚本:

def ostats(last_stat = nil)
stats = Hash.new(0)
ObjectSpace.each_object {|o| stats[o.class] += 1}

  stats.sort {|(k1,v1),(k2,v2)| v2 <=> v1}.each do |k,v|
    printf "%-30s  %10d", k, v
    printf " | delta %10d", (v - last_stat[k]) if last_stat
    puts
  end

  stats
end

stats = nil
c = Class.new
stats = ostats(stats)
puts 'Press to continue'
option = gets
stats = ostats(stats)
puts 'Press to continue'
option = gets
GC.start
puts 'Executing GC...'
stats = ostats(stats)

脚本获取内存中加载的每个类的对象数,方法之前,方法之后以及执行GarbageCollector之后再次显示,输出显示没有太大区别:

String                               10093
Array                                  880
RubyVM::InstructionSequence            674
Class                                  278
Encoding                               100
Regexp                                  67
Hash                                    59
Module                                  22
MatchData                               20
Gem::Requirement                        19
File                                    18
RubyVM::Env                             11
Gem::Version                             9
Proc                                     9
Binding                                  7
Gem::Specification                       6
Time                                     5
Float                                    4
Mutex                                    3
IO                                       3
Object                                   3
Bignum                                   2
Thread::Backtrace                        2
LoadError                                2
Rational                                 2
Complex                                  1
ThreadGroup                              1
IOError                                  1
Thread                                   1
RubyVM                                   1
NoMemoryError                            1
SystemStackError                         1
Random                                   1
ARGF.class                               1
Data                                     1
fatal                                    1
Range                                    1
#<Class:0x007efce0bbed68>                1
Gem::Platform                            1
Monitor                                  1

------------------
String                               10339 | delta        246
Array                                  923 | delta         43
RubyVM::InstructionSequence            674 | delta          0
Class                                  278 | delta          0
Encoding                               100 | delta          0
Regexp                                  67 | delta          0
Hash                                    60 | delta          1
Module                                  22 | delta          0
MatchData                               20 | delta          0
Gem::Requirement                        19 | delta          0
File                                    18 | delta          0
RubyVM::Env                             11 | delta          0
Gem::Version                             9 | delta          0
Proc                                     9 | delta          0
Binding                                  7 | delta          0
Gem::Specification                       6 | delta          0
Time                                     5 | delta          0
Mutex                                    4 | delta          1
Float                                    4 | delta          0
IO                                       3 | delta          0
Object                                   3 | delta          0
Bignum                                   2 | delta          0
Thread::Backtrace                        2 | delta          0
LoadError                                2 | delta          0
Rational                                 2 | delta          0
Complex                                  1 | delta          0
ThreadGroup                              1 | delta          0
IOError                                  1 | delta          0
Thread                                   1 | delta          0
RubyVM                                   1 | delta          0
NoMemoryError                            1 | delta          0
SystemStackError                         1 | delta          0
Random                                   1 | delta          0
ARGF.class                               1 | delta          0
Data                                     1 | delta          0
fatal                                    1 | delta          0
Range                                    1 | delta          0
#<Class:0x007efce0bbed68>                1 | delta          0
Gem::Platform                            1 | delta          0
Monitor                                  1 | delta          0

Executing GC...

String                                5018 | delta      -5321
RubyVM::InstructionSequence            596 | delta        -78
Class                                  278 | delta          0
Array                                  251 | delta       -672
Encoding                               100 | delta          0
Regexp                                  67 | delta          0
Hash                                    29 | delta        -31
Module                                  22 | delta          0
Gem::Requirement                        14 | delta         -5
Gem::Version                             9 | delta          0
Proc                                     9 | delta          0
Gem::Specification                       6 | delta          0
RubyVM::Env                              5 | delta         -6
Mutex                                    4 | delta          0
Float                                    4 | delta          0
Time                                     4 | delta         -1
IO                                       3 | delta          0
Object                                   3 | delta          0
Bignum                                   2 | delta          0
Complex                                  1 | delta          0
ThreadGroup                              1 | delta          0
IOError                                  1 | delta          0
Binding                                  1 | delta         -6
Thread                                   1 | delta          0
RubyVM                                   1 | delta          0
NoMemoryError                            1 | delta          0
SystemStackError                         1 | delta          0
Random                                   1 | delta          0
ARGF.class                               1 | delta          0
Data                                     1 | delta          0
fatal                                    1 | delta          0
Range                                    1 | delta          0
#<Class:0x007efce0bbed68>                1 | delta          0
Gem::Platform                            1 | delta          0
Monitor                                  1 | delta          0

0 个答案:

没有答案