我有一个类通过公开的Web服务接收大型(4Mb)XML文件,然后继续将文件保存到系统,另一个方法将文件加载到内存并处理它并保存它和结果进入数据库的过程。
据我所知,将完整文件加载到内存应该使用大量内存,但问题是在处理文件(关闭和删除)后,应用程序内存未释放。
最初我们把这个任务作为sidekiq工作,并且sidekiq使用的内存没有被释放,担心sidekiq中的内存泄漏我们在sidekiq之外运行进程并且整个应用程序的内存遭受同样的泄漏&# 39;
我们使用Nokogiri加载文件并通过xpath。
要尝试分析泄漏,我们正在运行此脚本:
def ostats(last_stat = nil)
stats = Hash.new(0)
ObjectSpace.each_object {|o| stats[o.class] += 1}
stats.sort {|(k1,v1),(k2,v2)| v2 <=> v1}.each do |k,v|
printf "%-30s %10d", k, v
printf " | delta %10d", (v - last_stat[k]) if last_stat
puts
end
stats
end
stats = nil
c = Class.new
stats = ostats(stats)
puts 'Press to continue'
option = gets
stats = ostats(stats)
puts 'Press to continue'
option = gets
GC.start
puts 'Executing GC...'
stats = ostats(stats)
脚本获取内存中加载的每个类的对象数,方法之前,方法之后以及执行GarbageCollector之后再次显示,输出显示没有太大区别:
String 10093
Array 880
RubyVM::InstructionSequence 674
Class 278
Encoding 100
Regexp 67
Hash 59
Module 22
MatchData 20
Gem::Requirement 19
File 18
RubyVM::Env 11
Gem::Version 9
Proc 9
Binding 7
Gem::Specification 6
Time 5
Float 4
Mutex 3
IO 3
Object 3
Bignum 2
Thread::Backtrace 2
LoadError 2
Rational 2
Complex 1
ThreadGroup 1
IOError 1
Thread 1
RubyVM 1
NoMemoryError 1
SystemStackError 1
Random 1
ARGF.class 1
Data 1
fatal 1
Range 1
#<Class:0x007efce0bbed68> 1
Gem::Platform 1
Monitor 1
------------------
String 10339 | delta 246
Array 923 | delta 43
RubyVM::InstructionSequence 674 | delta 0
Class 278 | delta 0
Encoding 100 | delta 0
Regexp 67 | delta 0
Hash 60 | delta 1
Module 22 | delta 0
MatchData 20 | delta 0
Gem::Requirement 19 | delta 0
File 18 | delta 0
RubyVM::Env 11 | delta 0
Gem::Version 9 | delta 0
Proc 9 | delta 0
Binding 7 | delta 0
Gem::Specification 6 | delta 0
Time 5 | delta 0
Mutex 4 | delta 1
Float 4 | delta 0
IO 3 | delta 0
Object 3 | delta 0
Bignum 2 | delta 0
Thread::Backtrace 2 | delta 0
LoadError 2 | delta 0
Rational 2 | delta 0
Complex 1 | delta 0
ThreadGroup 1 | delta 0
IOError 1 | delta 0
Thread 1 | delta 0
RubyVM 1 | delta 0
NoMemoryError 1 | delta 0
SystemStackError 1 | delta 0
Random 1 | delta 0
ARGF.class 1 | delta 0
Data 1 | delta 0
fatal 1 | delta 0
Range 1 | delta 0
#<Class:0x007efce0bbed68> 1 | delta 0
Gem::Platform 1 | delta 0
Monitor 1 | delta 0
Executing GC...
String 5018 | delta -5321
RubyVM::InstructionSequence 596 | delta -78
Class 278 | delta 0
Array 251 | delta -672
Encoding 100 | delta 0
Regexp 67 | delta 0
Hash 29 | delta -31
Module 22 | delta 0
Gem::Requirement 14 | delta -5
Gem::Version 9 | delta 0
Proc 9 | delta 0
Gem::Specification 6 | delta 0
RubyVM::Env 5 | delta -6
Mutex 4 | delta 0
Float 4 | delta 0
Time 4 | delta -1
IO 3 | delta 0
Object 3 | delta 0
Bignum 2 | delta 0
Complex 1 | delta 0
ThreadGroup 1 | delta 0
IOError 1 | delta 0
Binding 1 | delta -6
Thread 1 | delta 0
RubyVM 1 | delta 0
NoMemoryError 1 | delta 0
SystemStackError 1 | delta 0
Random 1 | delta 0
ARGF.class 1 | delta 0
Data 1 | delta 0
fatal 1 | delta 0
Range 1 | delta 0
#<Class:0x007efce0bbed68> 1 | delta 0
Gem::Platform 1 | delta 0
Monitor 1 | delta 0