模块加载多个进程

时间:2014-10-02 20:17:39

标签: parallel-processing multiprocessing julia

当使用工作进程时,我对Julia在加载模块时的行为感到困惑。

我需要使用一个相当繁重的PyPlot模块,这需要花费大量的时间来加载。这个计划:

using PyPlot
pygui(true)
println("Loaded")

在我的笔记本电脑上加载大约需要11秒钟:

% time julia test.jl 
INFO: Loading help data...
Loaded
julia test.jl  11,10s user 0,18s system 99% cpu 11,323 total

请注意INFO: Loading help data...行。它似乎是由PyPlot模块发出的,因为如果省略using PyPlot行,它就不会出现。

然而,当我运行这个程序时:

using PyPlot
pygui(true)
@everywhere println("Loaded")

我得到了这些结果:

% time julia -p 4 test.jl 
INFO: Loading help data...
INFO: Loading help data...
INFO: Loading help data...
INFO: Loading help data...
INFO: Loading help data...
Loaded
    From worker 2:  Loaded
    From worker 5:  Loaded
    From worker 3:  Loaded
    From worker 4:  Loaded
julia -p 4 test.jl  88,94s user 1,19s system 266% cpu 33,865 total

它不仅运行了33秒(三倍!),但它似乎也在每个工作人员上加载PyPlot模块

但我确信为了让每个工作人员都能使用模块,它必须是@everywhere d!的确,这个简单的程序崩溃了:

module Example
    export x
    x = 10
end

using Example

@everywhere println("x: $x")

调用:

% julia -p 4 test2.jl
x: 10
exception on 2: exception on exception on exception on 4: 5: 3: ERROR: x not defined
 in eval at /usr/bin/../lib/julia/sys.so
ERROR: x not defined
 in eval at /usr/bin/../lib/julia/sys.so
ERROR: x not defined
 in eval at /usr/bin/../lib/julia/sys.so
ERROR: x not defined
 in eval at /usr/bin/../lib/julia/sys.so

那么,即使我没有请求,为什么PyPlot模块会加载到所有工作人员身上呢?

更有趣的是,有一种解决方法:

using PyPlot
pygui(true)
addprocs(4)
@everywhere println("Loaded")

当我使用julia test.jl运行此程序时,我得到15秒:

% time julia test.jl     
INFO: Loading help data...
Loaded
    From worker 2:  Loaded
    From worker 4:  Loaded
    From worker 5:  Loaded
    From worker 3:  Loaded
julia test.jl  21,98s user 0,46s system 143% cpu 15,678 total

这正是我对原始版本所期望的julia -p 4 test.jl。但我不喜欢这种解决方法,因为它强制我的程序使用addprocs()

当使用-p X参数启动Julia时,有没有办法将模块加载限制为主进程?

1 个答案:

答案 0 :(得分:2)

让我们使用DummyModule.jl进行测试:

module DummyModule

export MyType, f

type MyType
    a::Int
end

f(x) = x^2+1

println("loaded")

end

有了这个,我们可以看到实际上有3种可能性。所有这些实验都是从作为julia -p 2调用的新的julia会话中运行的。

在所有进程

上使用DummyModule
julia> @everywhere using DummyModule
loaded
        From worker 3:  loaded
        From worker 2:  loaded

julia> @everywhere println(f(4))
17
        From worker 2:  17
        From worker 3:  17

julia> rr = RemoteRef(2)
RemoteRef(2,1,24)

julia> put!(rr, MyType(7))
RemoteRef(2,1,24)

julia> fetch(rr)
MyType(7)

仅在驱动程序进程

上使用DummyModule
julia> include("DummyModule.jl")
loaded

julia> using DummyModule

julia> @everywhere println(f(4))
exception on 2: 17
exception on 3: ERROR: f not defined
 in eval at /home/tim/src/julia/base/sysimg.jl:7
 in anonymous at multi.jl:1383
 in anonymous at multi.jl:819
 in run_work_thunk at multi.jl:592
 in run_work_thunk at multi.jl:601
 in anonymous at task.jl:6
ERROR: f not defined
 in eval at /home/tim/src/julia/base/sysimg.jl:7
 in anonymous at multi.jl:1383
 in anonymous at multi.jl:819
 in run_work_thunk at multi.jl:592
 in run_work_thunk at multi.jl:601
 in anonymous at task.jl:6

julia> rr = RemoteRef(2)
RemoteRef(2,1,14)

julia> put!(rr, MyType(7))
WARNING: Module DummyModule not defined on process 2
fatal error on 2: ERROR: DummyModule not defined
 in deserialize at serialize.jl:376
 in handle_deserialize at serialize.jl:351
 in deserialize at serialize.jl:505
 in handle_deserialize at serialize.jl:351
 in deserialize at serialize.jl:334
 in anonymous at serialize.jl:354
 in ntuple at tuple.jl:30
 in deserialize_tuple at serialize.jl:354
 in handle_deserialize at serialize.jl:346
 in anonymous at task.jl:824
Worker 2 terminated.ERROR: ProcessExitedException()
 in wait at ./task.jl:284
 in wait at ./task.jl:194
 in wait_full at ./multi.jl:574
 in remotecall_fetch at multi.jl:675
 in remotecall_fetch at multi.jl:680
 in call_on_owner at multi.jl:722
 in put! at multi.jl:743

(请注意,打印不同邮件的顺序不确定;在exception on 2: 17中,exception ...部分是由于流程2发出错误而17来了在流程1上打印f(4)。)

仅在进程1上使用,但在工作程序之间传递数据

julia> using DummyModule
loaded
        From worker 3:  loaded
        From worker 2:  loaded

julia> @everywhere println(f(4))
exception on 2: 17
exception on 3: ERROR: f not defined
 in eval at /home/tim/src/julia/base/sysimg.jl:7
 in anonymous at multi.jl:1383
 in anonymous at multi.jl:819
 in run_work_thunk at multi.jl:592
 in run_work_thunk at multi.jl:601
 in anonymous at task.jl:6
ERROR: f not defined
 in eval at /home/tim/src/julia/base/sysimg.jl:7
 in anonymous at multi.jl:1383
 in anonymous at multi.jl:819
 in run_work_thunk at multi.jl:592
 in run_work_thunk at multi.jl:601
 in anonymous at task.jl:6

julia> rr = RemoteRef(2)
RemoteRef(2,1,19)

julia> put!(rr, MyType(7))
RemoteRef(2,1,19)

julia> using DummyModule

julia> fetch(rr)
MyType(7)

最后一种情况的区别在于,序列化程序实际上知道如何在所有进程上处理MyType,从而可以安全地在工作程序之间传递数据。