Question

我在使用parfeval而不是parfor时遇到了问题。我这样做是为了更好地控制作业调度和监控进度。

基本上，在使用parfeval时，如何获得与parfor类似的效率？

parfor实现，运行时间为1分钟。

parfor i = 1 : params.nperm
    sampledData(i) =  parfun( maprf.datam.ModelData( data.spikes(perm(i,:)), data.movie) );
end

Parfeval实施;消耗13gb内存，崩溃了Matlab。

for i = 1 : params.nperm
  F(i) = parfeval(parfun, 1,  maprf.datam.ModelData( data.spikes(perm(i,:)), data.movie));
end
% Build a waitbar to track progress
h = waitbar(0,'Waiting for FevalFutures to complete...');
results = zeros(1,params.nperm);
for idx = 1:params.nperm
    [completedIdx,thisResult] = fetchNext(F);
    % store the result
    results(completedIdx) = thisResult;
    % update waitbar
    waitbar(idx/params.nperm,h,sprintf('Latest result: %d',thisResult));
end
delete(h)

使用自定义调度程序进行Parfeval;花了一个多小时。

MyPool = gcp();
mapJobs = zeros(MyPool.NumWorkers, 1);
for i = 1 : MyPool.NumWorkers
    F(i) = parfeval(parfun, 1, maprf.datam.ModelData(data.spikes(perm(i,:)), data.movie));
    mapJobs(i) = i;
end

% Build a waitbar to track progress
progress = MyPool.NumWorkers;
h = waitbar(progress, 'Waiting for FevalFutures to complete...');
while ~all([F.Read])
    % find next available results
    [completedIdx, thisResult] = fetchNext(F);
    % find which is the actual index of the job
    %                                 idx = map(completedIdx);
    % store the results
    results(completedIdx) = thisResult;
    % first of all we increment the counter of processed jobs
    i = i + 1;
    progress = progress + 1;
    % now we have an available worker at position completedIdx
    if i < params.nperm
        % there are still pending jobs,
        % then we assign the job to the worker
        F(completedIdx) = parfeval(parfun, 1, maprf.datam.ModelData(data.spikes(perm(i,:)), data.movie));
        % and we keep track of which job is being processed by the worker
        mapJobs(completedIdx) = i;
    end
    % finally we update the waitbar for the happiness of the user
    waitbar(progress / params.nperm, h);%, sprintf('Latest result: %d',thisResult));
end
% at this point i should be equal to progress and to params.nperm
assert((i == progress) && (progress == params.nperm)); % only for debug purposes (it raises an error if the condition is not satisfied)
% at this point we delete the pool of workers (or whatever they are) and close the waitbar
cancel(F);
delete(h)

parfeval问题 - RAM太慢或太多

0 个答案: