Parallel.ForEach（...）系统内存不足异常
关于这个问题，我已经看了Parallel.ForEach - System Out of Memory Exception，但没有给出太多解决方案。我使用Parallel.ForEach非常新，所以我试图找出正在发生的事情。
诊断工具在1023重复出现（我明白这是x86到x64的限制，但我想以两种格式提供该程序。）我也不觉得任何程序应该达到这个阈值。当我在x64中编译程序时，我使用MaxDegree坐在1.1-1.4GB左右。为了测试，我在每个Parallel.ForEach迭代结束时运行GC.Collection（）（我知道这不是一个好的做法，我现在只是尝试排除故障。）
这是我所看到的：
现在，如果我尝试使用分区程序方法，例如：
var checkforfinished = Parallel.ForEach<ListViewItem>(Partitioner.Create(0,lstBackupUsers.Items.Count), lstBackupUsers.Items.Cast<ListViewItem>(), opts, name =>
我收到错误：
＆＃34;方法没有过载＆＃39; ForEach＆＃39;需要4个参数＆＃34;
没关系，我修改了我的Parallel.ForEach语句，看起来像这样：
var checkforfinished = Parallel.ForEach(Partitioner.Create(0,lstBackupUsers.Items.Count), lstBackupUsers.Items, opts, name =>（我删除了演员表）
然后我的ForEach方法不会接受该语句，因为它希望我明确告诉它它正在处理listviewbox.items方法。
我对如何做很困惑。
我是否创建了分区程序，如果我这样做，如何让Parallel.ForEach方法了解如何处理listviewbox？
更新1
我想尝试尽可能多地提供详细信息，因为这只是粗略的。我确定它很简单，我只是过度复制了n度。
我在后台工作程序函数中运行了Parallel.ForEach（//）。我的DoWork流程超过300行（我不是C＃的专家，我只是把工作中的程序放在一起。）
以下是其基本结构的要点
- 用户点击＆＃34;开始备份＆＃34;截图中显示的按钮
- Button启动一个单独的函数，该函数检查用户选择从哪个方法中获取用户名列表（LDAP或平面文本文件）
- 该函数然后发送bgw_dowork（）请求
-
在DoWork请求中，它看起来像是摘要：
- 检查初步陈述（例如bgw.cancellationpending）
- 继续从configurationmanager.appsettings 获取一些设置
- 开始＆＃34;复合体＆＃34; Parallel.ForEach命令读取列表框记录行和foreach行执行很长的命令列表以完成一个用户的操作
- 整个程序，特别是bgw_dowork大量使用Google的v3 Drive API以用户身份登录，抓取由准备备份用户目录的其他功能记录的文件（以用户身份登录的单独功能） ，记录他们的文件（和fileIds）及其目录/子目录），bgw_dowork执行实际下载功能的一大块，然后调用其他功能完成文件下载后的移动。
实际的＆＃34;代码＆＃34;我用的是（我保证它不漂亮......）
private void bgW_DoWork(object sender, DoWorkEventArgs e)
{
{
try
{
txtFile.ReadOnly = true;
btnStart.Text = "Cancel Backup";
var appSettings = ConfigurationManager.AppSettings;
string checkreplace = ConfigurationManager.AppSettings["checkreplace"];
string userfile = txtFile.Text;
int counter = 0;
int arraycount = 0;
if (bgW.CancellationPending)
{
e.Cancel = true;
stripLabel.Text = "Operation was canceled!";
}
else
{
for (int z = 0; z >= counter; z++)
{
if (bgW.CancellationPending)
{
e.Cancel = true;
stripLabel.Text = "Operation was canceled!";
break;
}
else
{
double totalresource = int.Parse(ConfigurationManager.AppSettings["multithread"]);
totalresource = (totalresource / 100);
//var opts = new ParallelOptions { MaxDegreeOfParallelism = Convert.ToInt32(Math.Ceiling((Environment.ProcessorCount * totalresource) * 1.0)) };
var opts = new ParallelOptions { MaxDegreeOfParallelism = 2 };
var part = Partitioner.Create(1, 100);
//foreach (ListViewItem name in lstBackupUsers.Items)
var checkforfinished = Parallel.ForEach(lstBackupUsers.Items.Cast<ListViewItem>(), name =>
{
try
{
string names = name.SubItems[0].Text;
lstBackupUsers.Items[arraycount].Selected = true;
lstBackupUsers.Items[arraycount].BackColor = Color.CornflowerBlue;
arraycount++;
stripLabel.Text = "";
Console.WriteLine("Selecting user: " + names.ToString());
txtLog.Text += "Selecting user: " + names.ToString() + Environment.NewLine;
txtCurrentUser.Text = names.ToString();
// Define parameters of request.
string user = names.ToString();
// Check if directory exists, create if not.
string savelocation = ConfigurationManager.AppSettings["savelocation"] + user + "\\";
if (File.Exists(savelocation + ".deltalog.tok"))
File.Delete(savelocation + ".deltalog.tok");
FileInfo testdir = new FileInfo(savelocation);
testdir.Directory.Create();
string savedStartPageToken = "";
var start = CreateService.BuildService(user).Changes.GetStartPageToken().Execute();
// This token is set by Google, it defines changes made and
// increments the token value automatically.
// The following reads the current token file (if it exists)
if (File.Exists(savelocation + ".currenttoken.tok"))
{
StreamReader curtokenfile = new StreamReader(savelocation + ".currenttoken.tok");
savedStartPageToken = curtokenfile.ReadLine().ToString();
curtokenfile.Dispose();
}
else
{
// Token record didn't exist. Create a generic file, start at "1st" token
// In reality, I have no idea what token to start at, but 1 seems to be safe.
Console.Write("Creating new token file.\n");
//txtLog.Text += ("Creating new token file.\n" + Environment.NewLine);
StreamWriter sw = new StreamWriter(savelocation + ".currenttoken.tok");
sw.Write(1);
sw.Dispose();
savedStartPageToken = "1";
}
string pageToken = savedStartPageToken;
int gtoken = int.Parse(start.StartPageTokenValue);
int mytoken = int.Parse(savedStartPageToken);
txtPrevToken.Text = pageToken.ToString();
txtCurrentToken.Text = gtoken.ToString();
if (gtoken <= 10)
{
Console.WriteLine("Nothing to save!\n");
//txtLog.Text += ("User has nothing to save!" + Environment.NewLine);
}
else
{
if (pageToken == start.StartPageTokenValue)
{
Console.WriteLine("No file changes found for " + user + "\n");
//txtLog.Text += ("No file changes found! Please wait while I tidy up." + Environment.NewLine);
}
else
{
// .deltalog.tok is where we will place our records for changed files
Console.WriteLine("Changes detected. Making notes while we go through these.");
lblProgresslbl.Text = "Scanning Drive directory.";
// Damnit Google, why did you change how the change fields work?
if (savedStartPageToken == "1")
{
statusStripLabel1.Text = "Recording folder list ...";
txtLog.Text = "Recording folder list ..." + Environment.NewLine;
exfunctions.RecordFolderList(savedStartPageToken, pageToken, user, savelocation);
statusStripLabel1.Text = "Recording new/changed files ... This may take a bit!";
txtLog.Text += Environment.NewLine + "Recording new/changed list for: " + user;
exfunctions.ChangesFileList(savedStartPageToken, pageToken, user, savelocation);
}
else
{
//proUserclass = proUser;
statusStripLabel1.Text = "Recording new/changed files ... This may take a bit!";
txtLog.Text += Environment.NewLine + "Recording new/changed list for: " + user + Environment.NewLine;
exfunctions.ChangesFileList(savedStartPageToken, pageToken, user, savelocation);
}
// Get all our files for the user. Max page size is 1k
// after that, we have to use Google's next page token
// to let us get more files.
StreamWriter logFile = new StreamWriter(savelocation + ".recent.log");
string[] deltafiles = File.ReadAllLines(savelocation + ".deltalog.tok");
int totalfiles = deltafiles.Count();
int cnttototal = 0;
Console.WriteLine("\nFiles to backup:\n");
if (deltafiles == null)
{
return;
}
else
{
double damn = ((gtoken - double.Parse(txtPrevToken.Text)));
damn = Math.Round((damn / totalfiles));
if (damn <= 0)
damn = 1;
foreach (var file in deltafiles)
{
try
{
if (bgW.CancellationPending)
{
stripLabel.Text = "Backup canceled!";
e.Cancel = true;
break;
}
DateTime dt = DateTime.Now;
string[] foldervalues = File.ReadAllLines(savelocation + "folderlog.txt");
cnttototal++;
bgW.ReportProgress(cnttototal);
proUser.Maximum = int.Parse(txtCurrentToken.Text);
stripLabel.Text = "File " + cnttototal + " of " + totalfiles;
double? mathisfun;
mathisfun = ((100 * cnttototal) / totalfiles);
if (mathisfun <= 0)
mathisfun = 1;
double mathToken = double.Parse(txtPrevToken.Text);
mathToken = Math.Round((damn + mathToken));
// Bring our token up to date for next run
txtPrevToken.Text = mathToken.ToString();
File.WriteAllText(savelocation + ".currenttoken.tok", mathToken.ToString());
int proval = int.Parse(txtPrevToken.Text);
int nowval = int.Parse(txtCurrentToken.Text);
if (proval >= nowval)
proval = nowval;
proUser.Value = (proval);
lblProgresslbl.Text = ("Current progress: " + mathisfun.ToString() + "% completed.");
// Our file is a CSV. Column 1 = file ID, Column 2 = File name
var values = file.Split(',');
string fileId = values[0];
string fileName = values[1];
string mimetype = values[2];
mimetype = mimetype.Replace(",", "_");
string folder = values[3];
string ext = null;
int folderfilelen = foldervalues.Count();
fileName = GetSafeFilename(fileName);
Console.WriteLine("Filename: " + values[1]);
logFile.WriteLine("ID: " + values[0] + " - Filename: " + values[1]);
logFile.Flush();
// Things get sloppy here. The reason we're checking MimeTypes
// is because we have to export the files from Google's format
// to a format that is readable by a desktop computer program
// So for example, the google-apps.spreadsheet will become an MS Excel file.
switch (mimetype)
{
(switch statement here removed due to body length issues for this post.)
}
if (ext.Contains(".doc") || ext.Contains(".xls"))
{
string whatami = null;
if (ext.Contains(".xls"))
{
whatami = "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet";
}
else if (ext.Contains(".doc"))
{
whatami = "application/vnd.openxmlformats-officedocument.wordprocessingml.document";
}
else if (ext.Contains(".ppt"))
{
whatami = "application/vnd.openxmlformats-officedocument.presentationml.presentation";
}
if (fileName.Contains(".mov") || ext == ".ggl" || fileName.Contains(".mp4"))
{
txtLog.Text += Environment.NewLine + "Skipping file.";
return;
}
var requestfileid = CreateService.BuildService(user).Files.Export(fileId, whatami);
statusStripLabel1.Text = (savelocation + fileName + ext);
txtCurrentUser.Text = user;
string dest1 = Path.Combine(savelocation, fileName + ext);
var stream1 = new System.IO.FileStream(dest1, FileMode.OpenOrCreate, FileAccess.ReadWrite);
scrolltobtm();
requestfileid.MediaDownloader.ProgressChanged +=
(IDownloadProgress progress) =>
{
switch (progress.Status)
{
case DownloadStatus.Downloading:
{
Console.WriteLine(progress.BytesDownloaded);
logFile.WriteLine("Downloading: " + progress.BytesDownloaded);
txtLog.Text += ("Downloading ... " + progress.BytesDownloaded + Environment.NewLine);
scrolltobtm();
logFile.Flush();
break;
}
case DownloadStatus.Completed:
{
Console.WriteLine("Download complete.");
logFile.WriteLine("[" + user + "] Download complete for: " + requestfileid.ToString());
txtLog.Text += ("[" + user + "] Download complete for: " + fileName + Environment.NewLine);
logFile.Flush();
break;
}
case DownloadStatus.Failed:
{
Console.WriteLine("Download failed.");
logFile.WriteLine("Download failed.");
logFile.Flush();
break;
}
}
};
scrolltobtm();
GC.Collect();
GC.WaitForPendingFinalizers();
requestfileid.Download(stream1);
stream1.Close();
stream1.Dispose();
}
else
{
scrolltobtm();
var requestfileid = CreateService.BuildService(user).Files.Get(fileId);
//Generate the name of the file, and create it as such on the local filesystem.
statusStripLabel1.Text = (savelocation + fileName + ext);
string dest1 = Path.Combine(savelocation, fileName + ext);
var stream1 = new System.IO.FileStream(dest1, FileMode.OpenOrCreate, FileAccess.ReadWrite);
requestfileid.MediaDownloader.ProgressChanged +=
(IDownloadProgress progress) =>
{
switch (progress.Status)
{
case DownloadStatus.Downloading:
{
Console.WriteLine(progress.BytesDownloaded);
logFile.WriteLine("Downloading: " + progress.BytesDownloaded);
txtLog.Text += ("Downloading ... " + progress.BytesDownloaded + Environment.NewLine);
scrolltobtm();
logFile.Flush();
break;
}
case DownloadStatus.Completed:
{
Console.WriteLine("Download complete.");
logFile.WriteLine("Download complete for: " + requestfileid.ToString());
txtLog.Text += (Environment.NewLine + "[" + user + "] Download complete for: " + fileName + Environment.NewLine);
logFile.Flush();
break;
}
case DownloadStatus.Failed:
{
Console.WriteLine("Download failed.");
logFile.WriteLine("Download failed.");
logFile.Flush();
break;
}
}
};
scrolltobtm();
GC.Collect();
GC.WaitForPendingFinalizers();
requestfileid.Download(stream1);
stream1.Close();
stream1.Dispose();
}
}
catch (Google.GoogleApiException ex)
{
Console.Write("\nInfo: ---> " + ex.Message.ToString() + "\n");
}
}
}
exfunctions.MoveFiles(savelocation);
Console.WriteLine("\n\n\tBackup completed for selected user!");
txtLog.Text += ("\n\nBackup completed for selected user.\n\n");
statusStripLabel1.Text = "";
//logFile.Close();
//logFile.Dispose();
}
}
}
catch (Google.GoogleApiException ex)
{
Console.WriteLine("Info: " + ex.Message.ToString());
}
}
);
if (checkforfinished.IsCompleted == true)
{
MessageBox.Show("Parallel.ForEach() Finished!");
Console.WriteLine("Parallel.ForEach() Finished!");
}
else
{
MessageBox.Show("Parallel.ForEach() not completed!");
Console.WriteLine("Parallel.ForEach() not completed!");
}
}
}
}
}
catch (Google.GoogleApiException ex)
{
Console.WriteLine("Info: " + ex.Message.ToString());
}
}
}
你可以看到我启动Parallel.ForEach（...）的位置，然后看看它负责做什么。它很多，我明白它并不漂亮，所以我很欣赏建设性的批评。
0 个答案:
没有答案
相关问题
最新问题
- 我写了这段代码，但我无法理解我的错误
- 我无法从一个代码实例的列表中删除 None 值，但我可以在另一个实例中。为什么它适用于一个细分市场而不适用于另一个细分市场？
- 是否有可能使 loadstring 不可能等于打印？卢阿
- java中的random.expovariate()
- Appscript 通过会议在 Google 日历中发送电子邮件和创建活动
- 为什么我的 Onclick 箭头功能在 React 中不起作用？
- 在此代码中是否有使用“this”的替代方法？
- 在 SQL Server 和 PostgreSQL 上查询，我如何从第一个表获得第二个表的可视化
- 每千个数字得到
- 更新了城市边界 KML 文件的来源？