将多个文件从文件夹加载到sql表-性能太慢

时间:2020-04-24 17:43:26

标签: ssis

我对以下代码(SSIS中的脚本任务)有问题,该代码将来自多个csv的数据加载到表sql中。我从一个站点获得了代码,并且可以正常工作,但是它的速度太慢了……

我需要某人来帮助我优化此代码,以使其运行更快,因为目前它的导入速度为每小时100万条记录...

string datetime = DateTime.Now.ToString("yyyyMMddHHmmss");
        try
        {

            //Declare Variables
            string SourceFolderPath = Dts.Variables["User::SourceFolder"].Value.ToString();
            string FileExtension = Dts.Variables["User::FileExtension"].Value.ToString();
            string FileDelimiter = Dts.Variables["User::FileDelimiter"].Value.ToString();
            string TableName = Dts.Variables["User::DestinationTable"].Value.ToString();
            string ArchiveFolder = Dts.Variables["User::ArchiveFolder"].Value.ToString();
            //string ColumnList = "";

            //Reading file names one by one
            string SourceDirectory = SourceFolderPath;
            string[] fileEntries = Directory.GetFiles(SourceDirectory, "*" + FileExtension);
            foreach (string fileName in fileEntries)
            {

                SqlConnection myADONETConnection = new SqlConnection();
                myADONETConnection = (SqlConnection)
                (Dts.Connections["hiddenoninternet"].AcquireConnection(Dts.Transaction) as SqlConnection);

                //Writing Data of File Into Table
                int counter = 0;
                string line;
                //MessageBox.Show(fileName);

                System.IO.StreamReader SourceFile =
                new System.IO.StreamReader(fileName);
                while ((line = SourceFile.ReadLine()) != null)
                {
                    if (counter > 0)
                    {

                        string query = "Insert into " + TableName + " Values ('";
                        query += line.Replace(FileDelimiter, "','") + "','" + fileName + "')";
                        //MessageBox.Show(query.ToString());
                        SqlCommand myCommand1 = new SqlCommand(query, myADONETConnection);
                        myCommand1.ExecuteNonQuery();
                    }

                    counter++;
                }

                SourceFile.Close();
                //move the file to archive folder after adding datetime to it
                File.Move(fileName, ArchiveFolder + "\\" + (fileName.Replace(SourceFolderPath, "")).Replace(FileExtension, "") + "_" + datetime + FileExtension);
                Dts.TaskResult = (int)ScriptResults.Success;
            }
        }
        catch (Exception exception)
        {
            // Create Log File for Errors
            using (StreamWriter sw = File.CreateText(Dts.Variables["User::LogFolder"].Value.ToString()
                + "\\" + "ErrorLog_" + datetime + ".log"))
            {
                sw.WriteLine(exception.ToString());
                Dts.TaskResult = (int)ScriptResults.Failure;
            }

        }
    }

0 个答案:

没有答案