如何在SSIS中将大型excel文件拆分为多个小型文件？

Question

我想读取一个Excel文件，该文件大250 mb，具有30万多行。因此，当我尝试在SSIS中的excel源中对其进行处理时，它花费了太多时间，最终无法处理。谁能告诉我如何分割文件或其他任何有帮助的方法？

Answer 1

SQLCMD更好，因为超时时间更长。但是您不必将Excel保存为CSV，而且我不知道您是否可以这样做。

如果您有CSV文件（即BulkDataFile.csv），并创建了以下文件：

BULK INSERT dbo.TableForBulkData
FROM 'C:\BulkDataFile.csv'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = '\n'
)

然后，您可以使用SQLCMD -E -I import.sql（将文件另存为import.sql）

Answer 2

Best way is to read data by chunks, i have done this to read huge tables from SQL databases. But to read big excel files, i implemented this in a C# application not in SSIS since it is more complicated. I will provide some links for more info:

• Reading Huge volume of data from Sqlite to SQL Server fails at pre-execute (SQLite)
• SSIS failing to save packages and reboots Visual Studio (Oracle)
• Optimizing SSIS package for millions of rows with Order by / sort in SQL command and Merge Join (MySQL)
• Getting top n to n rows from db2 (DB2)
• OutOfMemoryException while trying to read big Excel file into DataTable (Excel C#)

If you need to do this only using SSIS, i can provide a detailed answer later similar to what i provided in the first link above (SQLite).

SchemaMapper: C# schema mapping class library

Recently i started a new project on Git-Hub, which is a class library developed using C#. You can use it to import tabular data from excel, word , powerpoint, text, csv, html, json and xml into SQL server table with a different schema definition using schema mapping approach. It gives the ability to import big excel files. check it out at:

• SchemaMapper: C# Schema mapping class library

You can follow this Wiki page for a step-by-step guide:

• Import data from multiple files into one SQL table step by step guide