Importing Excel file but the number of columns keep changing

Question

My Client sends me a file each month that I import into SQL through C#. The number of columns changes as per his requirement. The Columns are basically Accounts no. in which he receives money so if in any he does not receive money in any account he deletes the columns.

So i tried inputting the data base in a transpose manner like

Excel File Name PAN Acc1 Acc2 Acc3 Acc4

I import in SQL ROW1 - Name PAN Acc1 Amt ROW2 - Name PAN Acc2 Amt ROW3 - Name PAN Acc3 Amt

But this takes too much time when I execute a sql query for many month and all. Any Help please

Answer 1

I've found in the past when dealing with variable columns you can get yourself in a bit of a pickle for a number of reasons. So for example, you can't just append a column to a table each time a new account comes along, so you have to use a row entry as opposed to the column entry etc. You get other problems with joining data etc.

A good approach I've used in the past, with SQL Server anyway, is to use the data as an XML type in the SQL Server DB and create Stored Procs that use Cross Apply to deal with the data joining. You can also use XML schema validation for the data.

So, maybe that's an option for you.

Hope that helps.

Answer 2

If you are not processing any other logic than just "importing data into a SQL table", using C# may not be the most efficient solution, as it adds an unnecessary extra layer. Personally I prefer BCP since it is a built-in command and you can schedule a SQL Job to run in a monthly basis. Powershell is pretty good though but you would need to have it installed which is pretty easy and lightweight. Finally SSIS could also work but you need to have Microsoft SQL Server - Developer edition & Enterprise edition. I could help you with BCP or Powershell if you want.