mysql - Bulk CSV File Import in MySQL Removing duplicates while Importing dynamic colmns from CSV

I have to import CSV File for different clients in my system, some with [,] some [|] Etc… separated. Always very big files. While importing I need to filter duplicate records, duplicate records should not insert in dB.

Problem is columns can be different for different clients. I have database for every client for CSV DATA which I import every day, Week or month depending on client need. I keep data for every import so we can generate reports & what data we receive in CSV file our system do processing after import.

Data structure example:

Client 1 database:

First_Name | Last_Name | Email | Phone | Ect…

95% data always same every new CSV file. Some records new comes & some records they delete from csv. so our system only process those records those newly imported .

Currently what happening we import data every time in new table. We keep table name with time stamp so we can keep track for import. it Is expensive process, it duplicate records and tables. Im thinking solution and I need your suggestion on it. Keeping one table every time import when I import CSV file data in table, I’ll alter existing table add new column, column name will be “current date” (byte or Boolean) add true false on import??

My other question is first time when I import CSV file …I need to write script: While importing CSV data, if table already exists then my date logic will work else if table does not exist it should create table given or provided “client name” as table name. Challenge is columns I don’t know it should create columns from CSV file.

Table already exist some new records came in it should insert else update.

Is it do able in mysql?? although I have to do something for mssql also but right now I need solution for my sql.

Please help me... im not good in MySQL :(

Bulk CSV File Import in MySQL Removing duplicates while Importing dynamic colmns from CSV

1 个答案: