Question

我有一个名为 “转储”的文件夹。 此文件夹包含各种.CSV文件。文件夹位置为 'C：\ Dump'

我想将这些文件的内容导入SQL Server。我想要粗略的代码以及适当的注释，以便我理解它。

我尝试了一些我在网上找到的代码。但是由于一些奇怪的原因，他们并没有为我做好准备。

我想要的步骤是

步骤1：将文件夹中的所有文件名复制到表

第2步：遍历表并使用批量插入从文件中复制数据。

有人请帮我解决这个问题。非常感谢提前：）

Answer 1

    --BULK INSERT MULTIPLE FILES From a Folder 

    --a table to loop thru filenames drop table ALLFILENAMES
    CREATE TABLE ALLFILENAMES(WHICHPATH VARCHAR(255),WHICHFILE varchar(255))

    --some variables
    declare @filename varchar(255),
            @path     varchar(255),
            @sql      varchar(8000),
            @cmd      varchar(1000)


    --get the list of files to process:
    SET @path = 'C:\Dump\'
    SET @cmd = 'dir ' + @path + '*.csv /b'
    INSERT INTO  ALLFILENAMES(WHICHFILE)
    EXEC Master..xp_cmdShell @cmd
    UPDATE ALLFILENAMES SET WHICHPATH = @path where WHICHPATH is null


    --cursor loop
    declare c1 cursor for SELECT WHICHPATH,WHICHFILE FROM ALLFILENAMES where WHICHFILE like '%.csv%'
    open c1
    fetch next from c1 into @path,@filename
    While @@fetch_status <> -1
      begin
      --bulk insert won't take a variable name, so make a sql and execute it instead:
       set @sql = 'BULK INSERT Temp FROM ''' + @path + @filename + ''' '
           + '     WITH ( 
                   FIELDTERMINATOR = '','', 
                   ROWTERMINATOR = ''\n'', 
                   FIRSTROW = 2 
                ) '
    print @sql
    exec (@sql)

      fetch next from c1 into @path,@filename
      end
    close c1
    deallocate c1


    --Extras

    --delete from ALLFILENAMES where WHICHFILE is NULL
    --select * from ALLFILENAMES
    --drop table ALLFILENAMES

Answer 2

这将为您提供每个文件的单独表格。

--BULK INSERT MULTIPLE FILES From a Folder 
drop table allfilenames
--a table to loop thru filenames drop table ALLFILENAMES
CREATE TABLE ALLFILENAMES(WHICHPATH VARCHAR(255),WHICHFILE varchar(255))

--some variables
declare @filename varchar(255),
        @path     varchar(255),
        @sql      varchar(8000),
        @cmd      varchar(1000)


--get the list of files to process:
SET @path = 'D:\Benihana\backup_csv_benihana_20191128032207_part_1\'
SET @cmd = 'dir ' + @path + '*.csv /b'
INSERT INTO  ALLFILENAMES(WHICHFILE)
EXEC Master..xp_cmdShell @cmd
UPDATE ALLFILENAMES SET WHICHPATH = @path where WHICHPATH is null

delete from ALLFILENAMES where  WHICHFILE is null
--SELECT replace(whichfile,'.csv',''),* FROM dbo.ALLFILENAMES


--cursor loop
declare c1 cursor for SELECT WHICHPATH,WHICHFILE FROM ALLFILENAMES where WHICHFILE like '%.csv%' order by WHICHFILE desc
open c1
fetch next from c1 into @path,@filename
While @@fetch_status <> -1
  begin
  --bulk insert won't take a variable name, so make a sql and execute it instead:
   set @sql = 

   'select * into '+ Replace(@filename, '.csv','')+'
    from openrowset(''MSDASQL''
    ,''Driver={Microsoft Access Text Driver (*.txt, *.csv)}''
    ,''select * from '+@Path+@filename+''')' 


print @sql
exec (@sql)

  fetch next from c1 into @path,@filename
  end
close c1
deallocate c1

Answer 3

对于第1步，也许你可以看一下：

http://www.sql-server-performance.com/forum/threads/copying-filenames-to-sql-table.11546/

或

How to list files inside a folder with SQL Server

然后是第2步

How to cast variables in T-SQL for bulk insert?

HTH

Answer 4

要解决步骤1，也可以使用xp_dirtree列出所有文件和文件夹。

请记住，这是一个未记录的功能。必须考虑安全预防措施。精心设计的文件名可能是入侵向量。

Answer 5

在python中，您可以使用d6tstack来简化

import d6tstack
import glob

c = d6tstack.combine_csv.CombinerCSV(glob.glob('*.csv'))
c.to_mssql_combine('mssql+pymssql://usr:pwd@localhost/db', 'tablename')

请参见SQL examples。它还处理data schema changes，创建表并允许您预处理数据。它利用BULK INSERT，因此应该一样快。

Answer 6

要扩展SarangArd的答案，如果文件名与表名匹配，则可以用以下内容替换temp。

' + Left(@filename, Len(@filename)-4) + '

Answer 7

您可能需要先启用xp_cmdshell：

sp_configure 'show advanced options', '1'
RECONFIGURE
go

sp_configure 'xp_cmdshell', '1' 
RECONFIGURE
go

并且，要启用ad_hoc，

sp_configure 'show advanced options', 1;
RECONFIGURE;
GO

sp_configure 'Ad Hoc Distributed Queries', 1;
RECONFIGURE;
GO

Answer 8

此代码将为导入的每个CSV文件创建一个新表。最好从CSV文件填充空数据库。

CREATE TABLE ALLFILENAMES
(
             WHICHPATH VARCHAR(255)
            ,WHICHFILE VARCHAR(255)
)

DECLARE @filename VARCHAR(255),
        @path     VARCHAR(255),
        @sql      VARCHAR(8000),
        @cmd      VARCHAR(1000)

SET @path = 'L:\DATA\SOURCE\CSV\'  --PATH TO YOUR CSV FILES (CHANGE TO YOUR PATH)
SET @cmd = 'dir ' + @path + '*.csv /b'

INSERT INTO ALLFILENAMES(WHICHFILE)
EXEC Master..xp_cmdShell @cmd

UPDATE ALLFILENAMES
  SET WHICHPATH = @path
WHERE WHICHPATH IS NULL

DECLARE c1 CURSOR
FOR SELECT WHICHPATH
          ,WHICHFILE
    FROM ALLFILENAMES
    WHERE WHICHFILE LIKE '%.csv%'

OPEN c1

FETCH NEXT FROM c1 INTO @path,
                        @filename

WHILE @@fetch_status <> -1
BEGIN
    CREATE TABLE #Header
    (
                 HeadString NVARCHAR(MAX)
    )
    DECLARE @Columns NVARCHAR(MAX) = ''
    DECLARE @Query NVARCHAR(MAX) = ''
    DECLARE @QUERY2 NVARCHAR(MAX) = ''
    DECLARE @HeaderQuery NVARCHAR(MAX) = ''

    SELECT @HeaderQuery = @HeaderQuery + 'bulk insert #Header from ''' + @path + @filename + '''  
        with(firstrow=1,lastrow=1)'
    EXEC (@HeaderQuery)

    SELECT @Columns = (SELECT QUOTENAME(value) + ' nvarchar(max)' + ','
                       FROM #Header
                            CROSS APPLY STRING_SPLIT(HeadString,',') FOR xml PATH(''))
    IF ISNULL(@Columns,'') <> ''
    BEGIN
        SET @Columns = LEFT(@Columns,LEN(@Columns) - 1)

        SELECT @Query = @Query + 'CREATE TABLE ' + Replace(@filename,'.csv','') + ' (' + replace(@Columns,'"','') + ')'
        PRINT @Query
        EXEC (@QUERY)
    END

    SELECT @QUERY2 = @QUERY2 + 'bulk insert ' + replace(Replace(@filename,'.csv',''),'.TPS','') + ' from ''' + @path + @filename + '''  
        with(firstrow=2,FORMAT=''csv'',FIELDTERMINATOR='','',ROWTERMINATOR=''\n'')'
    EXEC (@QUERY2)

    DROP TABLE #Header
    FETCH NEXT FROM c1 INTO @path,
                            @filename
END

CLOSE c1

DEALLOCATE c1

从文件夹导入多个CSV文件到SQL Server

8 个答案: