Question

我在SQL Server中有4个表：AspNetUsers，CustomerFile，CustomerOption和LastPullRecords。应用程序从Excel文件上载客户记录。此Excel文件将转换为DataTable，然后对于DataTable的每一行调用此存储过程。

在CustomerFile表格上应用了一个触发器。在此存储过程中，我们首先检查FirstName，LastName，StreetAddress，City，State和Zip是否未更改，然后仅更新官员详细说明其他明智更新所有细节，设置更新动作（'U'），并将记录发送到第三方第二天。其次，如果该客户不在，则添加它并将操作设置为add（'A'）。如果可用，那么我们将根据客户记录更新另外两个表。

ALTER PROC [dbo].[InsertUpdateRecords]
(
    @FullName NVARCHAR(50) =NULL,
    @FirstName NVARCHAR(50) =NULL,
    @LastName NVARCHAR(50) =NULL,
    @StreetAddress NVARCHAR(50) =NULL,
    @City NVARCHAR(50) =NULL,
    @State NVARCHAR(50) =NULL,
    @Zip INT =NULL,
    @SSN NVARCHAR(50) =NULL,
    @Email NVARCHAR(150) =NULL,
    @OfficerEmail NVARCHAR(50) =NULL,
    @OfficerId NVARCHAR(50)=NULL,
    @OfficerName NVARCHAR(50) =NULL,
    @Option NVARCHAR(50) =NULL,
    @DownloadedFromFTP BIT =NULL,
    @LastPullDate DATETIME=NULL
)
AS
BEGIN
DECLARE @IsActive BIT
DECLARE @FileID INT
DECLARE @CompanyId INT

SET @IsActive=1

--Get Company ID based on OfficerID
Select @CompanyId=CompanyId from AspNetUsers where Email=@OfficerEmail

select top (1) @FileID=cf.fileId from CustomerFile cf inner join AspNetUsers usr on usr.Id=cf.OfficerId  where cf.SSN = @SSN and usr.CompanyId=@CompanyId order by cf.FileReceivedDate, cf.FileId desc

if ((@FileID<>'') or(@FileID is not null))
    Begin
        -- COMPARE IF ONLY OFFICER IS CHANGED
        If EXISTS(select 1 from CustomerFile where FirstName=@FirstName and LastName=@LastName and StreetAddress=@StreetAddress and City=@City and State=@State and Zip=@Zip
         and FileId=@FileID         
            )
            BEGIN
                UPDATE top (1) CustomerFile SET OfficerEmail=@OfficerEmail,
                OfficerName=@OfficerName,Email=@Email,
                ----FileModifiedDate=GETDATE(),
                DownloadedFromFTP=@DownloadedFromFTP,IsActive=@IsActive,OfficerId=@OfficerId
                WHERE FileId=@FileID                
            END
        Else
            BEGIN
                Update top (1) CustomerFile set FullName=@FullName, FirstName=@FirstName, LastName=@LastName, StreetAddress=@StreetAddress, City=@City,State=@State,Zip=@Zip,
                OfficerEmail=@OfficerEmail,OfficerName=@OfficerName,Email=@Email,
                --FileReceivedDate=GETDATE(),
                FileModifiedDate=GETDATE(),DownloadedFromFTP=@DownloadedFromFTP,IsActive=@IsActive,Action='U'
                where FileId=@FileID
            END
    End
Else
    BEGIN
        declare @IdentityOutput table ( ID int )

        INSERT INTO CustomerFile(FullName,FirstName,LastName,StreetAddress,City,State,Zip,SSN,OfficerEmail,OfficerId,OfficerName,
        FileReceivedDate,DownloadedFromFTP,IsActive,Action,Email
        ) 
        output inserted.FileId into @IdentityOutput
        VALUES(@FullName,@FirstName,@LastName,@StreetAddress,@City,
        @State,@Zip,@SSN,@OfficerEmail,@OfficerId,@OfficerName,
        GETDATE(),@DownloadedFromFTP,@IsActive,'A',@Email)

        select @FileID = (select ID from @IdentityOutput)
    END
---------------------------------------------------------------------------
-- Set Option
---------------------------------------------------------------------------


if ((@Option<>'') or(@Option is not null))
Begin
    if exists(select 1 from CustomerOption where CustomerFileID=@FileID)
        Begin
            Update CustomerOption Set Option=@Option where CustomerFileID=@FileID
        End
    else
        Begin
            Insert into CustomerOption (CustomerFileID, Option) values (@FileID, @Option)
        End
End

---------------------------------------------------------------------------
-- Insert Last Pull if exist
---------------------------------------------------------------------------

if ((@LastPullDate<>'') or(@LastPullDate is not null) or CONVERT(varchar(10),@LastPullDate,101)!='01/01/1900')

Begin

if((@FileID<>'') OR (@FileID<>0))
Begin
if exists (Select * from LastPullRecords where CustomerId=@FileID and CompanyId=@CompanyId)
bEGIN
Update LastPullRecords
set  LastPullDate=@LastPullDate,
     IsSelfPull=1,
     ModifiedDateTime=getdate()
where CustomerId=@FileID and CompanyId=@CompanyId
End

ELSE

Begin
iNSERT INTO LastPullRecords
(CompanyId,CustomerId,LastPullDate,IsSelfPull,IsRTS,CreatedDateTime)
values
(@CompanyId,@FileID,@LastPullDate,1,0,getdate())

End
End
end

END

问题是可能有数千条记录，这个查询需要很长时间才能上传所有这些记录。为了测试，我只上传了10K记录，花了13分钟。

我尝试将DataTable作为参数发送，为数据表定义自定义用户表类型，使用while循环，然后是光标，但所有这些实验没有任何区别。

请建议任何优化的上传这些记录的方式，以便花费更少的时间。

Answer 1

如果您想继续使用.NET执行此任务，我认为您可以从两个选项中进行选择：

使用SqlBulkCopy将数据复制到中间表（可能是您可以在没有中间表的情况下编写插入），然后编写一个命令以插入所有新数据，使用第二个命令更新所有已更改的数据。
使用Table-Valued Parameters代替SqlBulkCopy和中间表（如果您的MSSQL不低于2012）

请注意，对于小型数据集，SqlBulkCopy 可能比表值参数慢，但对于大数据集则更快。

如何优化此查询以插入/更新SQL Server中的数百万条记录

1 个答案: