Question

I have a table with a very large number of rows which I wish to execute via dynamic SQL. They are basically existence checks and insert statements and I want to migrate data from one production database to another - we are merging transactional data. I am trying to find the optimal way to execute the rows.

I've been finding the coalesce method for appending all the rows to one another to not be efficient for this particularly when the number of rows executed at a time is greater than ~100.

Assume the structure of the source table is something arbitrary like this:

CREATE TABLE [dbo].[MyTable]
(
[ID] [int] IDENTITY(1,1) NOT NULL,
[DataField1] [int] NOT NULL,
[FK_ID1] [int] NOT NULL,
[LotsMoreFields] [NVARCHAR] (MAX),
CONSTRAINT [PK_MyTable] PRIMARY KEY CLUSTERED ([ID] ASC)
)

CREATE TABLE [dbo].[FK1]
(
[ID] [int] IDENTITY(1,1) NOT NULL,
[Name] [int] NOT NULL, -- Unique constrained value
CONSTRAINT [PK_FK1] PRIMARY KEY CLUSTERED ([ID] ASC)
)

The other requirement is I am tracking the source table PK vs the target PK and whether an insert occurred or whether I have already migrated that row to the target. To do this, I'm tracking migrated rows in another table like so:

CREATE TABLE [dbo].[ChangeTracking]
(
[ReferenceID] BIGINT IDENTITY(1,1),
[Src_ID] BIGINT,
[Dest_ID] BIGINT,
[TableName] NVARCHAR(255),
CONSTRAINT [PK_ChangeTracking] PRIMARY KEY CLUSTERED ([ReferenceID] ASC)
)

My existing method is executing some dynamic sql generated by a stored procedure. The stored proc does PK lookups as the source system has different PK values for table [dbo].[FK1]. E.g.

IF NOT EXISTS (<ignore this existence check for now>)
BEGIN
    INSERT INTO [Dest].[dbo].[MyTable] ([DataField1],[FK_ID1],[LotsMoreFields]) VALUES (333,(SELECT [ID] FROM [Dest].[dbo].[FK1] WHERE [Name]=N'ValueFoundInSource'),N'LotsMoreValues');
    INSERT INTO [Dest].[dbo].[ChangeTracking] ([Src_ID],[Dest_ID],[TableName]) VALUES (666,SCOPE_IDENTITY(),N'MyTable'); --666 is the PK in [Src].[dbo].[MyTable] for this inserted row
END

So when you have a million of these, it isn't quick.

Is there a recommended performant way of doing this?

Answer 1

如前所述，当您查看复杂的JOIN条件时，MERGE语句很有效（如果这些字段中的任何一个不同，则更新记录以匹配）。您还可以考虑创建整个记录的HASHBYTES哈希，以快速找到源表和目标表之间的差异，尽管这对于非常大的数据集来说也很耗时。

Answer 2

听起来您正在像前端开发人员那样进行这些更新，方法是检查每一行是否匹配然后进行插入。使用单个查询执行插入会更有效。下面是一个示例，它查找tblNewClient表中的名称，但不在tblClient表中查找：

INSERT INTO tblClient
        ( [Name] ,
          TypeID ,
          ParentID 
          )
SELECT nc.[Name] ,
    nc.TypeID ,
    nc.ParentID
FROM tblNewClient nc
    LEFT JOIN tblClient cl
        ON nc.[Name] = cl.[Name]
WHERE cl.ID IS NULL;

这将方式比做RBAR（通过痛苦行划线）更有效。

Answer 3

从@RusselFox得到两个答案并将它们放在一起，我达到了这个试验性的解决方案（但看起来效率更高）：

MERGE INTO [Dest].[dbo].[MyTable] [MT_D]
USING (SELECT [MT_S].[ID] as [SrcID],[MT_S].[DataField1],[FK_1_D].[ID] as [FK_ID1],[MT_S].[LotsMoreFields]
FROM [Src].[dbo].[MyTable] [MT_S]
JOIN [Src].[dbo].[FK_1] ON [MT_S].[FK_ID1] = [FK_1].[ID]
JOIN [Dest].[dbo].[FK_1] [FK_1_D] ON [FK_1].[Name] = [FK_1_D].[Name]
) [SRC] ON 1 = 0 
WHEN NOT MATCHED THEN
INSERT([DataField1],[FL_ID1],[LotsMoreFields])
VALUES ([DataField1],[FL_ID1],[LotsMoreFields])
OUTPUT [SRC].[SrcID],INSERTED.[ID],0,N'MyTable' INTO [Dest].[dbo].[ChangeTracking]([Src_ID],[Dest_ID],[AlreadyExists],[TableName]);

Dynamic SQL to execute large number of rows from a table

3 个答案: