Question

我有以下T_SQL存储过程，该过程目前占用了将新导入的记录上的所有进程运行到后端分析套件所需的总时间的50％。不幸的是，这些数据每次都需要导入，并且随着我们的数据库大小的增长而导致瓶颈。

基本上，我们正在尝试识别记录中的所有副本，并只保留其中一个。

DECLARE @status INT
SET @status = 3


DECLARE @contactid INT
DECLARE @email VARCHAR (100)


--Contacts
DECLARE email_cursor CURSOR FOR 
SELECT email FROM contacts WHERE  (reference  = @reference AND status = 1 ) GROUP BY email HAVING (COUNT(email) > 1)
OPEN email_cursor

FETCH NEXT FROM email_cursor INTO @email


WHILE @@FETCH_STATUS = 0
    BEGIN
        PRINT @email
        UPDATE contacts SET duplicate  = 1, status = @status  WHERE email = @email and reference = @reference  AND status = 1
        SELECT TOP 1 @contactid = id FROM contacts where  reference = @reference and email = @email AND duplicate = 1
        UPDATE contacts SET duplicate  =0, status = 1 WHERE id = @contactid
        FETCH NEXT FROM email_cursor INTO @email
    END


CLOSE email_cursor
DEALLOCATE email_cursor

我已经添加了我可以从查询执行计划中看到的所有索引，但是有可能更新整个SP以不同的方式运行，因为我已经设法与其他人一起工作。

Answer 1

使用此单一查询进行重复数据删除。

;with tmp as (
select *
      ,rn=row_number() over (partition by email, reference order by id)
      ,c=count(1) over (partition by email, reference)
  from contacts
 where status = 1
)
update tmp
   set duplicate = case when rn=1 then 0 else 1 end
      ,status = case when rn=1 then 1 else 3 end
 where c > 1
;

它只会在记录where status=1中进行重复数据删除，并将具有相同（电子邮件，参考）组合的行视为重复。

SQL服务器中导入记录的重复数据删除

1 个答案: