Question

我正在使用try catch块来捕获带有约束错误的数据。例如。如果null插入非空列或插入重复记录或发生类型不匹配，则所有带错误的源记录都应转到错误日志表，其余记录应转到目标表。为此我使用try catch所以我不能使用批量插入，因此使用While循环逐行插入，这需要永远运行，因为我必须插入3000000记录。有什么方法可以在循环中提高性能吗？所以它可以在最短的时间内插入3000000条记录？目前需要2个小时或更长时间:(

Answer 1

尝试批量进行插入。例如，尝试循环尝试一次插入10,000 / 1,000/100个记录作为批量插入。如果批处理中存在错误，请捕获它并以逐行操作的形式重新执行该批处理。您必须使用批量大小并使其足够小，以便大多数批次作为批量插入处理，并且只需要偶尔逐行批处理。

Answer 2

以下演示了使用＆＃34;二分搜索＆＃34;批量处理一堆样本数据。发生错误时的批量大小。

set nocount on;

-- Set the processing parameters.
declare @InitialBatchSize as Int = 1024;
declare @BatchSize as Int = @InitialBatchSize;

-- Create some sample data with somewhat random   Divisor   values.
declare @RowsToProcess as Int = 10000;
declare @SampleData as Table ( Number Int, Divisor Int );
with Digits as ( select Digit from ( values (0), (1), (2), (3), (4), (5), (6), (7), (8), (9) ) as Digits( Digit ) ),
  Numbers as (
  select ( ( ( Ten_4.Digit * 10 + Ten_3.Digit ) * 10 + Ten_2.Digit ) * 10 + Ten_1.Digit ) * 10 + Ten_0.Digit + 1 as Number
    from Digits as Ten_0 cross join Digits as Ten_1 cross join Digits as Ten_2 cross join
      Digits as Ten_3 cross join Digits as Ten_4 )
  insert into @SampleData
    select Number, Abs( Checksum( NewId() ) ) % 1000 as Divisor -- Adjust "1000" to vary the chances of a zero divisor.
      from Numbers
      where Number < @RowsToProcess;

-- Process the data.  
declare @FailedRows as Table ( Number Int, Divisor Int, ErrorMessage NVarChar(2048) );
declare @BitBucket as Table ( Number Int, Divisor Int, Quotient Int );
declare @RowCount as Int = 1; -- Force at least one loop execution.
declare @LastProcessedNumber as Int = 0;
while @RowCount > 0
  begin
  begin try
    -- Subject-to-failure   INSERT .
    insert into @BitBucket
      select top ( @BatchSize ) Number, Divisor, 1 / Divisor as Quotient
        from @SampleData
        where Number > @LastProcessedNumber
        order by Number;
    set @RowCount = @@RowCount;
    select @LastProcessedNumber = Max( Number ) from @BitBucket;
    print 'Processed ' + Cast( @RowCount as VarChar(10) ) + ' rows.';
  end try
  begin catch
    if @BatchSize > 1
      begin
      -- Try a smaller batch.
      set @BatchSize /= 2;
      end
    else
      begin
      -- This is a failing row.  Log it with the error and reset the batch size.
      set @LastProcessedNumber += 1;
      print 'Row failed. Row number ' + Cast( @LastProcessedNumber as VarChar(10) ) + ', error: ' + Error_Message() + '.';
      insert into @FailedRows
        select Number, Divisor, Error_Message()
          from @SampleData
          where Number = @LastProcessedNumber;
      set @BatchSize = @InitialBatchSize;
      end
  end catch
  end;

-- Dump the results.
select * from @FailedRows order by Number;
select * from @SampleData order by Number;
select * from @BitBucket order by Number;

在insert语句中使用while循环的性能问题

2 个答案: