Question

我有一个庞大的数据集要在SQL Server中返回（大约3200万行）这是在视图中实现的，源代码如下：

SELECT Idenitifier = ISNULL(mle.MIdeer, mle.Ider) + em.MemberId,
       EffectiveDate = ISNULL(em.EffectiveDate, 
                                (SELECT TOP 1 EffectiveDate
                                 FROM   c
                                 WHERE  SourceType = em.SourceType
                                    AND GroupNumber = em.GroupNumber
                                    AND ISNULL(GroupDivision, '') = 
                                        ISNULL(em.GroupDivision, '')))
FROM   a em
       JOIN b mle
         ON mle.Identifier = em.GroupNumber + ISNULL('-' + em.GroupDivision, '')

            -- Filter invalid legal entities
            AND ISNULL(mle.Filter, 0) = 0

       --- Gets a resultset of 531798 rows 

       CROSS JOIN -- this returns 63 rows , so  
       -- I am presuming 531798*63 rows here.
       (SELECT *
        FROM   map
        WHERE  domaintype = 'MC')b;

我需要使用SSIS将此数据集加载到表中。在1600万行之后，当我给出select * from <<view>>时，我在sql server中得到了一个system.out内存异常。如何使用SSIS在表中加载此数据集，避免此异常..

还有哪些更好的方法可以有效地进行此查询，因为它需要运行超过30分钟？

Answer 1

我仍在考虑这个问题，但您可能需要将CROSS JOIN：

分开

;WITH cte AS (SELECT  ISNULL(mle.MIdeer, mle.Ider) + em.MemberId AS Idenitifier
                      , ISNULL(em.EffectiveDate,
                               ( SELECT TOP 1
                                        EffectiveDate
                                 FROM   c
                                 WHERE  SourceType = em.SourceType
                                        AND GroupNumber = em.GroupNumber
                                        AND ISNULL(GroupDivision, '') = ISNULL(em.GroupDivision,
                                                                              '')
                               )) AS EffectiveDate
                FROM    a em
                JOIN    b mle ON mle.Identifier = em.GroupNumber + ISNULL('-'+ em.GroupDivision,'')
                            AND ISNULL(mle.Filter, 0) = 0)
SELECT *
FROM cte
CROSS JOIN ( SELECT *
             FROM   map
             WHERE  domaintype = 'MC') b;

需要使用SSIS将巨大的数据集（3200万）加载到表中

1 个答案: