Question

对于我们在Google Big Query中面临的缓存问题需要一些建议。

我正在使用Talend tBigQueryInput组件来运行Google Big Query SQL。但是，SQL不返回当前数据。我们怀疑它正在从缓存中返回陈旧的数据。

是否有可以在SQL中嵌入的禁用缓存选项，以便我们说严格禁止从缓存中获取大查询？

Answer 1

有两个快速的想法：

您可以在UI中检查作业历史记录，以查看Talend正在运行的作业。他们将从统计信息中报告是否从缓存中提供了结果。您也可以检查审核日志以获取此信息。
您可以利用非确定性方面执行更改后的测试查询，这将确保无法从缓存中提供结果。为此，CURRENT_TIMESTAMP()，CURRENT_DATE()等功能就足够了。

通过扫描公开的Talend文档，没有配置设置，我可以看到该设置映射为查询作业将configuration.query.useQueryCache设置为false，这是您所要求的行为。

Answer 2

是否有可以在SQL中嵌入的禁用缓存选项

不！查询中没有此类选项

同时，要强制不使用缓存，您可以在查询中添加以下内容

    http
            .authorizeRequests()
            .anyRequest().authenticated()
            .and()
            .oauth2Login()
            .userInfoEndpoint()
            .oidcUserService(oidcUserService);

这只是愚蠢的例子-但您应该有个主意：o）

Answer 3

另一种选择是创建UDF（User Defined Functions）。

没有执行UDF，就不会通过BigQuery WebUI进行查询缓存。

例如

    create  TABLE dbo.tblStaging
    (
      SourceSystemID int not null,

      Attribute1 varchar(128) not null 
        constraint DF_tblStaging_Attribute1 default 'N/A',

      Attribute2 varchar(128) not null 
        constraint DF_tblStaging_Attribute2 default 'N/A',


    )

    -- Create the type two slowly changing dimension table


    create table dbo.tblDimSCDType2Example

    (
      SurrogateKey int not null identity(1,1) PRIMARY KEY,

      SourceSystemID int not null,

      Attribute1 varchar(128) not null 
        constraint DF_tblDimSCDType2Example_Attribute1 default 'N/A',

      Attribute2 varchar(128) not null 
        constraint DF_tblDimSCDType2Example_Attribute2 default 'N/A',

     UpdateType varchar(10) default 'I'

    ,UpdateDate date default getdate()-10

    ,QualityInd char(10) default 'G',

     Lnd_Hash varchar(40) not null,



----------


      EffectiveDate date not null 
        constraint DF_tblDimSCDType2Example_EffectiveDate default getdate(),

      EndDate date not null 
        constraint DF_tblDimSCDType2Example_EndDate default '12/31/9999',

      CurrentRecord char(1) not null 
        constraint DF_tblDimSCDType2Example_CurrentRecord default 'Y',

      LastUpdated datetime  not null 
        constraint DF_tblDimSCDType2Example_LastUpdated default getdate()

    )

    ---- merge scd type 2 changes

    Insert into dbo.tblDimSCDType2Example 
    ( --Table and columns in which to insert the data
      SourceSystemID,
      Attribute1,
      Attribute2,
      UpdateType
    ,UpdateDate
    ,QualityInd
      ,Lnd_Hash,
      EffectiveDate,
      EndDate
    )
    SELECT SourceSystemID,
      Attribute1,
      Attribute2,
      UpdateType
    ,UpdateDate
    ,QualityInd
      ,Lnd_Hash,
      EffectiveDate,
      EndDate 
     FROM
     (
     MERGE INTO  dbo.tblDimSCDType2Example as target
     USING
     (SELECT SourcesystemID,Attribute1,Attribute2,
     CONVERT(varchar(50),HASHBYTES('MD5', CONCAT(SourcesystemID,Attribute1,Attribute2)),2) AS LND_HASH
     FROM dbo.tblStaging) AS SOURCE
     (SourceSystemID,
      Attribute1,
      Attribute2,
      Lnd_Hash)
      ON (
        target.SourceSystemID = source.SourceSystemID
      )

      WHEN MATCHED and target.lnd_hash<>source.lnd_hash and target.currentrecord='Y'
      THEN 
      UPDATE SET 
        EndDate=getdate()-1, 
        CurrentRecord='N'

        WHEN NOT MATCHED THEN  
      INSERT 
      (
        SourceSystemID, 
        Attribute1,
        Attribute2,
        UpdateType
        ,UpdateDate
        ,QualityInd
      , Lnd_hash
      )
      VALUES 
      (
        source.SourceSystemID, 
        source.Attribute1,
        source.Attribute2,
        'I',Getdate(),'G',
        source.Lnd_hash
      )
      OUTPUT $action, 
        source.SourceSystemID, 
        source.Attribute1,
        source.Attribute2,
        'I'
            ,GETDATE()
            ,'G',
        source.Lnd_hash,
        getdate(),
        '12/31/9999'
    ) -- the end of the merge statement
    as changes

    (
      action, 
      SourceSystemID, 
      Attribute1,
      Attribute2,
      UpdateType
    ,UpdateDate
    ,QualityInd
      , lnd_hash,
      EffectiveDate,
      EndDate
    )
    where action='UPDATE';

        declare @updatecount int
    declare @insertcount int

    select @updatecount= @@ROWCOUNT;

    --DROP TABLE dbo.tblDimSCDType2Example
    --DROP TABLE dbo.tblStaging


    --TRUNCATE TABLE dbo.tblDimSCDType2Example
    --TRUNCATE TABLE dbo.tblStaging


    --Code Sample 2
    --=============================================================================
    -- Start of Day 1 - truncate the staging table

    truncate table dbo.tblStaging

    -- insert a new record into the staging table into the staging table

    insert into dbo.tblStaging
    (SourceSystemID, Attribute1,   Attribute2)
    values (1      , 'Mary Brown', 'Single' )

    -- insert a new record into the staging table into the staging table

    insert into dbo.tblStaging
    (SourceSystemID, Attribute1,  Attribute2)
    values (2      , 'Ricky Green', 'Married'  )


    truncate table dbo.tblStaging

    -- insert a new record into the staging table

    insert into dbo.tblStaging
    (SourceSystemID, Attribute1, Attribute2)
    values (3      , 'Jane Doe', 'Single'  )

    -- insert a new record into the staging table

    insert into dbo.tblStaging
    (SourceSystemID, Attribute1, Attribute2)
    values (4      , 'John Doe', 'Married' )

    -- insert a changed record into the staging table

    insert into dbo.tblStaging
    (SourceSystemID, Attribute1,       Attribute2)
    values (2      , 'Ricky L. Green', 'Married' )


    truncate table dbo.tblStaging

    -- insert a changed record into the staging table

    insert into dbo.tblStaging
    (SourceSystemID, Attribute1,       Attribute2)
    values (3      , 'Jane Doe-Jones', 'Married' )

    -- insert a changed record into the staging table

    insert into dbo.tblStaging
    (SourceSystemID, Attribute1,  Attribute2)
    values (4      , 'John Doe', 'Married'  )

    -- insert a new record into the staging table

    insert into dbo.tblStaging
    (SourceSystemID, Attribute1,   Attribute2)
    values (5      , 'Bill Smith', 'Married' )

    -- insert a changed record into the staging table

    insert into dbo.tblStaging
    (SourceSystemID, Attribute1,   Attribute2)
    values (1      , 'Mary Brown', 'Single'  )


    --select * from tblStaging

    --select * from dbo.tblDimSCDType2Example
    --order by 2

运行SQL时禁用缓存-Google大查询

3 个答案: