Question

我正在将数据从S3文件复制到Snowflake。我通过尝试将“错误”数据推入雪花来进行测试，因此将varchar推入时间戳列。这是JSON：

 {
    object: "account",
    event: "userRegistered",
    data: {
      accountId: "52cc5f80-9556-40a8-b645-96d01200f831",
      accountNumber: "a8j3edqxl1k",
      firstName: "Ariel",
      lastName: "Cummings",
      email: "Ariel.Cummings809kr5348aj72@e3bmogriim8.ocn3oysypzo.org",
      mobileNumber: "977331296705",
      terms_version: "v4",
      marketingFlags: false,
      marketingChannels: "Text",
      marketingTypes: "Promo",
      provider: "Google"
    },
    eventId: "5089c4a0-d54b-4919-a4b3-348154ff57ce",
    timestamp: "foobarla",
    context: "faker"
  }];

这是我运行的SQL脚本

COPY INTO accounts(timestamp, context, object, event, event_id,
                              schema_version, data, filename, file_row_number) 
                  FROM (SELECT   CAST(parse_json($1):timestamp AS timestamp), 
                                 parse_json($1):context, 
                                 parse_json($1):object, 
                                 parse_json($1):event, 
                                 parse_json($1):event_id, 
                                 null,
                                 parse_json($1):data, 
                                 metadata$filename, 
                                 metadata$file_row_number 
                        FROM @DEV_UNWRAPT_ACCOUNTS_STAGE(file_format => JSON))  
                        pattern='.*dev-unwrapt-data-account.*'
                        on_error = 'skip_file';

当我这样做时，我从Snowflake收到一个错误对象，该错误对象是console.logging。

返回的错误对象是：

error { OperationFailedError: Failed to cast variant value "foobarla" to TIMESTAMP_NTZ
    at createError (C:\Users\206580214\WebstormProjects\unwrapt-analytics-snowflake\terraform\lambda_src\snowflakeImport\node_modules\snowflake-sdk\lib\errors.js:474:15)
    at Object.exports.createOperationFailedError (C:\Users\206580214\WebstormProjects\unwrapt-analytics-snowflake\terraform\lambda_src\snowflakeImport\node_modules\snowflake-sdk\lib\errors.
js:282:10)
    at Object.callback (C:\Users\206580214\WebstormProjects\unwrapt-analytics-snowflake\terraform\lambda_src\snowflakeImport\node_modules\snowflake-sdk\lib\services\sf.js:525:26)
    at Request._callback (C:\Users\206580214\WebstormProjects\unwrapt-analytics-snowflake\terraform\lambda_src\snowflakeImport\node_modules\snowflake-sdk\lib\http\base.js:104:21)
    at Request.self.callback (C:\Users\206580214\WebstormProjects\unwrapt-analytics-snowflake\terraform\lambda_src\snowflakeImport\node_modules\request\request.js:186:22)
    at emitTwo (events.js:126:13)
    at Request.emit (events.js:214:7)
    at Request.<anonymous> (C:\Users\206580214\WebstormProjects\unwrapt-analytics-snowflake\terraform\lambda_src\snowflakeImport\node_modules\request\request.js:1163:10)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (C:\Users\206580214\WebstormProjects\unwrapt-analytics-snowflake\terraform\lambda_src\snowflakeImport\node_modules\request\request.js:1085:12)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:111:20)
    at IncomingMessage.emit (events.js:208:7)
    at endReadableNT (_stream_readable.js:1064:12)
    at _combinedTickCallback (internal/process/next_tick.js:138:11)
    at process._tickDomainCallback (internal/process/next_tick.js:218:9)
  name: 'OperationFailedError',
  code: '100071',
  message: 'Failed to cast variant value "foobarla" to TIMESTAMP_NTZ',
  sqlState: '22000',
  data:
   { internalError: false,
     errorCode: '100071',
     age: 0,
     sqlState: '22000',
     queryId: 'ae3b88e1-7cb1-42de-b66d-9fd987c958d3' } }

是否有任何方法可以在S3中找到从该对象或其他路径发生此错误的文件名？

Answer 1

您可以使用COPY语法的VALIDATION_MODE option浏览文件并报告错误。

或者，您也可以使用VALIDATE命令测试先前运行的COPY语句中发生的情况。

需要找到AWS Lambda将数据从S3复制到Snowflake时发生SQL错误的文件名

1 个答案: