Question

Netezza外部表的间歇性问题。

外部表失败，系统自身生成的文件（外部表生成的文件不是来自其他来源）。但是我们尝试通过nzload实用程序将同一个文件加载到另一个表，并且没有任何问题这个问题不一致，大多数时候都无法复制。

CREATE EXTERNAL TABLE SP_PORTFOLIO_EXT_DATA_6128_140
(
    CLIENT_ID INTEGER,
    CONFIG_ID INTEGER,
    SCENARIO_ID INTEGER,
    PORTFOLIO_ID INTEGER,
    PORTFOLIO_NAME CHARACTER VARYING(200),
    CUSTOM13 CHARACTER VARYING(600),
    CUSTOM12 CHARACTER VARYING(500),
    CUSTOM11 CHARACTER VARYING(500),
    CUSTOM10 CHARACTER VARYING(500),
    CUSTOM9 CHARACTER VARYING(500),
    CUSTOM8 CHARACTER VARYING(500),
    CUSTOM7 CHARACTER VARYING(500),
    CUSTOM6 CHARACTER VARYING(2000),
    CUSTOM3 CHARACTER VARYING(500),
    CUSTOM2 CHARACTER VARYING(3000),
    CUSTOM1 CHARACTER VARYING(500),
    CREATIVE CHARACTER VARYING(512),
    PLACEMENT CHARACTER VARYING(5000),
    IMPRESSIONS NUMERIC(38,0),
    CLICKS NUMERIC(38,0),
    CONVERSIONS INTEGER,
    TRUE_CONVERSIONS NUMERIC(38,6),
    OPTMETRIC NUMERIC(38,6),
    LASTAD_OPTMETRIC NUMERIC(38,6),
    CURRSPEND NUMERIC(38,6)
)
USING
(
    DATAOBJECT('/san5/Netezza/CAR/CAR_ZEUS/SPBU/test/SP_PORTFOLIO_EXT_DATA_6128_140.csv')
    DELIMITER 254
    ESCAPECHAR '/'
    TIMESTYLE '24HOUR'
    LOGDIR '/tmp'
    Y2BASE 2000
    ENCODING 'internal'
);

命令成功完成

select COUNT(*) from SP_PORTFOLIO_EXT_DATA_6128_140;
ERROR [HY000] ERROR:  External Table : count of bad input rows reached maxerrors limit

NZLOAD方法

CREATE TABLE TEST_LOAD
(
    CLIENT_ID INTEGER,
    CONFIG_ID INTEGER,
    SCENARIO_ID INTEGER,
    PORTFOLIO_ID INTEGER,
    PORTFOLIO_NAME CHARACTER VARYING(200),
    CUSTOM13 CHARACTER VARYING(600),
    CUSTOM12 CHARACTER VARYING(500),
    CUSTOM11 CHARACTER VARYING(500),
    CUSTOM10 CHARACTER VARYING(500),
    CUSTOM9 CHARACTER VARYING(500),
    CUSTOM8 CHARACTER VARYING(500),
    CUSTOM7 CHARACTER VARYING(500),
    CUSTOM6 CHARACTER VARYING(2000),
    CUSTOM3 CHARACTER VARYING(500),
    CUSTOM2 CHARACTER VARYING(3000),
    CUSTOM1 CHARACTER VARYING(500),
    CREATIVE CHARACTER VARYING(512),
    PLACEMENT CHARACTER VARYING(5000),
    IMPRESSIONS NUMERIC(38,0),
    CLICKS NUMERIC(38,0),
    CONVERSIONS INTEGER,
    TRUE_CONVERSIONS NUMERIC(38,6),
    OPTMETRIC NUMERIC(38,6),
    LASTAD_OPTMETRIC NUMERIC(38,6),
    CURRSPEND NUMERIC(38,6)
)
DISTRIBUTE ON RANDOM;

# Loading data  from the same file using Nzload

nzload -host 10.200.29.30 -u xxxxx -pw xxxxx -db SPBU_REPORT_DB_TEST -t test_load -delim 254 -ctrlChars  -df /san5/Netezza/CAR/CAR_ZEUS/SPBU/test/SP_PORTFOLIO_EXT_DATA_6128_140.csv

Load session of table 'TEST_LOAD' completed successfully

[ja.prod@inet11026 ~]$ cat /san5/Netezza/CAR/CAR_ZEUS/SPBU/test/SP_PORTFOLIO_EXT_DATA_6128_140.csv|wc -l
191322

select count(*) from test_load;
191322

添加nzlog

File Buffer Size (MB): 8                  Load Replay Region (MB): 0
  Encoding:              INTERNAL           Max errors:            1
  Skip records:          0                  Max rows:              0
  FillRecord:            No                 Truncate String:       No
  Escape Char:           '/'                Accept Control Chars:  No
  Allow CR in string:    No                 Ignore Zero:           No
  Quoted data:           NO                 Require Quotes:        No

  BoolStyle:             1_0                Decimal Delimiter:     '.'

  Disable NFC:           No
  Date Style:            YMD                Date Delim:            '-'
  Y2Base:                2000
  Time Style:            24HOUR             Time Delim:            ':'
  Time extra zeros:      No

Found bad records

bad #: input row #(byte offset to last char examined) [field #, declaration] diagnostic, "text consumed"[last char examined]
----------------------------------------------------------------------------------------------------------------------------
1: 25(184) [21, INT4] expected field delimiter or end of record, "0"[.]

Statistics

  number of records read:      25
  number of bad records:       1
  -------------------------------------------------
  number of records loaded:    0

  Elapsed Time (sec): 0.0

-----------------------------------------------------------------------------
Load completed at: 08-Oct-15 09:59:04 EDT

包含坏行的.nzbad数据（管道符号代表实际分隔符的可读性）：

140|1305|6128||NULL|SEO|SEO|test.com/vehicledetail/detail/632888199/overview|SEO|SEO|SEO|SEO Brand|SEO Brand|best Tracking|Google(Seo)|SEO|Impression Tracker|Unknown|0|1|0|0.000000|0.000000|0.000000|0.000000

Answer 1

从nzlog我们可以看出负载在第25行失败了。具体来说，当它尝试加载第21列时，它遇到的值不是整数。

日志显示它遇到0，然后是句点。所以数据可能有0.0或0.1234这样的东西，不能作为整数加载。

bad #: input row #(byte offset to last char examined) [field #, declaration] diagnostic, "text consumed"[last char examined]
----------------------------------------------------------------------------------------------------------------------------
1: 25(184) [21, INT4] expected field delimiter or end of record, "0"[.]

使用您提供的.nzbad数据（此处使用＆＃39; |＆＃39;而不是您的实际分隔符以便于阅读）：

140|1305|6128||NULL|SEO|SEO|test.com/vehicledetail/detail/632888199/overview|SEO|SEO|SEO|SEO Brand|SEO Brand|best Tracking|Google(Seo)|SEO|Impression Tracker|Unknown|0|1|0|0.000000|0.000000|0.000000|0.000000

我注意到的一件事是你有一个varchar字段＆＃39; /＆＃39;在里面。外部表和nzload方法之间的区别之一是外部表指定了escapechar＆＃39; /＆＃39;虽然nzload没有。

你会发现你的数据＆＃39; test.com/vehicledetail/detail/632888199/overview'将被加载为＆＃39; test.comvehicledetaildetail632888199overview＆＃39;因为＆＃39; /＆＃39;字符将被删除，因为它们本身不会被转义（例如＆＃39; //＆＃39;）。

如果＆＃39; /＆＃39;直接在数据中的列分隔符之前，它将指示它将列分隔符视为数据的一部分，并且会认为数据中的第22列实际上是表中的第21列，这与我们在此处看到的相匹配

Answer 2

ScottMcG正如你所说，我比较了Nzload和External表生成的nzlog文件，发现转义字符是唯一的区别。所以我注释掉那个部分并再次尝试并且工作正常。

CREATE EXTERNAL TABLE SP_PORTFOLIO_EXT_DATA_6128_140

（ CLIENT_ID INTEGER， CONFIG_ID INTEGER， SCENARIO_ID INTEGER， PORTFOLIO_ID INTEGER， PORTFOLIO_NAME CHARACTER VARYING（200）， CUSTOM13 CHARACTER VARYING（600）， CUSTOM12 CHARACTER VARYING（500）， CUSTOM11 CHARACTER VARYING（500）， CUSTOM10 CHARACTER VARYING（500）， CUSTOM9 CHARACTER VARYING（500）， CUSTOM8 CHARACTER VARYING（500）， CUSTOM7 CHARACTER VARYING（500）， CUSTOM6 CHARACTER VARYING（2000）， CUSTOM3 CHARACTER VARYING（500）， CUSTOM2 CHARACTER VARYING（3000）， CUSTOM1 CHARACTER VARYING（500），创造性角色变化（512）， PLACEMENT CHARACTER VARYING（5000）， IMPRESSIONS NUMERIC（38,0），点击数字（38,0）， CONVERSIONS INTEGER， TRUE_CONVERSIONS NUMERIC（38,6）， OPTMETRIC NUMERIC（38,6）， LASTAD_OPTMETRIC NUMERIC（38,6）， CURRSPEND NUMERIC（38,6））使用（数据对象（＆＃39; /san5/Netezza/CAR/CAR_ZEUS/SPBU/test/SP_PORTFOLIO_EXT_DATA_6128_140.csv'） DELIMITER 254 TIMESTYLE＆＃39; 24小时＆＃39; LOGDIR＆＃39; / tmp＆＃39; Y2BASE 2000 编码＆＃39;内部＆＃39; ）;

从SP_PORTFOLIO_EXT_DATA_6128_140中选择计数（*）;

191322

Answer 3

必须按如下所示更改数据类型：CHARACTER VARYING替换VARCHAR / NVARCHAR

CREATE TABLE TEST_LOAD
(
CLIENT_ID INTEGER,
CONFIG_ID INTEGER,
SCENARIO_ID INTEGER,
PORTFOLIO_ID INTEGER,
PORTFOLIO_NAME VARCHAR(200),
CUSTOM13 VARCHAR(600),
CUSTOM12 VARCHAR(500),
CUSTOM11 VARCHAR(500),
CUSTOM10 VARCHAR(500),
CUSTOM9 VARCHAR(500),
CUSTOM8 VARCHAR(500),
CUSTOM7 VARCHAR(500),
CUSTOM6 VARCHAR(2000),
CUSTOM3 VARCHAR(500),
CUSTOM2 VARCHAR(3000),
CUSTOM1 VARCHAR(500),
CREATIVE VARCHAR(512),
PLACEMENT VARCHAR(5000),
IMPRESSIONS NUMERIC(38,0),
CLICKS NUMERIC(38,0),
CONVERSIONS INTEGER,
TRUE_CONVERSIONS NUMERIC(38,6),
OPTMETRIC NUMERIC(38,6),
LASTAD_OPTMETRIC NUMERIC(38,6),
CURRSPEND NUMERIC(38,6)
)
DISTRIBUTE ON RANDOM;

NZLOAD正在工作，而Netezza中的外部表失败，输入行数达到最大错误数

3 个答案: