Question

这是我的数据：

"2001-08-10 01:09:29","3820166553","<P><TABLE border=0 cellSpacing=0 cellPadding=0 width="100%"><TBODY><TR><TD style="BACKGROUND-COLOR: #eee">
<TABLE border=0 cellSpacing=0 cellPadding=0 align=center><TBODY><TR><TD style="BACKGROUND-COLOR: #fff" class=maincontent><TABLE style="MARGIN: 0px auto" border=0 cellSpacing=0 cellPadding=0 width=580 align=center><TBODY><TR><TD colSpan=2><A href="general/forms/YHMN?ci=0c01026bd7fbc394&amp;ags=D778083&amp;cn=KOBY&amp;pfnn="><IMG border=0 alt="Thanks for contacting us click here button" src="http:/edm/tcp-images/yhan-edm-banner-purple_button.jpg" width=580 height=255></A></TD></TR><TR><TD style="PADDING-BOTTOM: 35px; PADDING-LEFT: 32px; PADDING-RIGHT: 32px; PADDING-TOP: 35px" colSpan=2><P style="TEXT-ALIGN: left; LINE-HEIGHT: 18px; MARGIN-TOP: 0px; FONT-FAMILY: Arial, Helvetica, sans-serif; COLOR: #414141; FONT-SIZE: 14px" align=left></P><P style="TEXT-ALIGN: left; LINE-HEIGHT: 18px; FONT-FAMILY: Arial, Helvetica, sans-serif; COLOR: #414141; FONT-SIZE: 14px" align=left>If...",,"778083"

猪脚本：

stage = LOAD '/filename' USING org.apache.pig.piggybank.storage.CSVExcelStorage(',','YES_MULTILINE','UNIX','READ_INPUT_HEADER') as (col1,col2,col3,col4,col5);
col3_data=FOREACH stage GENERATE col3;
DUMP col3_data;

输出：

(<P><TABLE border=0 cellSpacing=0 cellPadding=0 width=100%"><TBODY><TR><TD style=BACKGROUND-COLOR: #eee">
<TABLE border=0 cellSpacing=0 cellPadding=0 align=center><TBODY><TR><TD style=BACKGROUND-COLOR: #fff" class=maincontent><TABLE style=MARGIN: 0px auto" border=0 cellSpacing=0 cellPadding=0 width=580 align=center><TBODY><TR><TD colSpan=2><A href=general/forms/YHMN?ci=0c01026bd7fbc394&amp;ags=D778083&amp;cn=KOBY&amp;pfnn="><IMG border=0 alt=Thanks for contacting us click here button" src=http:/edm/tcp-images/yhan-edm-banner-purple_button.jpg" width=580 height=255></A></TD></TR><TR><TD style=PADDING-BOTTOM: 35px; PADDING-LEFT: 32px; PADDING-RIGHT: 32px; PADDING-TOP: 35px" colSpan=2><P style=TEXT-ALIGN: left; LINE-HEIGHT: 18px; MARGIN-TOP: 0px; FONT-FAMILY: Arial)

我正在使用csvExcelStorage在pig中加载我的数据文件。文件中的一列具有行尾字符。数据文件看起来像

"2015-08-17 23:55:59","12345","<P>this is test data,
<TR>
<\TR><BODY>Text-Align: Arial, Roman,feed this; end of input...","column 4"

Load语句如下：

stage=LOAD '/data/file' using csvExcelStorage(',','YES_MULTILINE','NOCHANGE','READ_INPUT_HEADER') as (col1,col2,col3,col4)

在我转储数据时加载后，我得到以下

col3 = <P>this is test data,
<TR>
<\TR><BODY>"Text-Align: Arial

和

col4 = Roman

因此col3包含不完整的数据，而col4包含错误的数据。

请你帮忙找出问题所在。

Answer 1

尝试上面的用例，只有改变是参考CSVExcelStorage.Its获取field3的预期值。

我相信您已将别名定义为csvExcelStorage来引用。到这堂课。

猪版：Apache Pig版本0.14.0（r1640057）于2014年11月16日18:01:24编译。

输入：

"2015-08-17 23:55:59","12345","<P>this is test data,
<TR>
<\TR><BODY>Text-Align: Arial, Roman,feed this; end of input...","column 4"

猪脚本：

csvexceldata = LOAD 'csvdata.csv' USING org.apache.pig.piggybank.storage.CSVExcelStorage(',','YES_MULTILINE','NOCHANGE','READ_INPUT_HEADER') AS (col1,col2,col3,col4);
col3_data = FOREACH csvexceldata GENERATE col3;
DUMP col3_data;

输出

(<P>this is test data,
<TR>
<\TR><BODY>Text-Align: Arial, Roman,feed this; end of input...)

Apache Pig CsvExcelStorage

1 个答案: