vsql / vertica,如何将文本输入文件的名称复制到目标表中

时间:2014-03-18 22:36:48

标签: vertica vsql

我必须将输入文本文件(text_file.txt)复制到表(table_a)。我还需要将输入文件的名称包含在表中。

我的代码是:

\set t_pwd `pwd`
\set input_file '\'':t_pwd'/text_file.txt\'' 

copy table_a
( column1
 ,column2
 ,column3
 ,FileName :input_file
)
from :input_file

最后一行不会复制表格中的输入文本文件名。 如何将输入文本文件的名称复制到表中? (无需手动输入文件名)

2 个答案:

答案 0 :(得分:2)

解决方案1 ​​
 这可能不是您工作的完美解决方案,但我认为可以胜任:

您可以获取表名并将其存储在TBL变量中,然后在CSV文件中每行末尾添加此变量,您将要加载到Vertica中。
 现在,根据您的CSV文件大小,这可能会耗费大量时间和CPU消耗。

export TBL=`ls -1 | grep *.txt` | sed -e 's/$/,'$TBL'/' -i $TBL

示例:

[dbadmin@bih001 ~]$ cat load_data1
1|2|3|4|5|6|7|8|9|10

[dbadmin@bih001 ~]$ export TBL=`ls -1 | grep load*` | sed -e 's/$/|'$TBL'/' -i $TBL

[dbadmin@bih001 ~]$ cat load_data1
1|2|3|4|5|6|7|8|9|10||load_data1

解决方案2
您可以使用DEFAULT CONSTRAINT,请参阅示例:
 的 1。使用 DEFAULT CONSTRAINT

创建表格
[dbadmin@bih001 ~]$ vsql
Password:
Welcome to vsql, the Vertica Analytic Database interactive terminal.

Type:  \h or \? for help with vsql commands
       \g or terminate with semicolon to execute query
       \q to quit

dbadmin=> create table TBL (id int ,CSV_FILE_NAME varchar(200) default 'TBL');
CREATE TABLE
dbadmin=> \dt
              List of tables
 Schema | Name | Kind  |  Owner  | Comment
--------+------+-------+---------+---------
 public | TBL  | table | dbadmin |
(1 row)

查看 DEFAULT CONSTRAINT ,它有' TBL'默认值

dbadmin=> \d TBL
                                       List of Fields by Tables
 Schema | Table |    Column     |     Type     | Size | Default | Not Null | Primary Key | Foreign Key
--------+-------+---------------+--------------+------+---------+----------+-------------+-------------
 public | TBL   | id            | int          |    8 |         | f        | f           |
 public | TBL   | CSV_FILE_NAME | varchar(200) |  200 | 'TBL'   | f        | f           |
(2 rows)

<强> 2。现在设置您的COPY变量
- 插入一些数据并将 DEFAULT CONSTRAINT 值更改为当前的:input_file值。

dbadmin=> \set t_pwd `pwd`
dbadmin=> \set CSV_FILE `ls -1 | grep load*`
dbadmin=> \set input_file '\'':t_pwd'/':CSV_FILE'\''
dbadmin=>
dbadmin=>
dbadmin=> insert into TBL values(1);
 OUTPUT
--------
      1
(1 row)

dbadmin=> select * from TBL;
 id | CSV_FILE_NAME
----+---------------
  1 | TBL
(1 row)

dbadmin=> ALTER TABLE TBL  ALTER COLUMN CSV_FILE_NAME SET DEFAULT :input_file;
ALTER TABLE
dbadmin=> \dt TBL;
              List of tables
 Schema | Name | Kind  |  Owner  | Comment
--------+------+-------+---------+---------
 public | TBL  | table | dbadmin |
(1 row)

dbadmin=> \d TBL;
                                                 List of Fields by Tables
 Schema | Table |    Column     |     Type     | Size |          Default           | Not Null | Primary Key | Foreign Key
--------+-------+---------------+--------------+------+----------------------------+----------+-------------+-------------
 public | TBL   | id            | int          |    8 |                            | f        | f           |
 public | TBL   | CSV_FILE_NAME | varchar(200) |  200 | '/home/dbadmin/load_data1' | f        | f           |
(2 rows)

dbadmin=> insert into TBL values(2);
 OUTPUT
--------
      1
(1 row)

dbadmin=> select * from TBL;
 id |      CSV_FILE_NAME
----+--------------------------
  1 | TBL
  2 | /home/dbadmin/load_data1
(2 rows)

现在,您可以在copy脚本中实现此功能 示例:

\set t_pwd `pwd`
\set CSV_FILE `ls -1 | grep load*`
\set input_file '\'':t_pwd'/':CSV_FILE'\''
ALTER TABLE TBL  ALTER COLUMN CSV_FILE_NAME SET DEFAULT :input_file;
copy TBL from :input_file DELIMITER '|' DIRECT;

解决方案3

使用LOAD_STREAMS
例如:
加载表格时会给它stream name - 这样就可以识别文件名/流名称:

COPY mytable FROM myfile DELIMITER '|' DIRECT STREAM NAME 'My stream name';

*以下是查询load_streams表的方法:*

=> SELECT stream_name, table_name, load_start, accepted_row_count, 
   rejected_row_count, read_bytes, unsorted_row_count, sorted_row_count,
   sort_complete_percent FROM load_streams;
-[ RECORD 1 ]----------+---------------------------
stream_name            | fact-13
table_name             | fact
load_start             | 2010-12-28 15:07:41.132053
accepted_row_count     | 900
rejected_row_count     | 100
read_bytes             | 11975
input_file_size_bytes  | 0
parse_complete_percent | 0
unsorted_row_count     | 3600
sorted_row_count       | 3600
sort_complete_percent  | 100

有道理吗?希望这有帮助!

答案 1 :(得分:1)

如果您不需要纯粹从vsql内部进行操作,可能会有点作弊,并在bash之外导出逻辑,例如:

    FILE=text_file.txt
    (
      while read LINE; do
        echo "$LINE|$FILE"
      done < "$FILE"
    ) | vsql -c 'copy table_a (...) FROM STDIN'

这样你就基本上COPY FROM STDIN,在它到达Vertica之前为每一行添加文件名。