TSV上的“ BCP复制失败”

时间:2018-08-22 02:53:20

标签: sql-server bcp

注意:这与我在此处发布的问题("BCP copy in failed" on tsv (no further error message info))非常相似,但是我用另一种解决方案解决了该问题,这种解决方案在这种情况下似乎不起作用。 (问题的标题类似,因此任何其他具有相同模糊错误的人都可以找到解决问题的多种可能的解决方案。)


尝试在Linux CentOS7上使用BCP utilityhttps://docs.microsoft.com/en-us/sql/tools/bcp-utility?view=sql-server-2017)将tsv文件复制到远程MSSQL Server DB(在Windows Server 2012计算机中),得到令人沮丧的简短错误消息

  

开始复制...

     

BCP复制失败

有问题的BCP bash脚本具有以下形式

TO_SERVER_ODBCDSN="-D -S MyMSSQLServer"
TO_SERVER_IP="-S 172.99.9.29"
DB="mydb"
TABLE="mytable" 
# getting MSSQL Server credentials
USER=$(tail -n+1 $basedir/src/mssql-creds.txt | head -1)
PASSWORD=$(tail -n+2 $basedir/src/mssql-creds.txt | head -1)
DATAFILES="$storagedir/tsv"
TARGET_GLOB="*.tsv"
RECOMMEDED_IMPORT_MODE='-c' # see https://stackoverflow.com/a/16310219/8236733
DELIMITER="\t" # DO NOT use format like "'\t'", nested quotes seem to cause hard-to-catch error

if [[ -z "${TABLE// }" ]]; then
    echo -e "\nNo table specified"
    exit
fi

{
echo -e "Truncating destination table: $DB/$TABLE"
sqlcmd -Q "select count(*) from dbo.$TABLE; truncate table dbo.$TABLE; select count(*) from dbo.$TABLE;" \
    $TO_SERVER_ODBCDSN \
    -U $USER -P $PASSWORD \
    -d $DB
} || { echo -e "\nFailed to truncate MSSQL DB"; exit 255; }

echo -e "\nConnecting with BCP utility as $USER..."

for filename in $DATAFILES/$TARGET_GLOB; do

    if [ ! -f $filename ]; then
        echo -e "\nFile $filename not found!"
    else
        echo -e "\nImporting $filename data to $DB/$TABLE"
    fi

    echo -e "Removing header from TSV file $filename"
    echo "$(tail -n +2 $filename)" > $filename

    echo -e "Replacing null literal values with empty chars"
    NULL_WITH_TAB="null\t" # WARN: assumes the first field is prime-key so never null
    TAB="\t"
    sed -i -e "s/$NULL_WITH_TAB/$TAB/g" $filename

    echo -e "Starting BCP export threads for $filename"
    /opt/mssql-tools/bin/bcp "$TABLE" in "$filename" \
        $TO_SERVER_ODBCDSN \
        -U $USER -P $PASSWORD \
        -d $DB \
        $RECOMMEDED_IMPORT_MODE \
        -t "\t" \
        &

done

(提供的不仅仅是这里的最小示例,因为我不知道为什么BCP程序为何仅对某些数据集失败而对其他数据集工作,因为错误消息如此含糊)并运行它会产生输出

[me@mapr001 src]$ time ./hdfs2mssql.pq.sh mytable

Truncating destination table: mydb/mytable

-----------
          0

(1 rows affected)

-----------
          0

(1 rows affected)

Connecting with BCP utility as myuser...

Importing /mapr/my.cluster.local//etl/path/to/MYTABLE/tsv/0_0_0.tsv data to mydb/mytable
Removing header from TSV file /mapr/uceramapr.cluster.local//etl/path/to/MYTABLE/tsv/0_0_0.tsv
Replacing null literal values with empty chars
Starting BCP export threads for /mapr/uceramapr.cluster.local//etl/path/to/MYTABLE/tsv/0_0_0.tsv

Starting copy...

BCP copy in failed


All export tasks collected and completed

real    0m2.182s
user    0m0.826s
sys 0m0.624s

由此,它告诉一些调试信息。

  1. 它告诉我sqlcmd能够实际连接到远程数据库(因为能够成功截断并获得计数),即。不太可能是网络连接问题。
  2. 还告诉我们,问题(可能)与我在另一篇SO帖子中解决的有关BCP无法对TSV文件("BCP copy in failed" on tsv (no further error message info))执行“ in”操作的问题不同,因为它已经替换了带有标签“ \ t”的文字“空”字符串。

尝试通过awk '{print gsub(/\t/,"")}' /path/to/0_0_0.tsv检查TSV文件中每一行的选项卡数量似乎将所有行显示为具有149个选项卡。这似乎与远程数据库目标表的模式中的149 + 1 = 150 相匹配(如下所示通过sp_help <tablename>进行格式化,如果格式不正确,则感到抱歉):

Column_name Type    Computed    Length  Prec    Scale   Nullable    TrimTrailingBlanks  FixedLenNullInSource    Collation
PROC_ID float   no  8   53      NULL    yes (n/a)   (n/a)   NULL
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    float   no  8   53      NULL    yes (n/a)   (n/a)   NULL
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    smalldatetime   no  4                   yes (n/a)   (n/a)   NULL
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    nvarchar    no  510                 yes (n/a)   (n/a)   SQL_Latin1_General_CP1_CI_AS
xxxx    smalldatetime   no  4                   yes (n/a)   (n/a)   NULL

这是tsv中的模拟示例行

272647  KXXXX,XXXXSR TRNS>10KILO    31999999                                        19999               57              Y   2   9/1/17 XXX,NO VOL X XXX (XXX XXX XXXX)              Y   999                                             1999    2999                            999         3                       1   1                           Y               2012-05-31 17:41:00         1   KXX,PRXXXX T                                                                                    999                                                                                                     XXXT,PRXXXX TXXX>10KILO                                                                                                                     2018-08-21 14:04:57

此处带有制表符转义符

272647^IKXXXX,XXXXSR TRNS>10KILO^I31999999^I^I^I^I^I^I^I^I^I^I19999^I^I^I^I57^I^I^I^IY^I2^I9/1/17 XXX,NO VOL X XXX (XXX XXX XXXX)^I^I^I^IY^I999^I^I^I^I^I^I^I^I^I^I^I^I1999^I2999^I^I^I^I^I^I^I999^I^I^I3^I^I^I^I^I^I1^I1^I^I^I^I^I^I^IY^I^I^I^I2012-05-31 17:41:00^I^I^I1^IKXX,PRXXXX T^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I999^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^IXXXT,PRXXXX TXXX>10KILO^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I^I2018-08-21 14:04:57

就提供最少的工作示例而言,脚本中的代码为

#!/bin/bash

filename=$1

# can use ODBC driver...
TO_SERVER_ODBCDSN="-D -S MyMSSQLServer"
# ...or host IP
TO_SERVER_IP="-S <your MSSQL Server IP>"
DB="mydb"
TABLE="mytable"
USER=<your MSSQL Server login>
PASSWORD=<your MSSQL Server login password>
DATAFILES=</path/to/tsv/file(s)>
TARGET_GLOB="*.tsv"
RECOMMEDED_IMPORT_MODE='-c' # see https://stackoverflow.com/a/16310219/8236733
DELIMITER="\t" # DO NOT use format like "'\t'", nested quotes seem to cause hard-to-catch error

echo -e "Removing header from TSV file $filename"
echo "$(tail -n +2 $filename)" > $filename
echo "First line of file is now..."    
echo $(head -n 1 $filename)

echo -e "Replacing null literal values with empty chars"
NULL_WITH_TAB="null\t" # WARN: assumes the first field is prime-key so never null
TAB="\t"
sed -i -e "s/$NULL_WITH_TAB/$TAB/g" $filename

echo -e "Starting BCP export threads for $filename"
/opt/mssql-tools/bin/bcp "$TABLE" in "$filename" \
    $TO_SERVER_ODBCDSN \
    -U $USER -P $PASSWORD \
    -d $DB \
    $RECOMMEDED_IMPORT_MODE \
    -t "\t" \

(其中一些值必须由用户提供,因为取决于系统如何设置MSSQL Server和ODBC驱动程序等)。但是,无法为此提供示例数据,因为我当前正在使用的数据是敏感数据,而且我不知道数据集是什么引起了错误(如我所说,该脚本似乎适用于某些数据集而不适用于其他数据集) ),以便在线生成一些会产生类似错误的模拟数据集。


使用不同的表和数据文件 起作用尝试此命令,此时错误消息太短了,无法给我更多调试信息(从未使用过) BCP实用工具之前)。如果有人有使用此工具的经验,将不胜感激任何调试建议或如何解决此问题。

1 个答案:

答案 0 :(得分:0)

最终找到了解决方案,并且似乎与BCP没有关系(尽管问题标题照原样,因为BCP是发现问题的地方,所以其他人可能会找到这个帖子)。

TLDR:

TSV中的datetime值溢出了目标表中的destination字段,因为该表模式期望使用smalldate类型,但是在源端,我们将数据视为我们认为目标可以处理该特定列中的datetime类型。

故事的寓意

在调试ETL /数据传输问题时,请确保检查源与目标之间是否存在任何可疑的数据类型冲突

长版:

在引起问题的实际数据集文件上使用 binarysearch调试方法(例如,使用数据的前半部分,看看会发生什么,一半发生,再试一次,等等),能够避免出现“ BCP复制失败”错误,但是后来发现(即使未抛出任何失败消息,但仍复制了0行(所以我真的不知道为什么“ BCP复制失败”错误不再弹出))。将-e选项添加到最小示例中,以获取此处的bcp复制尝试的错误日志(例如-e filename.bcperror.log),看到错误

  

#@第1行,第150列:日期时间字段溢出。小数秒精度超出了参数绑定中指定的范围。 @#

在生成的错误文件的每一行的顶部(列150是该行的最后一列,例如2018-08-29 11:34:14)。

在查看BCP试图复制到MSSQL Server中的表时,我注意到(与BCP成功写入的表不同),在我的情况下,最后一个字段(第150列)设置为{{1 }}类型,而其他表则使用smalldate。同样,将该字段更改为datetime,BCP就能将有问题的TSV复制到表中而不会出现问题。