使用pandas to_sql进入VARCHAR(MAX)SQL列时的数据截断

时间:2019-02-14 17:12:09

标签: python sql-server pandas sqlalchemy

问题:

将数据框导入SQL表时,出现截断错误:

sqlalchemy.exc.DataError: (pyodbc.DataError) ('22001', '[22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]String or binary data would be truncated. (8152) (SQLExecDirectW); [22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The statement has been terminated. (3621)')

有问题的列是 ProjectDescription (请参见下面的列列表)。我已引用产生错误的行。我不完全确定这是问题的根源,但我认为该列最初创建为VARCHAR(300)。引起麻烦的值是307个字符长; 9被截断如下。剩下298。原来的大小限制为300,为2表示,我想剩下298?不确定,但是要截短是一个奇怪的长度。

导致问题的代码:

dataframe.to_sql('tblImport', eng, schema='dbo', if_exists='append', index=False, dtype=dict_SQL_dtypes)

尝试的解决方案:

  • 在SQL中设置TEXTSIZE
  • 将列类型更改为VARCHAR / NVARCHAR,将长度更改为1000或MAX
  • 将数据类型(dtype)设置为sqlalchemy.types.VARCHAR(来自here的想法)
  • 在导入之前,按ProjectDescription列的长度降序对数据框进行排序,以将最长的值放在第一行中
  • 如果表的原始长度仍然以某种方式限制了导入,请删除表并以数据类型VARCHAR(MAX)的形式重新创建列
  • 直接通过SQL查询更改列类型

版本:

  • 操作系统:Windows 10
  • python:3.6
  • 熊猫:0.24.1
  • sqlalchemy:1.2.17
  • pyodbc:4.0.25
  • 数据库管理工具:MSSMS
  • SQL服务器12

数据库连接:

pyodbc.connect("DRIVER={{ODBC Driver 13 for SQL Server}};SERVER={0};DATABASE={1};UID={2};PWD={3}".format(serv, db, usr, pwd))

引擎:

create_engine('mssql+pyodbc://{}@sqlserver:{}@{}:1433/{}?driver=ODBC+Driver+13+for+SQL+Server'.format(usr, pwd, serv, db), echo=True)

数据库架构中的ProjectDescription列character_maximum_length:-1

列:

Column Name          | Data Type      | Allow Nulls
ProjectUID           | varchar(60)    | Unchecked
Framework            | char(5)        | Checked
Partner              | char(3)        | Checked
SCUID                | char(9)        | Checked
ClientName           | varchar(255)   | Checked
PartnerProjectNumber | varchar(40)    | Checked
ProjectName          | varchar(255)   | Checked
ProjectDescription   | varchar(MAX)   | Checked

跟踪:

  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 1216, in _execute_context
    cursor, statement, parameters, context
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\default.py", line 533, in do_executemany
    cursor.executemany(statement, parameters)
pyodbc.DataError: ('22001', '[22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]String or binary data would be truncated. (8152) (SQLExecDirectW); [22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The statement has been terminated. (3621)')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "run_for_tests.py", line 258, in <module>
    outmsg = run()
  File "run_for_tests.py", line 252, in run
    process_result = process_data(file_type, file_name, relative_path, dict_clients)
  File "run_for_tests.py", line 204, in process_data
    update_sql_tables(df_base)
  File "server_interaction.py", line 74, in update_sql_tables
    dataframe.to_sql('tblImport', eng, schema='dbo', if_exists='append', index=False, dtype=dict_SQL_dtypes)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pandas\core\generic.py", line 2532, in to_sql
    dtype=dtype, method=method)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pandas\io\sql.py", line 460, in to_sql
    chunksize=chunksize, dtype=dtype, method=method)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pandas\io\sql.py", line 1174, in to_sql
    table.insert(chunksize, method=method)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pandas\io\sql.py", line 686, in insert
    exec_insert(conn, keys, chunk_iter)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\pandas\io\sql.py", line 599, in _execute_insert
    conn.execute(self.table.insert(), data)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 980, in execute
    return meth(self, multiparams, params)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\sql\elements.py", line 273, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 1099, in _execute_clauseelement
    distilled_params,
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 1240, in _execute_context
    e, statement, parameters, cursor, context
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 1458, in _handle_dbapi_exception
    util.raise_from_cause(sqlalchemy_exception, exc_info)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\util\compat.py", line 296, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\util\compat.py", line 276, in reraise
    raise value.with_traceback(tb)
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\base.py", line 1216, in _execute_context
    cursor, statement, parameters, context
  File "AppData\Local\Programs\ArcGIS\Pro\bin\Python\envs\arcgispro-py3\lib\site-packages\sqlalchemy\engine\default.py", line 533, in do_executemany
    cursor.executemany(statement, parameters)
  sqlalchemy.exc.DataError: (pyodbc.DataError) ('22001', '[22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]String or binary data would be truncated. (8152) (SQLExecDirectW); [22001] [Microsoft][ODBC Driver 13 for SQL Server][SQL Server]The statement has been terminated. (3621)') [SQL: 'INSERT INTO dbo.[tblImport] ([ProjectUID], [Framework], [Partner], [SCUID], [ClientName], [PartnerProjectNumber], [ProjectName], [ProjectDescription]) VALUES (?, ?, ?, ?, ?, ?, ?, ?)'] [parameters: (('xxx', 'xxxxx', 'xxx', 'xxxxxxx', 'ABC', 123, 'ABC', 'Provide competition facilities. Build this facility with a capacity of 9000 spectators  ... (9 characters truncated) ... m pool and diving facilities and athletes changing facilities to a competion standard and then, post games, convert it to a council operated facility'), ....)] (Background on this error at: http://sqlalche.me/e/9h9h)

0 个答案:

没有答案