使用params从SQL Server读取:pandas(或pyodbc)无法正常运行

时间:2016-05-05 13:00:03

标签: python sql sql-server pandas pyodbc

我在SQL Server中使用一个查询,该查询需要一个范围来检查数字是否在该范围内(例如,在下面检查DemographicGroupDimID是否为(1,2或3)。在做了一些谷歌搜索后,我发现能够做到这一点的唯一方法是:

SQL

DECLARE @adults table (Id int)
INSERT INTO @adults VALUES (1), (2), (3)

SELECT [date], [station], [impression] = SUM([impressions]) / COUNT(DISTINCT [datetime] )
       FROM 
       (SELECT [datetime] = DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), [date] = ddt.DateKey, [station] = nd.Name, [impressions] = SUM(naf.Impression)
       FROM [Nielsen].[dbo].[NielsenAnalyticsFact] as naf
       LEFT JOIN [dbo].[DateDim] AS ddt
       ON naf.StartDateDimID = ddt.DateDimID
       LEFT JOIN [dbo].NetworkDim as nd
       ON naf.NetworkDimID = nd.NetworkDimID
       LEFT JOIN [dbo].TimeDim as td
       ON naf.QuarterHourDimID = td.TimeDimID
       WHERE (naf.NielsenMarketDimID = 1
                     AND naf.RecordTypeDimID = 2
                     AND naf.AudienceEstimateTypeDimID = 1
                     AND naf.DailyOrWeeklyDimID = 1
                     AND naf.RecordSequenceCodeDimID = 5
                     AND naf.ViewingTypeDimID = 4
                     AND naf.QuarterHourDimID IS NOT NULL
                     AND naf.DemographicGroupDimID < 31
                     AND nd.Affiliation = 'Cable'
                     AND naf.NetworkDimID != 1278
                     AND naf.DemographicGroupDimID in (SELECT Id FROM @adults))
       GROUP BY DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), nd.Name, ddt.DateKey) 
AS grouped_table
GROUP BY [date], [station]
ORDER BY [date], [station]

如果我需要使用不同的范围动态执行此操作,则会失败,如下所示:

Pandas查询

from queries import DB_CREDENTIALS
import pyodbc
import pandas as pd

sql_ = """DECLARE @adults table (Id int)
INSERT INTO @adults VALUES ?

SELECT [date], [station], [impression] = SUM([impressions]) / COUNT(DISTINCT [datetime] )
       FROM
       (SELECT [datetime] = DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), [date] = ddt.DateKey, [station] = nd.Name, [impressions] = SUM(naf.Impression)
       FROM [Nielsen].[dbo].[NielsenAnalyticsFact] as naf
       LEFT JOIN [dbo].[DateDim] AS ddt
       ON naf.StartDateDimID = ddt.DateDimID
       LEFT JOIN [dbo].NetworkDim as nd
       ON naf.NetworkDimID = nd.NetworkDimID
       LEFT JOIN [dbo].TimeDim as td
       ON naf.QuarterHourDimID = td.TimeDimID
       WHERE (naf.NielsenMarketDimID = 1
                     AND naf.RecordTypeDimID = 2
                     AND naf.AudienceEstimateTypeDimID = 1
                     AND naf.DailyOrWeeklyDimID = 1
                     AND naf.RecordSequenceCodeDimID = 5
                     AND naf.ViewingTypeDimID = 4
                     AND naf.QuarterHourDimID IS NOT NULL
                     AND naf.DemographicGroupDimID < 31
                     AND nd.Affiliation = 'Cable'
                     AND naf.NetworkDimID != 1278
                     AND naf.DemographicGroupDimID in (SELECT Id FROM @adults))
       GROUP BY DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), nd.Name, ddt.DateKey)
AS grouped_table
GROUP BY [date], [station]
ORDER BY [date], [station]"""

with pyodbc.connect(DB_CREDENTIALS) as cnxn:
    df = pd.read_sql(sql=sql_, con=cnxn, params=['(30)'])

错误:

---------------------------------------------------------------------------
DatabaseError                             Traceback (most recent call last)
<ipython-input-5-4b63847d007f> in <module>()
      1 with pyodbc.connect(DB_CREDENTIALS) as cnxn:
----> 2     df = pd.read_sql(sql=sql_, con=cnxn, params=['(30)'])

C:\Users\mburke\AppData\Local\Continuum\Anaconda64\lib\site-packages\pandas\io\sql.pyc in read_sql(sql, con, index_col, coerce_float, params, parse_dates, columns, chunksize)
    497             sql, index_col=index_col, params=params,
    498             coerce_float=coerce_float, parse_dates=parse_dates,
--> 499             chunksize=chunksize)
    500 
    501     try:

C:\Users\mburke\AppData\Local\Continuum\Anaconda64\lib\site-packages\pandas\io\sql.pyc in read_query(self, sql, index_col, coerce_float, params, parse_dates, chunksize)
   1593 
   1594         args = _convert_params(sql, params)
-> 1595         cursor = self.execute(*args)
   1596         columns = [col_desc[0] for col_desc in cursor.description]
   1597 

C:\Users\mburke\AppData\Local\Continuum\Anaconda64\lib\site-packages\pandas\io\sql.pyc in execute(self, *args, **kwargs)
   1570             ex = DatabaseError(
   1571                 "Execution failed on sql '%s': %s" % (args[0], exc))
-> 1572             raise_with_traceback(ex)
   1573 
   1574     @staticmethod

C:\Users\mburke\AppData\Local\Continuum\Anaconda64\lib\site-packages\pandas\io\sql.pyc in execute(self, *args, **kwargs)
   1558                 cur.execute(*args, **kwargs)
   1559             else:
-> 1560                 cur.execute(*args)
   1561             return cur
   1562         except Exception as exc:

DatabaseError: Execution failed on sql 'DECLARE @adults table (Id int)
INSERT INTO @adults VALUES ?

SELECT [date], [station], [impression] = SUM([impressions]) / COUNT(DISTINCT [datetime] )
       FROM
       (SELECT [datetime] = DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), [date] = ddt.DateKey, [station] = nd.Name, [impressions] = SUM(naf.Impression)
       FROM [Nielsen].[dbo].[NielsenAnalyticsFact] as naf
       LEFT JOIN [dbo].[DateDim] AS ddt
       ON naf.StartDateDimID = ddt.DateDimID
       LEFT JOIN [dbo].NetworkDim as nd
       ON naf.NetworkDimID = nd.NetworkDimID
       LEFT JOIN [dbo].TimeDim as td
       ON naf.QuarterHourDimID = td.TimeDimID
       WHERE (naf.NielsenMarketDimID = 1
                     AND naf.RecordTypeDimID = 2
                     AND naf.AudienceEstimateTypeDimID = 1
                     AND naf.DailyOrWeeklyDimID = 1
                     AND naf.RecordSequenceCodeDimID = 5
                     AND naf.ViewingTypeDimID = 4
                     AND naf.QuarterHourDimID IS NOT NULL
                     AND naf.DemographicGroupDimID < 31
                     AND nd.Affiliation = 'Cable'
                     AND naf.NetworkDimID != 1278
                     AND naf.DemographicGroupDimID in (SELECT Id FROM @adults))
       GROUP BY DATEADD(minute,td.Minute,DATEADD(hour,td.NielsenLocalHour,CONVERT(smalldatetime, ddt.DateKey))), nd.Name, ddt.DateKey)
AS grouped_table
GROUP BY [date], [station]
ORDER BY [date], [station]': ('42000', "[42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Incorrect syntax near '@P1'. (102) (SQLExecDirectW); [42000] [Microsoft][ODBC SQL Server Driver][SQL Server]Statement(s) could not be prepared. (8180)")

这是因为declare语句需要在select语句本身的范围内吗?我不确定pandas如何处理pyodbc光标对象,因此我不确定此错误源自何处。

编辑:请注意,我在这个实例中传递的参数是(30)只是为了使用当范围中只有一个数字失败时的简单情况。对于像(1), (2), (3)这样的更复杂的字符串,它当然也失败了,就像上面的例子一样。

1 个答案:

答案 0 :(得分:6)

如果在SQL中使用prepared statements,则无法为一个占位符/参数/绑定变量放置多个值!

除此之外,您只能使用占位符/参数/绑定变量来代替literals,您不能将它用于文字的SQL语句的一部分。

在您的情况下,您尝试将()作为SQL的一部分,但将文字作为参数。

使用参数/预备语句/绑定变量也可以保护您免受某些SQL注入。

说,尝试按如下方式更改代码:

变化

INSERT INTO @adults VALUES ?

INSERT INTO @adults VALUES (?)

df = pd.read_sql(sql=sql_, con=cnxn, params=['(30)'])

df = pd.read_sql(sql=sql_, con=cnxn, params=['30'])

<强>更新

您可以这样准备SQL:

In [9]: vals = [20,30,40]

In [32]: vals
Out[32]: [20, 30, 40]

In [33]: ' (?)' * len(vals)
Out[33]: ' (?) (?) (?)'

然后:

In [14]: sql_ = """DECLARE @adults table (Id int)
   ....: INSERT INTO @adults VALUES {}
   ....:
   ....: SELECT [date],
   ....: """

In [15]: sql_.format(' (?)' * len(vals))
Out[15]: 'DECLARE @adults table (Id int)\nINSERT INTO @adults VALUES (?) (?) (?)\n\nSELECT [date],\n'

注意生成的(?) (?) (?)

最后调用你的SQL:

df = pd.read_sql(sql=sql_.format(' (?)' * len(vals)), con=cnxn, params=vals)