Question

我已经从数据库中创建了一个包含三列（id，author，title）的镶木地板文件，并希望使用条件读取镶木地板文件（title =＆＃39; Learn Python＆＃39;）。下面提到的是我用于此POC的python代码。

import pyarrow as pa
import pyarrow.parquet as pq
import pandas as pd
import pyodbc

def write_to_parquet(df, out_path, compression='SNAPPY'):
arrow_table = pa.Table.from_pandas(df)
if compression == 'UNCOMPRESSED':
    compression = None
pq.write_table(arrow_table, out_path, use_dictionary=False,
               compression=compression)

def read_pyarrow(path, nthreads=1):
return pq.read_table(path, nthreads=nthreads).to_pandas()


path = './test.parquet'
sql = "SELECT * FROM [dbo].[Book] (NOLOCK)"

conn = pyodbc.connect(r'Driver={SQL 
Server};Server=.;Database=APP_BBG_RECN;Trusted_Connection=yes;')
df = pd.io.sql.read_sql(sql, conn)

write_to_parquet(df, path)

df1 = read_pyarrow(path)

如何在read_pyarrow方法中添加条件（title =＆＃39; Learn Python＆＃39;）？

Answer 1

目前尚不支持此功能。我们打算将来开发此功能。我建议在从Arrow表转换后使用pandas进行过滤。

Answer 2

过滤器现在可用 read_table

table = pq.read_table(
        df, filters=[("title", "in", {'Learn Python'}), 
                     ("year", ">=", 1950)]
    )

如何在Python中使用pyarrow读取带有条件的镶木地板文件

2 个答案: