Question

我试图在csv中读到我最终会推送到数据库。我有一个预定的数据模型，数据最终会成为。此外，csv没有标题。

如果我定义一个模型，例如：

class SamFoo(Base):
    __tablename__ = 'foo'

    duns = Column(Text)
    duns_plus_four = Column(Text)
    cage_code = Column(Text)
    dodaac = Column(Text)
    sam_extract_code = Column(Text)
    purpose_of_registration = Column(Text)
    initial_registration_date = Column(DateTime)
    ...

如果我尝试read_csv，

sam_name_type_dict = {c.name: c.type for c in SamFoo.__table__.c}
sam_name_type_dict.pop('id', None) # id isn't in csv data.
raw_data = pd.read_csv(
        data,
        sep='|',
        skiprows=1,
        header=None,
        names=list(sam_name_type_dict.keys()),
        dtype=sam_name_type_dict,
    )

我得到TypeError: data type not understood，所以我的问题是，有没有办法将sqlachemy数据类型映射到pandas dtypes？

Answer 1

以下是我解决这个问题的方法：

from sqlalchemy.types import DateTime, Integer, Boolean, Text
import numpy as np
types_map = {
    Text: object,
    DateTime: np.datetime64,
    Integer: np.float64, # NA integer gotcha https://pandas.pydata.org/pandas-docs/stable/gotchas.html#support-for-integer-na
    Boolean: bool,
}

# had to split dates and regular dtypes into separate dicts
sam_sql_types = {c.name: c.type for c in SamFoo.__table__.c}
sam_sql_types.pop('id', None) # no id in csv
sam_dtypes = {k: types_map[type(v)] for k, v in sam_sql_types.items()}
sam_dates = {k: v for k, v in sam_dtypes.items() if v == np.datetime64}
sam_dtypes_non_dates = {k: v for k, v in sam_dtypes.items() if v != np.datetime64}

raw_data = pd.read_csv(
    data,
    sep='|',
    skiprows=1,
    header=None,
    names=list(sam_sql_types.keys()),
    dtype=sam_dtypes_non_dates,
    parse_dates=list(sam_dates.keys())
)

pandas read_csv有没有办法将sqlachemy数据类型转换为pandas dtypes？

1 个答案: