熊猫无法写入Postgres db引发“ KeyError ：（“ SELECT name from sqlite_master ...”

Question

我创建了一个程序包，允许用户将数据写入sqlite或Postgres db。我创建了一个用于连接数据库的模块和一个提供写入功能的单独模块。在后一个模块中，write是一个简单的熊猫内部函数调用：

category

写入sqlite数据库（使用'sqlite3'连接）成功，但是写入Postgres数据库时，出现以下错误：

@Document
public class Type {
    @Id
    private String id;
    @DBRef
    private Job job;
    @DBRef
    private List<Category> categories;
}

public class Job {
    @Id
    private String id;
    private String code;
}

public class Category {
    @Id
    private String id;
    private String code;
}

public interface TypeRepository extends MongoRepository<Type, String> {
    @Query("{ 'job.code': ?0, 'category.code': { $in: ?1 }}")
    Type findByJobAndCategoriesCode(String codeJob, List<String> codeCategories);
}

我将错误跟踪到以下文件：

indata.to_sql('pay_' + table, con, if_exists='append', index=False)

似乎正在发生的事情是，'。to_sql'函数已配置为试图在此时写入'sql.py'文件中名为'sqlite_master'的数据库：

Traceback (most recent call last):
  File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 1778, in execute
    ps = cache['ps'][key]
KeyError: ("SELECT name FROM sqlite_master WHERE type='table' AND name=?;", ((705, 0, <function Connection.__init__.<locals>.text_out at 0x7fc3205fb510>),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pandas/io/sql.py", line 1595, in execute
    cur.execute(*args)
  File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 861, in execute
    self._c.execute(self, operation, args)
  File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 1837, in execute
    self.handle_messages(cursor)
  File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 1976, in handle_messages
    raise self.error
pg8000.core.ProgrammingError: {'S': 'ERROR', 'V': 'ERROR', 'C': '42P01', 'M': 'relation "sqlite_master" does not exist', 'P': '18', 'F': 'parse_relation.c', 'L': '1180', 'R': 'parserOpenTable'}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pandas/io/sql.py", line 1610, in execute
    raise_with_traceback(ex)
  File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pandas/compat/__init__.py", line 46, in raise_with_traceback
    raise exc.with_traceback(traceback)
  File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pandas/io/sql.py", line 1595, in execute
    cur.execute(*args)
  File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 861, in execute
    self._c.execute(self, operation, args)
  File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 1837, in execute
    self.handle_messages(cursor)
  File "/anaconda3/envs/PCAN_v1/lib/python3.7/site-packages/pg8000/core.py", line 1976, in handle_messages
    raise self.error
pandas.io.sql.DatabaseError: Execution failed on sql 'SELECT name FROM sqlite_master WHERE type='table' AND name=?;': {'S': 'ERROR', 'V': 'ERROR', 'C': '42P01', 'M': 'relation "sqlite_master" does not exist', 'P': '18', 'F': 'parse_relation.c', 'L': '1180', 'R': 'parserOpenTable'}

更仔细地查看错误，您可以看到与数据库的连接正确，但是pandas正在寻找sqlite数据库：

我知道数据库名称是我半年前刚开始使用sqlite时使用的数据库名称，因此我认为我在某个地方设置了配置值。所以：

1. 我的推理正确吗？
2. 如果是，该如何更改配置？
3. 如果没有，那可能是怎么回事？

Answer 1

每个pandas.DataFrame.to_sql文档：

  

con ：sqlalchemy.engine.Engine或sqlite3.Connection

     

使用SQLAlchemy可以使用该数据库支持的任何数据库   图书馆。为sqlite3.Connection对象提供了旧版支持。

这意味着只有SQLite允许to_sql方法的原始连接。包括Postgres在内的所有其他RDBM必须对此方法使用SQLAlchemy连接，以创建结构和追加数据。请注意：read_sql不需要SQLAlchemy，因为它不会进行持久更改。

因此，此原始DB-API连接无法工作：

import psycopg2
con = psycopg2.connect(host="localhost", port=5432, dbname="mydb", user="myuser", password="mypwd")

indata.to_sql('pay_' + table, con, if_exists='append', index=False)

但是，此SQLAlchemy连接可以正常工作：

from sqlalchemy import create_engine    

engine = create_engine('postgresql+psycopg2://myuser:mypwd@localhost:5432/mydb')

indata.to_sql('pay_' + table, engine, if_exists='append', index=False)

最好对两个数据库都使用SQLAlchemy，这里对SQLite：

engine = create_engine("sqlite:///path/to/mydb.db")