尝试将Pandas数据框导入sqlite数据库时遇到锁定错误

时间：2019-05-29 04:03:15

标签： python-3.x pandas sqlite apsw

当尝试使用pandas.Dataframe.to_sql函数将pandas.Dataframe实例导入python中的SQLite数据库时，遇到数据库锁定错误。

我首先使用sqlite3（python 3.7.3）建立与数据库文件的连接。

在使用sqlite3建立与数据库文件的连接时，我将超时设置为10s和30s。但是错误仍然会发生，并且似乎在等待我为timeout参数指定的时间后立即发生了错误。

然后我读了一个问题here。它告诉我设置sql.isolation_level = None。我做到了，错误仍然发生。

然后，我使用apsw建立与数据库文件的连接。并设置connection.setbusytimeout（10000）（10s），但仍然发生错误。

重要的一点是，当尝试使用pandas.Dataframe.to_sql将数据帧导入数据库时​​。表名已经存在，我将if_exist参数设置为'replace'。

错误消息指示从pandas.Dataframe.to_sql执行DROP命令时何时引发锁定错误。

以上过程全部在PyCharm IDE（专业版）的平台上进行。我使用PyCharm运行.py磁盘文件。所有错误消息均由PyCharm提供。我用只有20多行记录的小型tsv文件测试了这些代码。

我还检查了文件夹，并且数据库文件的同一目录下没有-journal文件。另外，在测试代码之前，我已授予数据库文件完全权限。

这是我用来与sqlite3建立连接的代码。

```python
import sys
import os.path
import pandas as pd
import sqlite3

def compare_causal(c, d, e):
    #Read a tsv file into pandas.dataframe
    os.chdir(c)
    patient_df = pd.read_csv(d, sep='\t')
    patient_table = d[0:-4]
    patient_db_path = os.path.join(e, "PID_patients_data.db")

    #Build a connection
    pat_connect = sqlite3.connect(patient_db_path, timeout=10.0)
    pat_connect.isolation_level = None
    #Import the pandas.dataframe to sql database
    patient_df.to_sql(name=patient_table, con=pat_connect, if_exists='replace')

# There are other subsequent codes but they are all in the definition of the function compare_causal.
```

这是我用来与apsw建立连接的代码。 

```python
import sys
import os.path
import pandas as pd
import apsw

def compare_causal(c, d, e):
    #Read a tsv file into pandas.dataframe
    os.chdir(c)
    patient_df = pd.read_csv(d, sep='\t')
    patient_table = d[0:-4]
    patient_db_path = os.path.join(e, "PID_patients_data.db")

    #Build a connection
    pat_connect = apsw.Connection(patient_db_path)
    pat_connect.setbusytimeout(10000)
    #Import the pandas.dataframe to sql database
    patient_df.to_sql(name=patient_table, con=pat_connect, if_exists='replace')

# There are other subsequent codes but they are all in the definition of the function compare_causal.
```

这是我从sqlite连接获得的错误消息： 

Traceback (most recent call last):
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1431, in execute
    cur.execute(*args)
sqlite3.OperationalError: database is locked

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/skyxt/Google Drive/HKU_YANG_LAB/HKU_Academic_Issue/Python Learning/Python scripts/Database_related scripts/Compare_PID_causal_genes_sqlite.py", line 207, in <module>
    "/Users/skyxt/Google Drive/HKU_YANG_LAB/HKU_Academic_Issue/Project PID WES/Patients_Database")
  File "/Users/skyxt/Google Drive/HKU_YANG_LAB/HKU_Academic_Issue/Python Learning/Python scripts/Database_related scripts/Compare_PID_causal_genes_sqlite.py", line 37, in compare_causal
    patient_df.to_sql(name=patient_table, con=patient_connect, if_exists='replace')
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 2531, in to_sql
    dtype=dtype, method=method)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 460, in to_sql
    chunksize=chunksize, dtype=dtype, method=method)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1546, in to_sql
    table.create()
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 577, in create
    self.pd_sql.drop_table(self.name, self.schema)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1566, in drop_table
    self.execute(drop_sql)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1445, in execute
    raise_with_traceback(ex)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 420, in raise_with_traceback
    raise exc.with_traceback(traceback)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1431, in execute
    cur.execute(*args)
pandas.io.sql.DatabaseError: Execution failed on sql 'DROP TABLE "sorted_A170618_Wang_PathwayCommons11_All_network_search_results"': database is locked

这是我从apsw连接中收到的错误消息： 

Traceback (most recent call last):
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1431, in execute
    cur.execute(*args)
  File "src/cursor.c", line 236, in resetcursor
apsw.BusyError: BusyError: database is locked

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1435, in execute
    self.con.rollback()
AttributeError: 'apsw.Connection' object has no attribute 'rollback'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/skyxt/Google Drive/HKU_YANG_LAB/HKU_Academic_Issue/Python Learning/Python scripts/Database_related scripts/Compare_PID_causal_genes_sqlite.py", line 207, in <module>
    "/Users/skyxt/Google Drive/HKU_YANG_LAB/HKU_Academic_Issue/Project PID WES/Patients_Database")
  File "/Users/skyxt/Google Drive/HKU_YANG_LAB/HKU_Academic_Issue/Python Learning/Python scripts/Database_related scripts/Compare_PID_causal_genes_sqlite.py", line 37, in compare_causal
    patient_df.to_sql(name=patient_table, con=patient_connect, if_exists='replace')
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/core/generic.py", line 2531, in to_sql
    dtype=dtype, method=method)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 460, in to_sql
    chunksize=chunksize, dtype=dtype, method=method)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1546, in to_sql
    table.create()
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 577, in create
    self.pd_sql.drop_table(self.name, self.schema)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1566, in drop_table
    self.execute(drop_sql)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1440, in execute
    raise_with_traceback(ex)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/compat/__init__.py", line 420, in raise_with_traceback
    raise exc.with_traceback(traceback)
  File "/Users/skyxt/anaconda3/lib/python3.7/site-packages/pandas/io/sql.py", line 1435, in execute
    self.con.rollback()
pandas.io.sql.DatabaseError: Execution failed on sql: DROP TABLE "sorted_A170618_Wang_PathwayCommons11_All_network_search_results"
BusyError: database is locked
unable to rollback

0 个答案:

没有答案
相关问题
最新问题