如何将.csv文件中的数据传输到django中的sqlite数据库?

时间:2009-12-10 17:10:35

标签: python django csv

这是我的models.py

from django.db import models

class School(models.Model):    
    school = models.CharField(max_length=300)
    def __unicode__(self):
        return self.school
class Lawyer(models.Model):
    firm_url = models.URLField('Bio', max_length=200)
    firm_name = models.CharField('Firm', max_length=100)
    first = models.CharField('First Name', max_length=50)
    last = models.CharField('Last Name', max_length=50)
    year_graduated = models.IntegerField('Year graduated')
    school = models.CharField(max_length=300)
    school = models.ForeignKey(School)
    class Meta:
        ordering = ('?',)
   def __unicode__(self):
        return self.first    

来自csv文件的2个示例行:

"http://www.graychase.com/aabbas,Gray & Chase LLP, Amr A ,Abbas,The George Washington University Law School, 2005"
"http://www.graychase.com/kadam,Gray & Chase LLP, Karin ,Adam,Ernst Moritz Arndt University Greifswald, 2004"

谢谢。

编辑

您能否提供有关此脚本的更多详细信息?一些问题:

我的应用程序在

C:.../Documents/PROJECTS/Django/sw2/wkw2.

这条路已经在PYTHONPATH我还需要这条线吗?如果是这样,我会这样输入吗?

sys.path.append('C:\\sw2')

os.environ做了什么?我试着阅读文档,但我不明白。

os.environ['DJANGO_SETTINGS_MODULE'] = 'sw2.settings'

from django.core.management import setup_environ
from sw2 import settings
from sw2.wkw2.models import *

import csv

dataReader = csv.reader(open('csvtest1.csv'), delimiter=',', quotechar='"')

for row in dataReader:
    lawyer=Lawyer()
    lawyer.firm_url=row[0]
    lawyer.firm_name=row[1]
    lawyer.first=row[2]
    lawyer.last=row[3]
    lawyer.school=row[4]
    lawyer.year_graduated=row[5]
    lawyer.save()

谢谢!

编辑以回应大提琴的回答:

celopes:

我看到你的回答有点晚了。我一直在尝试使用

更新shell中的数据库
>>> p1 = Lawyer(school = "The George Washington University Law School", last = "Babbas", firm_url = "http://www.graychase.com/babbas", year_graduated = 2005, firm_name=  "Gray & Chase", first= "Amr A")

但我一直收到整数错误。最后,我意识到学校必须是school_id,即形式

>>> p1 = Lawyer(school_id = 1, last = "Babbas", firm_url = "http://www.graychase.com/babbas", year_graduated = 2005, firm_name=  "Gray & Chase", first= "Amr A")

由此我意识到,我需要知道每所学校的学校ID以更新律师表。由于那是不可能的,我决定删除ForeignKey,因为我不知道如何解决这个问题。 (我是Python和Django的新手。)

今天早上我看到了你的回答。现在我在开发服务器中更改了我的模型,我只有一个表:律师。我想这就是我将要使用的。我再次道歉,因为没有早点看到你的答案。

编辑12/14/09:

celopes:

再次感谢此脚本。它解决了我的问题。在保存到数据库之前,我不需要将csv转换为json或其他格式。我做了一些改变。首先如前所述,我将模型改为律师。此外,您使用“名称”修复了重复的学校。但我在list_display中有list_filtersearch_fieldsadmin.py,并且更改字段名称会导致错误太多。

class LawyerAdmin(admin.ModelAdmin):
    fieldsets = [
        ('Name',   {'fields': ['last', 'first', 'firm_name', 'firm_url', 'school', 'year_graduated']}),
        #('School', {'fields': ['school', 'year_graduated']}),
]
    list_display = ('last', 'first', 'firm_name', 'firm_url', 'school', 'year_graduated')
    list_filter = ['school', 'year_graduated']
    search_fields = ['last', 'school', 'firm_name']
    #search_fields = ['school__school']
    #search_fields = ['school__lawyer__last']

使用新的models.py,简化的脚本运行良好。这就是我正在使用的:

csv_filepathname="C:/Users/A/Documents/Projects/Django/sw2/wkw2/fixtures/data1.csv"
your_djangoproject_home="C:/Users/A/Documents/PROJECTS/Django/"

import sys,os
sys.path.append(your_djangoproject_home)
os.environ['DJANGO_SETTINGS_MODULE'] ='sw2.settings'

from sw2.wkw2.models import Lawyer

import csv
dataReader = csv.reader(open(csv_filepathname), delimiter=',', quotechar='"')

for row in dataReader:
    lawyer=Lawyer()
    lawyer.firm_url=row[0]
    lawyer.firm_name=row[1]
    lawyer.first=row[2]
    lawyer.last=row[3]
    lawyer.school=row[4]
    lawyer.year_graduated=row[5]
    lawyer.save()

我也删除了每行的引号。我注意到,如果我把年份放在引号中,我得到整数错误,没有引号工作正常。你是如何使用引号的?

再次感谢,这非常有帮助。现在我必须让它在生产服务器中运行。

1 个答案:

答案 0 :(得分:6)

我使用这些数据作为测试创建了一个完整的脚本:

"http://www.graychase.com/aabbas","Gray & Chase LLP","Amr A","Abbas","The George Washington University Law School","2005"
"http://www.graychase.com/kadam","Gray & Chase LLP","Karin","Adam","Ernst Moritz Arndt University Greifswald","2004"

请注意上面的 错误 中的CSV文件。 csv文件阅读器将整行读取为条目,因为整行都在引号中。从csv文件中的每一行中删除开头和尾随引号,或者 - 像我一样 - 将每个不同的值括在引号中的行中。

以下是适用于以下脚本的模型:

from django.db import models

class School(models.Model):    
    name = models.CharField(max_length=300, unique=True)

    def __unicode__(self):
        return self.name

class Lawyer(models.Model):
    firm_url = models.URLField('Bio', max_length=200, unique=True)
    firm_name = models.CharField('Firm', max_length=100)
    first = models.CharField('First Name', max_length=50)
    last = models.CharField('Last Name', max_length=50)
    year_graduated = models.IntegerField('Year graduated')
    school = models.ForeignKey(School)

    def __unicode__(self):
        return self.first

以下是将读取您的CSV文件的脚本(除非我将您的项目名称sw2和应用程序wkw2错误,然后修复这些引用):

############ All you need to modify is below ############
# Full path and name to your csv file
csv_filepathname="C:/Users/A/Documents/Projects/Django/sw2/wkw2/fixtures/data.csv"
# Full path to the directory immediately above your django project directory
your_djangoproject_home="C:.../Documents/PROJECTS/Django/"
############ All you need to modify is above ############

import sys,os
sys.path.append(your_djangoproject_home)
os.environ['DJANGO_SETTINGS_MODULE'] ='sw2.settings'

from sw2.wkw2.models import School, Lawyer

import csv
dataReader = csv.reader(open(csv_filepathname), delimiter=',', quotechar='"')

old_school = None
for row in dataReader:
    if old_school != row[4]:
        old_school = row[4]
        school = School()
        school.name = old_school
        school.save()

dataReader = csv.reader(open(csv_filepathname), delimiter=',', quotechar='"')

for row in dataReader:
    lawyer=Lawyer()
    lawyer.firm_url=row[0]
    lawyer.firm_name=row[1]
    lawyer.first=row[2]
    lawyer.last=row[3]

    lawyer_school=School.objects.get(name=row[4])
    lawyer.school=lawyer_school

    lawyer.year_graduated=row[5]
    lawyer.save()

该脚本首先从CSV文件中的可用学校创建每个可能的学校。然后它再次运行CSV并创建每一位律师。

我用测试数据运行了这个脚本。它工作正常并加载所有CSV数据。