Question

我想保存已删除的链接以保存在我的数据库中，以便在后续运行时，它只会抓取并仅附加到列表的新链接。

这是我的代码，但在一天结束时我的数据库是空的。我可以做些什么改变来克服这个？提前致谢

from django.template.loader import get_template
from django.shortcuts import render_to_response 
from bs4 import BeautifulSoup
import urllib2, sys
import urlparse
import re
from listing.models import jobLinks

#this function extract the links
def businessghana():
    site = "http://www.businessghana.com/portal/jobs"
    hdr = {'User-Agent' : 'Mozilla/5.0'}
    req = urllib2.Request(site, headers=hdr)
    jobpass = urllib2.urlopen(req)
    soup = BeautifulSoup(jobpass)
    for tag in soup.find_all('a', href = True):
        tag['href'] = urlparse.urljoin('http://www.businessghana.com/portal/', tag['href'])
    return map(str, soup.find_all('a', href = re.compile('.getJobInfo')))

# result from businssghana() saved to a variable to make them iterable as a list

all_links  = businessghana()

#this function should be saving the links to the database unless the link already exist
def save_new_links(all_links):
    current_links = jobLinks.objects.all()
    for i in all_links:
        if i not in current_links:
            jobLinks.objects.create(url=i)

# I called the above function here hoping that it will save to database
save_new_links(all_links)

# return my httpResponse with this function
def display_links(request):
    name = all_links()     
    return render_to_response('jobs.html', {'name' : name})

我的django models.py看起来像这样：

from django.db import models

class jobLinks(models.Model):
    links = models.URLField()
    pub_date = models.DateTimeField('date retrieved')

    def __unicode__(self):
        return self.links

Answer 1

您的代码存在一些问题。首先，jobLinks模型没有url字段。您应该在links=i语句中使用create()。其次，您正在检查字符串（url）是否在QuerySet中。这将永远不会是这种情况，因为QuerySet的元素是模型类的对象，在您的情况下是jobLinks个对象。相反，你可以这样做：

def save_new_links(all_links):
    new_links = []
    for i in all_links:
        if not jobLinks.objects.filter(links=i):
            new_links.append(jobLinks(links=i))
    jobLinks.objects.bulk_create(new_links)

我还建议您将来使用单数字作为模型。例如，在这种情况下，我会调用模型jobLink和字段link。

Python / Django将抓取的数据自动保存到我的数据库中

1 个答案: