Question

我有一个scrapy蜘蛛解析了这个link

我的蜘蛛看起来如下：

from scrapy.spider import BaseSpider
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.http import request
from scrapy.selector import HtmlXPathSelector
from medsynergies.items import MedsynergiesItem

class methodistspider(BaseSpider):

    name="samplemedsynergies"
    allowed_domains=['msi-openhire.silkroad.com/epostings/']
    start_urls=['https://msi-openhire.silkroad.com/epostings/index.cfm?fuseaction=app.jobinfo&jobid=1284&source=ONLINE&JobOwner=992700&company_id=16616&version=1&byBusinessUnit=NULL&bycountry=0&bystate=0&byRegion=&bylocation=NULL&keywords=&byCat=NULL&proximityCountry=&postalCode=&radiusDistance=&isKilometers=&tosearch=yes']

    #rules=(
    #Rule(SgmlLinkExtractor(allow=("epostings/index.cfm?fuseaction=app%2Ejobsearch&company_id",))),
    #Rule(SgmlLinkExtractor(allow=("epostings/index.cfm?fuseaction=app.jobinfo&jobid",)),callback="parse_job",follow=True),
    #)

    def parse(self, response):
        hxs=HtmlXPathSelector(response)
    titles=hxs.select('//*[@id="jobDesciptionDiv"]')
    items = []

    for titles in titles:
        item=MedsynergiesItem()
        item['job_id']=response.url
        item['title']=titles.select('//*[@id="jobTitleDiv"]/text()').extract()
        item['tracking_code']=titles.select('//*[@id="trackCodeDiv"]/text()').extract()
        item['job_description']=titles.select('.//p/text()').extract()
        item['responsibilities']=titles.select('.//ul/li/text()').extract()
        item['required_skills']=titles.select('//*[@id="jobRequiredSkillsDiv"]/ul/text()').extract()
        item['job_location']=titles.select('//*[@id="jobPositionLocationDiv"]/text()').extract()
        item['position_type']=titles.select('//*[@id="translatedJobPostingTypeDiv"]/text()').extract()
        items.append(item)
    print items
    return items

我得到的输出如下所示：

> [{'job_description': [u'The Operations Solution Architect creates the
> technical vision for Revenue Cycle Management delivery capabilities,
> ensuring that interdependent applications and infrastructures are
> aligned. The SA effectively translates business needs into supportable
> solutions that deliver an excellent customer experience.',
>                      u'Responsibilities:'],  'job_id': 'https://msi-openhire.silkroad.com/epostings/index.cfm?fuseaction=app.jobinfo&jobid=1284&source=ONLINE&JobOwner=992700&company_id=16616&version=1&byBusinessUnit=NULL&bycountry=0&bystate=0&byRegion=&bylocation=NULL&keywords=&byCat=NULL&proximityCountry=&postalCode=&radiusDistance=&isKilometers=&tosearch=yes',
> 'job_location': [u'\r\n\t\t\t\t\t\tIrving, Texas, United
> States\r\n\t\t\t\t\t'],  'position_type':
> [u'\r\n\t\t\t\t\t\tFull-Time/Regular\r\n\t\t\t\t\t'], 
> 'required_skills': [u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n',
>                      u'\r\n'],  'responsibilities': [u'Utilizes technical expertise to create strategic technical vision and
> architecting solutions for Revenue Cycle Manage delivery
> capabilities.',
>                       u'Responsible for gathering requirements, architecting the overall design, and executing the design and build
> phases to ensure RCM solutions and related infrastructures are
> effectively aligned.',
>                       u'Defines key milestones and deliverables related to new developments in collaboration with senior management
> and stakeholders.',
>                       u'Collaborates with Solutions Design, ITS and Operations Implementation team to define, design, price and execute
> new service requirements, new customer accounts, and expanded scope of
> services.',
>                       u'Develops portfolio strategic plan to ensure alignment with Industry trends and market needs to retaining
> MedSynergies industry leadership status.',
>                       u'Provides analysis, opportunity assessments and recommendations to optimize and profitably grow portfolio in alignment
> with established business strategy and goals.\xa0',
>                       u'Performs risk evaluations to ensure that business strategies and evaluations are implemented with clarity and
> consistency.',
>                       u'Serves as senior subject matter expert on content, processes, and procedures for applicable portfolio
> offerings.',
>                       u'Tracks project milestones and deliverables. Develops and delivers progress reports presentations to stake holders
> and senior management',
>                       u'Assists with the transfer of knowledge of technical skills. Provides coaching to less experienced employees.',
>                       u'Participates in special projects and/or completes other duties as assigned.'],  'title':
> [u'\r\n\t\t\t\t\tSolutions Architect\r\n\t\t\t\t'],  'tracking_code':
> [u'\r\n\t\t\t\t\t\tTracking Code\r\n\t\t\t\t\t']}]

所以我的问题是：我想知道是否有更好的方法来定义我的xpath，这样我就不会在输出中得到换行符（\ n）和制表符（\ t）。此外，required_skills字段无法从字段中删除任何文本。我想知道我哪里有错误。

提前谢谢！

Answer 1

如果您知道可以从XPath表达式中获得1个输出字符串值，则可以将XPath包装在normalize-space()中。此外，在for title in titles循环中，您应该使用 relative XPath表达式（从.//开始，而不是以//开头的绝对XPath表达式）

例如：

item['tracking_code']=titles.select('normalize-space(.//*[@id="trackCodeDiv"]/text())').extract()

对于required_skills，我建议您尝试normalize-space(.//*[@id="jobRequiredSkillsDiv"]/ul)：

item['required_skills']=titles.select('normalize-space(.//*[@id="jobRequiredSkillsDiv"]/ul)').extract()

Answer 2

你可以用python清理它：

def clean(item):
    data = {}
    for k, v in item.iteritems():
        data[k] = ' '.join([val.strip() for val in v]).strip()
    return data

Scrapy奇怪的输出

2 个答案: