Question

我正在尝试检索维基百科文章的第一段文本，在此示例中为 UNIX ，但它返回了一个不需要的输出。

对于我在维基百科api上阅读的内容以及StackOverflow上的内容，这是拨打电话的请求网址：

http://en.wikipedia.org/w/api.php?format=php&action=query&titles=unix&redirects=1&prop=revisions&rvprop=content&rvsection=0&rvlimit=1

我的预期输出将是：

Unix（正式注册为UNIX，有时也写为Unix 小型大写）是一种多任务，多用户计算机操作系统最初由贝尔的一组AT＆amp; T员工于1969年开发实验室，包括Ken Thompson，Dennis Ritchie，Brian Kernighan，Douglas McIlroy，Michael Lesk和Joe Ossanna。[1] Unix操作系统最初是用汇编语言开发的，但到了1973年已经差不多了完全用C语言记录，极大地促进了它的进一步发展并移植到其他硬件。今天的Unix系统演变是分裂的进入各个分支机构，随着时间的推移由AT＆amp; T以及各种分支机构开发商业供应商，大学（如加州大学，伯克利的BSD）和非营利组织。

我目前的结果：

{{Use dmy dates|date=August 2012}}
{{Infobox OS
|name               = Unix
|logo               = 
|screenshot         = [[File:Unix history-simple.svg|250px]]
|caption            = Evolution of Unix and Unix-like systems
|website            = [http://www.unix.org unix.org]
|developer          = [[Ken Thompson (computer programmer)|Ken Thompson]], [[Dennis Ritchie]], [[Brian Kernighan]], [[Douglas McIlroy]], and [[Joe Ossanna]] at [[Bell Labs]]
|source_model       = Historically [[Closed source software|closed source]], now some Unix projects ([[Berkeley Software Distribution|BSD]] family and [[Illumos]]) are [[open source]]d.
|frequently_updated = yes <!-- Release version update? Don't edit this page, just click on the version number! -->
|programmed_in      = [[C (programming language)|C]] 
|kernel_type        = [[Monolithic Kernel|Monolithic]]
|ui                 = [[Command-line interface]] & [[Graphical user interface|Graphical]] ([[X Window System]])
|language           = English 
|family             = Unix
|released           = {{start date and age|df=yes|1969}}
|license            = [[Proprietary software|Proprietary]]
|working_state      = Current 
}}

'''Unix''' (officially trademarked as '''UNIX''', sometimes also written as '''<span style="font-variant: small-caps;">Unix</span>''' in small caps) is a [[Computer multitasking|multitasking]], [[multi-user]] computer [[operating system]] originally developed in 1969 by a group of [[American Telephone & Telegraph|AT&T]] employees at [[Bell Labs]], including [[Ken Thompson]], [[Dennis Ritchie]], [[Brian Kernighan]], [[Douglas McIlroy]], [[Michael Lesk]] and [[Joe Ossanna]].<ref name=" Ritchie">{{cite journal
  | last = Ritchie
  | first = D.M.
  | authorlink = 
  | coauthors = Thompson, K.
  | title = The UNIX Time-Sharing System
  | journal = Bell System Tech. J.
  | volume = 57
  | issue = 6
  | pages = 1905-1929
  | publisher = American Tel. & Tel.
  | location = USA
  | date = July 1978
  | url = http://www.alcatel-lucent.com/bstj/vol57-1978/articles/bstj57-6-1905.pdf
  | issn = 
  | doi = 
  | id = 
  | accessdate = December 9, 2012}}</ref>  The Unix operating system was first developed in [[assembly language]], but by 1973 had been almost entirely recoded in [[C (programming language)|C]], greatly facilitating its further development and [[Software portability|porting]] to other hardware. Today's Unix system evolution is split into various branches, developed over time by AT&T as well as various commercial vendors, universities (such as [[University of California, Berkeley]]'s [[BSD]]), and [[non-profit]] organizations.

[[The Open Group]], an industry standards consortium, owns the UNIX trademark. Only systems fully compliant with and certified according to the [[Single UNIX Specification]] are qualified to use the trademark; others might be called ''Unix system-like'' or ''[[Unix-like]]'', although the Open Group disapproves<ref>[http://www.unix.org/questions_answers/faq.html#7a  What is a "Unix-like" operating system?] Unix.org FAQ</ref> of this term.  However, the term ''Unix'' is often used informally to denote any operating system that closely resembles the trademarked system.

During the late 1970s and early 1980s, the influence of Unix in academic circles led to large-scale adoption of Unix (particularly of the [[Berkeley Software Distribution|BSD]] variant, originating from the [[University of California, Berkeley]]) by commercial startups, the most notable of which are [[Solaris (operating system)|Solaris]], [[HP-UX]], [[Sequent Computer Systems|Sequent]], and [[AIX operating system|AIX]], as well as [[Darwin (operating system)|Darwin]], which forms the core set of components upon which [[Apple Inc.|Apple]]'s [[OS X]], [[Apple TV]], and [[IOS (Apple)|iOS]] are based.<ref>{{cite web|url=http://marketshare.hitslink.com/operating-system-market-share.aspx?qprid=8&qpcustomd=0 |title=Operating system market share |publisher=Marketshare.hitslink.com |date= |accessdate=2012-08-22}}</ref><ref>{{cite web|url=http://developer.apple.com/library/mac/#documentation/MacOSX/Conceptual/OSX_Technology_Overview/SystemTechnology/SystemTechnology.html#//apple_ref/doc/uid/TP40001067-CH207-BCICAIFJ |title=Loading |publisher=Developer.apple.com |date= |accessdate=2012-08-22}}</ref> Today, in addition to certified Unix systems such as those already mentioned, [[Unix-like]] operating systems such as [[MINIX]], [[Linux]], and [[BSD]] descendants ([[FreeBSD]], [[NetBSD]], [[OpenBSD]], and [[DragonFly BSD]]) are commonly encountered. The term ''traditional Unix'' may be used to describe an operating system that has the characteristics of either [[Version 7 Unix]] or [[UNIX System V]]."

检索文章的正确方法是什么？

提前致谢！

Answer 1

如果您只想要纯文字，请使用TextExtracts：http://en.wikipedia.org/w/api.php?format=json&action=query&prop=extracts&explaintext=1&titles=Unix

这会产生：Unix is a multitasking, multi-user computer operating system that exists in many variants. The original Unix was developed at AT&T's Bell Labs research center by Ken Thompson, Dennis Ritchie, and others. From the power user's or programmer's perspective, Unix systems are characterized by a modular design that is sometimes called the "Unix philosophy," meaning the OS provides a set of simple tools that each perform a limited, well-defined function, with a unified filesystem as the main means of communication and a shell scripting and command language to combine the tools to perform complex workflows.

Answer 2

我一直用Python处理这个问题。第一项任务是获得所需的文字;之后，您需要解析HTML并删除所有无关的信息。

以下函数将为您提供第n个文本部分（n = 0返回摘要）：

import requests

def getWikiSection(topic, n):
    url = 'http://en.wikipedia.org/w/api.php?action=parse&page=%s&format=json&prop=text&section=%s' % (topic, str(n))
    json_response = requests.get(url).json().items()
    if len(json_response) > 1 and json_response[1][0] == u'error':
        print json_response[1][1][u'info']
        return None
    return stripTags(json_response[0][1][u'text'][u'*'])

快速演练：首先，我们为给定主题创建URL;然后，我们抓住JSON响应;如果我们查询了无效的部分或主题（即，该主题没有页面，或者我们超出了页面的长度），我们会打印错误;否则，我们会清理响应。

清理响应由最后一行的'stripTags'函数处理，删除HTML标记。这是：

from HTMLParser import HTMLParser

class MLStripper(HTMLParser):
    def __init__(self):
        self.reset()
        self.fed = []

    def handle_data(self, d):
        self.fed.append(d)

    def get_data(self):
        return ''.join(self.fed)

def stripTags(html):
    s = MLStripper()
    s.feed(html)
    return s.get_data()

当然，这可以扩展为解析您喜欢的文本。例如，我删除了引用如下：

import re

def removeReferences(s):
    return re.sub(r'\[[0-9]+\]', '', s)

希望这有帮助。

Answer 3

我会查看MobileFrontend Extension，它至少会为您提供一些<p>。使用http://en.wikipedia.org/w/api.php?format=json&action=mobileview&page=Unix&sections=0&prop=text|sections给出了

{“mobileview”：{“sections”：[{“id”：0，“text”：“\ n \ nUnix \ n \ n \ n \ n \ n Unix和类Unix系统的演变\ n \ n \ n \ n \ n \ n \ n n \ n公司/开发人员\ nKen Thompson，Dennis Ritchie，Brian Kernighan，Douglas McIlroy和来自贝尔实验室的Joe Ossanna \ n \ n \ n编程为\ nC和汇编语言\ n \ n \ n OS系列\ nUnix \ n \ n \ n \ n \ n \ n n工作状态\ n当前\ n \ n \ n源模型\ n历史上闭源，现在一些Unix项目（BSD系列和Illumos）是开源的。\ n \ n \ n初始版本\ n 1969年2月20日; 44年前（4月20日， 1969）\ n \ n \ n可用的语言\ n英语\ n \ n \ n内核类型\ nMonolithic \ n \ n \ n默认用户界面\ n命令行界面＆amp;图形（X Window系统）\ n \ n \ n许可证\ n专有\ n \ n \ n官方网站\ nhttp：//www.unix.org \“＆gt; unix.org \ n \ n \ n

Unix （正式注册商标） as UNIX ，有时也写成 Unix 小写字母）是一个多任务，多用户计算机操作系统，最初由一组开发于1969年。

（剪断）

你必须接受它并以其他方式解析它（perl，bash，无论如何），但此时你也可以抛弃API并进行一些curl或wget动作，这会让事情变得简单。

Answer 4

尝试this url。

（注意：pagename是您要求的页面）

但这会返回一堆html垃圾...你可以通过这样做来过滤hml：

$.getJSON("http://en.wikipedia.org/w/api.php?"+"action=parse&format=json&prop=text&section=all&page=" + entry + "&redirects&callback=?", function(data)
    {   

        if (!data.error)
        {
            var markup = data.parse.text["*"];
            if (typeof markup !== "undefined")
            {
                $("#entry").text(entry).show();
                var blurb = $('<div id="articleText"></div>').html(markup);

                // remove links as they will not work
                blurb.find('a').each(function() { $(this).replaceWith($(this).html()); });

                // remove any references
                blurb.find('sup').remove();

                // remove cite error
                blurb.find('.mw-ext-cite-error').remove();
                $('#article').html($(blurb).find('p'));

                $("#article").append(link);
                // console.log(markup);
            }
        }
    });

您可以阅读more here

获取维基百科文章的第一段（并且仅文本）返回不期望的结果

4 个答案: