我想知道是否可以“自动化”输入搜索表单的条目和从结果中提取匹配项的任务。例如,我有一份期刊文章列表,我想获得DOI(数字对象标识符);为此手动我将转到期刊文章搜索页面(例如,http://pubs.acs.org/search/advanced),输入作者/标题/卷(等),然后从其返回结果列表中找到该文章,并选择DOI并将其粘贴到我的参考列表中。我经常使用R和Python进行数据分析(我的灵感来自于RCurl的一篇文章),但对网络协议的了解并不多......这是可能的(例如使用类似Python的BeautifulSoup?)。做任何类似于此任务的远程操作都有什么好的参考吗?我对学习网络抓取和网络抓取工具一样兴趣,就像完成这项特殊任务一样......感谢您的时间!
答案 0 :(得分:9)
美丽的汤非常适合解析网页 - 这是您想要做的事情的一半。 Python,Perl和Ruby都有一个版本的Mechanize,那是另一半:
http://wwwsearch.sourceforge.net/mechanize/
机械化让你控制一个浏览器:
# Follow a link
browser.follow_link(link_node)
# Submit a form
browser.select_form(name="search")
browser["authors"] = ["author #1", "author #2"]
browser["volume"] = "any"
search_response = br.submit()
使用Mechanize和Beautiful Soup,你有一个很好的开始。我考虑的另一个工具是Firebug,就像在这个快速红宝石刮擦指南中使用的那样:
http://www.igvita.com/2007/02/04/ruby-screen-scraper-in-60-seconds/
Firebug可以加速构建xpath以解析文档,为您节省大量时间。
祝你好运!答案 1 :(得分:2)
Python代码:用于搜索表单。
# import
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
from selenium.webdriver.support import expected_conditions as EC # available since 2.26.0
# Create a new instance of the Firefox driver
driver = webdriver.Firefox()
# go to the google home page
driver.get("http://www.google.com")
# the page is ajaxy so the title is originally this:
print driver.title
# find the element that's name attribute is q (the google search box)
inputElement = driver.find_element_by_name("q")
# type in the search
inputElement.send_keys("cheese!")
# submit the form (although google automatically searches now without submitting)
inputElement.submit()
try:
# we have to wait for the page to refresh, the last thing that seems to be updated is the title
WebDriverWait(driver, 10).until(EC.title_contains("cheese!"))
# You should see "cheese! - Google Search"
print driver.title
finally:
driver.quit()
答案 2 :(得分:1)
WebRequest req = WebRequest.Create("http://www.URLacceptingPOSTparams.com");
req.Proxy = null;
req.Method = "POST";
req.ContentType = "application/x-www-form-urlencoded";
//
// add POST data
string reqString = "searchtextbox=webclient&searchmode=simple&OtherParam=???";
byte[] reqData = Encoding.UTF8.GetBytes (reqString);
req.ContentLength = reqData.Length;
//
// send request
using (Stream reqStream = req.GetRequestStream())
reqStream.Write (reqData, 0, reqData.Length);
string response;
//
// retrieve response
using (WebResponse res = req.GetResponse())
using (Stream resSteam = res.GetResponseStream())
using (StreamReader sr = new StreamReader (resSteam))
response = sr.ReadToEnd();
// use a regular expression to break apart response
// OR you could load the HTML response page as a DOM
(改编自Joe Albahri的“C#简而言之”)
答案 3 :(得分:0)
网页抓取有很多工具。有一个很好的firefox插件叫做iMacros。它工作得很好,根本不需要编程知识。免费版可以从这里下载: https://addons.mozilla.org/en-US/firefox/addon/imacros-for-firefox/ 关于iMacros的最好的事情是,它可以让你在几分钟内启动,它也可以从bash命令行启动,也可以从bash脚本中调用。
更高级的步骤是selenium webdrive。我选择硒的原因是它以适合初学者的方式记录。阅读以下page:
可以让你立刻投入运行。 Selenium支持java,python,php,c所以如果你熟悉这些语言中的任何一种,你就会熟悉所需的所有命令。我更喜欢硒的webdrive变体,因为它打开浏览器,以便您可以检查字段和输出。使用webdrive设置脚本后,您可以轻松地将脚本迁移到IDE,从而无需运行。
要安装selenium,您可以输入命令
sudo easy_install selenium
这将照顾您所需的依赖项和所有内容。
要以交互方式运行脚本,只需打开终端,然后键入
即可python
你会看到python提示符,>>>你可以输入命令。
以下是您可以在终端中粘贴的示例代码,它将搜索谷歌中的单词奶酪
package org.openqa.selenium.example;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;
import org.openqa.selenium.support.ui.ExpectedCondition;
import org.openqa.selenium.support.ui.WebDriverWait;
public class Selenium2Example {
public static void main(String[] args) {
// Create a new instance of the Firefox driver
// Notice that the remainder of the code relies on the interface,
// not the implementation.
WebDriver driver = new FirefoxDriver();
// And now use this to visit Google
driver.get("http://www.google.com");
// Alternatively the same thing can be done like this
// driver.navigate().to("http://www.google.com");
// Find the text input element by its name
WebElement element = driver.findElement(By.name("q"));
// Enter something to search for
element.sendKeys("Cheese!");
// Now submit the form. WebDriver will find the form for us from the element
element.submit();
// Check the title of the page
System.out.println("Page title is: " + driver.getTitle());
// Google's search is rendered dynamically with JavaScript.
// Wait for the page to load, timeout after 10 seconds
(new WebDriverWait(driver, 10)).until(new ExpectedCondition<Boolean>() {
public Boolean apply(WebDriver d) {
return d.getTitle().toLowerCase().startsWith("cheese!");
}
});
// Should see: "cheese! - Google Search"
System.out.println("Page title is: " + driver.getTitle());
//Close the browser
driver.quit();
}}
我希望这可以为你提供一个良好的开端。
干杯:)