我正在抓取一个具有Oracle ADF回送脚本的网站，该脚本不断将我重定向到我的同一页面，因此如何绕过它？
以下是我的php代码。
<?php
$url = 'https://www.mywebsite.com/faces/index.jspx';
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__) . '/cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__) . '/cookie.txt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$header[] = 'User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36';
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
$data = curl_exec($ch);
curl_close($ch);
if (curl_errno($ch)) { // check for execution errors
echo 'Scraper error: ' . curl_error($ch);
exit;
}
echo $data;
?>
当我运行上述代码时，我被重定向到同一页面，
，它还添加了一些查询字符串参数，例如 ?_afrLoop=39478247795404&_afrWindowMode=0&_afrWindowId=null
在实际站点_afrWindowId中有一些随机的字母数字字符串，但我却得到了null。
手动停止页面重定向后，我得到了具有Oracle回送脚本的页面，如下所示
这会导致重定向，该怎么做对我有帮助。
环回脚本：
<html lang="el-GR"><head><script>
/*
** Copyright (c) 2008, Oracle and/or its affiliates. All rights reserved.
*/
/**
* This is the loopback script to process the url before the real page loads. It introduces
* a separate round trip. During this first roundtrip, we currently do two things:
* - check the url hash portion, this is for the PPR Navigation.
* - do the new window detection
* the above two are both controled by parameters in web.xml
*
* Since it's very lightweight, so the network latency is the only impact.
*
* here are the list of will-pass-in parameters (these will replace the param in this whole
* pattern:
* viewIdLength view Id length (characters),
* loopbackIdParam loopback Id param name,
* loopbackId loopback Id,
* loopbackIdParamMatchExpr loopback Id match expression,
* windowModeIdParam window mode param name,
* windowModeParamMatchExpr window mode match expression,
* clientWindowIdParam client window Id param name,
* clientWindowIdParamMatchExpr client window Id match expression,
* windowId window Id,
* initPageLaunch initPageLaunch,
* enableNewWindowDetect whether we want to enable new window detection
* jsessionId session Id that needs to be appended to the redirect URL
* enablePPRNav whether we want to enable PPR Navigation
*
*/
var id = null;
var query = null;
var href = document.location.href;
var hashIndex = href.indexOf("#");
var hash = null;
/* process the hash part of the url, split the url */
if (hashIndex > 0)
{
hash = href.substring(hashIndex + 1);
/* only analyze hash when pprNav is on (bug 8832771) */
if (false && hash && hash.length > 0)
{
hash = decodeURIComponent(hash);
if (hash.charAt(0) == "@")
{
query = hash.substring(1);
}
else
{
var state = hash.split("@");
id = state[0];
query = state[1];
}
}
href = href.substring(0, hashIndex);
}
/* process the query part */
var queryIndex = href.indexOf("?");
if (queryIndex > 0)
{
/* only when pprNav is on, we take in the query from the hash portion */
query = (query || (id && id.length>0))? query: href.substring(queryIndex);
href = href.substring(0, queryIndex);
}
var jsessionIndex = href.indexOf(';');
if (jsessionIndex > 0)
{
href = href.substring(0, jsessionIndex);
}
/* we will replace the viewId only when pprNav is turned on (bug 8832771) */
if (false)
{
if (id != null && id.length > 0)
{
href = href.substring(0, href.length - 11) + id;
}
}
var isSet = false;
if (query == null || query.length == 0)
{
query = "?";
}
else if (query.indexOf("_afrLoop=") >= 0)
{
isSet = true;
query = query.replace(/_afrLoop=[^&]*/, "_afrLoop=39279593944826");
}
else
{
query += "&";
}
if (!isSet)
{
query = query += "_afrLoop=39279593944826";
}
/* below is the new window detection logic */
var initWindowName = "_afr_init_"; // temporary window name set to a new window
var windowName = window.name;
// if the window name is "_afr_init_", treat it as redirect case of a new window
if ((true) && (!windowName || windowName==initWindowName ||
windowName!="null"))
{
/* append the _afrWindowMode param */
var windowMode;
if (true)
{
/* this is the initial page launch case,
also this could be that we couldn't detect the real windowId from the server side */
windowMode=0;
}
else if ((href.indexOf("/__ADFvDlg__") > 0) || (query.indexOf("__ADFvDlg__") >= 0))
{
/* this is the dialog case */
windowMode=1;
}
else
{
/* this is the ctrl-N case */
windowMode=2;
}
if (query.indexOf("_afrWindowMode=") >= 0)
{
query = query.replace(/_afrWindowMode=[^&]*/, "_afrWindowMode="+windowMode);
}
else
{
query = query += "&_afrWindowMode="+windowMode;
}
/* append the _afrWindowId param */
var clientWindowId;
/* in case we couldn't detect the windowId from the server side */
if (!windowName || windowName == initWindowName)
{
clientWindowId = "null";
// set window name to an initial name so we can figure out whether a page is loaded from
// cache when doing Ctrl+N with IE
window.name = initWindowName;
}
else
{
clientWindowId = windowName;
}
if (query.indexOf("_afrWindowId=") >= 0)
{
query = query.replace(/_afrWindowId=\w*/, "_afrWindowId="+clientWindowId);
}
else
{
query = query += "&_afrWindowId="+clientWindowId;
}
}
var sess = "";
if (sess.length > 0)
href += sess;
/* if pprNav is on, then the hash portion should have already been processed */
if ((false) || (hash == null))
document.location.replace(href + query);
else
document.location.replace(href + query + "#" + hash);
</script>
</head>
</html>
答案 0 :(得分：1)
抓取ADF页面的正确方法是在URL中传递参数
*domain.com*?org.apache.myfaces.trinidad.outputMode=webcrawler
来自脚本的所有GET请求。请记住，当您切换到搜寻器模式时，页面的外观将有所不同，因为它不是供人类使用的，但其中应包含您将要爬行的所有原始详细信息。
尽管这是一个古老的问题，OP可能早就将其转移到更好的事情上，想到在这里回答这个问题以帮助其他人解决相同的问题。