从以下网页中提取PDF网址很简单。
https://www.osapublishing.org/boe/abstract.cfm?uri=boe-11-5-2745
但是当我获取它时，它会在输出中显示类似的内容，而不是下载PDF文件。
<p>OSA has implemented a process that requires you to enter the letters and/or numbers below before you can download this article.</p>
由于网站使用Cookie cfid，因此应使用ColdFusion保护它。有人知道如何抓取这样的网页吗？谢谢。
https://cookiepedia.co.uk/cookies/CFID
编辑：Sev Roberts提供的wget解决方案不起作用。我检查了chrome devtools（在新的隐身窗口中），在发送https://www.osapublishing.org/boe/abstract.cfm?uri=boe-11-5-2745的第一个请求之后，发送了许多请求。我想这是因为wget不会发送这些请求，因此https://www.osapublishing.org/boe/viewmedia.cfm?uri=boe-11-5-2745&seq=0的后续wget（带有cookie）将无法工作。谁能说出这些提取请求中的哪些是必不可少的？谢谢。
答案 0 :(得分：1)
网站针对这种类型的抓取和直接链接或嵌入有几种方法。基本的旧方法包括：
cgi.http_referer变量以查看用户是否来自预期的来源。cgi.http_user_agent是否看起来像已知的人类浏览器-或检查用户代理看起来不像已知的机器人浏览器。当然还存在其他更智能的方法，但是根据我的经验，如果您需要的不只是以上所述，那么您将达到要求验证码和/或要求用户注册并登录的领域。
显然，通过手动设置标题可以很容易地欺骗（2）和（3）。对于（1），如果您正在使用cfhttp或其他语言的等效语言，则需要确保在站点响应的Set-Cookie标头中返回的cookie在后续标头中返回通过使用cfhttpparam请求。可以使用各种cfhttp包装程序和替代程序库（例如绕过cfhttp层的Java包装程序）来进行此操作。但是，如果您想了解一个简单的示例，那么Ben Nadel在这里有一个古老而又不错的示例：https://www.bennadel.com/blog/725-maintaining-sessions-across-multiple-coldfusion-cfhttp-requests.htm
使用问题链接中的pdf网址，Chrome浏览器进行了几分钟的修改后，显示出如果我丢失了上一页中的cookie并保留了http_referer，那么我会看到验证码挑战，但是如果保留cookie和失去了http_referer，然后我直接获得了pdf。这表明他们关心的是Cookie，而不是引荐来源。
Ben关于SO完整性示例的副本：
<cffunction
name="GetResponseCookies"
access="public"
returntype="struct"
output="false"
hint="This parses the response of a CFHttp call and puts the cookies into a struct.">
<!--- Define arguments. --->
<cfargument
name="Response"
type="struct"
required="true"
hint="The response of a CFHttp call."
/>
<!---
Create the default struct in which we will hold
the response cookies. This struct will contain structs
and will be keyed on the name of the cookie to be set.
--->
<cfset LOCAL.Cookies = StructNew() />
<!---
Get a reference to the cookies that werew returned
from the page request. This will give us an numericly
indexed struct of cookie strings (which we will have
to parse out for values). BUT, check to make sure
that cookies were even sent in the response. If they
were not, then there is not work to be done.
--->
<cfif NOT StructKeyExists(
ARGUMENTS.Response.ResponseHeader,
"Set-Cookie"
)>
<!---
No cookies were send back in the response. Just
return the empty cookies structure.
--->
<cfreturn LOCAL.Cookies />
</cfif>
<!---
ASSERT: We know that cookie were returned in the page
response and that they are available at the key,
"Set-Cookie" of the reponse header.
--->
<!---
Now that we know that the cookies were returned, get
a reference to the struct as described above.
--->
<!---
The cookies might be coming back as a struct or they
might be coming back as a string. If there is only
ONE cookie being retunred, then it comes back as a
string. If that is the case, then re-store it as a
struct.
---><!---<cfdump var="#arguments#" label="Line 305 - arguments for function GetResponseCookies" output="D:\web\safenet_GetResponseCookies.html" FORMAT="HTML">--->
<cfif IsSimpleValue(ARGUMENTS.Response.ResponseHeader[ "Set-Cookie" ])>
<cfset LOCAL.ReturnedCookies = {} />
<cfset LOCAL.ReturnedCookies[1] = ARGUMENTS.Response.ResponseHeader[ "Set-Cookie" ] />
<cfelse>
<cfset LOCAL.ReturnedCookies = ARGUMENTS.Response.ResponseHeader[ "Set-Cookie" ] />
</cfif>
<!--- Loop over the returned cookies struct. --->
<cfloop
item="LOCAL.CookieIndex"
collection="#LOCAL.ReturnedCookies#">
<!---
As we loop through the cookie struct, get
the cookie string we want to parse.
--->
<cfset LOCAL.CookieString = LOCAL.ReturnedCookies[ LOCAL.CookieIndex ] />
<!---
For each of these cookie strings, we are going to
need to parse out the values. We can treate the
cookie string as a semi-colon delimited list.
--->
<cfloop
index="LOCAL.Index"
from="1"
to="#ListLen( LOCAL.CookieString, ';' )#"
step="1">
<!--- Get the name-value pair. --->
<cfset LOCAL.Pair = ListGetAt(
LOCAL.CookieString,
LOCAL.Index,
";"
) />
<!---
Get the name as the first part of the pair
sepparated by the equals sign.
--->
<cfset LOCAL.Name = ListFirst( LOCAL.Pair, "=" ) />
<!---
Check to see if we have a value part. Not all
cookies are going to send values of length,
which can throw off ColdFusion.
--->
<cfif (ListLen( LOCAL.Pair, "=" ) GT 1)>
<!--- Grab the rest of the list. --->
<cfset LOCAL.Value = ListRest( LOCAL.Pair, "=" ) />
<cfelse>
<!---
Since ColdFusion did not find more than one
value in the list, just get the empty string
as the value.
--->
<cfset LOCAL.Value = "" />
</cfif>
<!---
Now that we have the name-value data values,
we have to store them in the struct. If we are
looking at the first part of the cookie string,
this is going to be the name of the cookie and
it's struct index.
--->
<cfif (LOCAL.Index EQ 1)>
<!---
Create a new struct with this cookie's name
as the key in the return cookie struct.
--->
<cfset LOCAL.Cookies[ LOCAL.Name ] = StructNew() />
<!---
Now that we have the struct in place, lets
get a reference to it so that we can refer
to it in subseqent loops.
--->
<cfset LOCAL.Cookie = LOCAL.Cookies[ LOCAL.Name ] />
<!--- Store the value of this cookie. --->
<cfset LOCAL.Cookie.Value = LOCAL.Value />
<!---
Now, this cookie might have more than just
the first name-value pair. Let's create an
additional attributes struct to hold those
values.
--->
<cfset LOCAL.Cookie.Attributes = StructNew() />
<cfelse>
<!---
For all subseqent calls, just store the
name-value pair into the established
cookie's attributes strcut.
--->
<cfset LOCAL.Cookie.Attributes[ LOCAL.Name ] = LOCAL.Value />
</cfif>
</cfloop>
</cfloop>
<!--- Return the cookies. --->
<cfreturn LOCAL.Cookies />
</cffunction>
假设您从第一页https://www.osapublishing.org/boe/abstract.cfm?uri=boe-11-5-2745获得cfhttp响应，并将该响应传递到上述函数中，并将其结果保存在名为cookieStruct的变量中，则可以在后续的cfhttp请求中使用此响应：
<cfloop item="strCookie" collection="#cookieStruct#">
<cfhttpparam type="COOKIE" name="#strCookie#" value="#cookieStruct[strCookie].Value#" />
</cfloop>
编辑：如果使用wget而不是cfhttp-您可以尝试从该问题的答案中解决问题-但无需输入用户名和密码，因为您实际上不需要登录表单< / p>
How to get past the login page with Wget?
例如
# Get a session.
wget --save-cookies cookies.txt \
--keep-session-cookies \
--delete-after \
https://www.osapublishing.org/boe/abstract.cfm?uri=boe-11-5-2745
# Now grab the page or pages we care about.
# You may also need to add valid http_referer or http_user_agent headers
wget --load-cookies cookies.txt \
https://www.osapublishing.org/boe/viewmedia.cfm?uri=boe-11-5-2745&seq=0
...尽管正如其他人所指出的那样，您可能违反了源代码的服务条款，所以我不建议您实际执行此操作。