Question

如何使用Powershell从给定网址检索网站元数据，例如标题，说明，关键字？

例如：给出以下网址

输入： www.amazon.com

输出

title: "Amazon.com: Online Shopping for Electronics, Apparel, Computers, Books, DVDs & more",
description: "Online shopping from the earth's biggest selection of books, magazines, music, DVDs, videos, electronics, computers, software, apparel & accessories, shoes, jewelry, tools & hardware, housewares, furniture, sporting goods, beauty & personal care, broadband & dsl, gourmet food & just about anything else.",
keyword: "Amazon, Amazon.com, Books, Online Shopping, Book Store, Magazine, Subscription, Music, CDs, DVDs, Videos, Electronics, Video Games, Computers, Cell Phones, Toys, Games, Apparel, Accessories, Shoes, Jewelry, Watches, Office Products, Sports & Outdoors, Sporting Goods, Baby Products, Health, Personal Care, Beauty, Home, Garden, Bed & Bath, Furniture, Tools, Hardware, Vacuums, Outdoor Living, Automotive Parts, Pet Supplies, Broadband, DSL"

输入 www.youtube.com

输出

title: YouTube
description: Enjoy the videos and music you love, upload original content and share it all with friends, family and the world on YouTube.
keywords: video, sharing, camera phone, video phone, free, upload

Answer 1

无法保证您访问的网站将以任何实际方式显示您想要公开的数据。您可以使用Invoke-WebRequest和Invoke-RestMethod轻松检索网页，但是您需要解析返回的标头/数据。

Answer 2

您可以使用Invoke-WebRequest，然后使用正则表达式匹配您想要的字符串：

$response = Invoke-WebRequest -Uri www.amazon.com -UseBasicParsing

PS C:\> $response.Content -match "<title>(?<title>.*)</title>" | out-null

$matches['title']
Amazon.com: Online Shopping for Electronics, Apparel, Computers, Books, DVDs & more

PS C:\> $response.Content -match "<meta name=`"description`" content=`"(?<description>.*)`">" | out-null

$matches['description']
Online shopping from the earth's biggest selection of books, magazines, music, DVDs, videos, electronics, computers, software, apparel & accessories, shoes, jewelry, tools & hardware, housewares, furniture, sporting goods, beauty & personal care, broadband & dsl, gourmet food & just about anything else.

PS C:\> $response.Content -match "<meta name=`"keywords`" content=`"(?<keywords>.*)`">" | out-null

$matches['keywords']
Amazon, Amazon.com, Books, Online Shopping, Book Store, Magazine, Subscription, Music, CDs, DVDs, Videos, Electronics, Video Games, Computers, Cell Phones, Toys, Games, Apparel, Accessories, Shoes, Jewelry, Watches, Office Products, Sports & Outdoors, Sporting Goods, Baby Products, Health, Personal Care, Beauty, Home, Garden, Bed & Bath, Furniture, Tools, Hardware, Vacuums, Outdoor Living, Automotive Parts, Pet Supplies, Broadband, DSL

这将取决于所有使用相同模式的元数据的网站。例如，上面的内容不适用于Stack Overflow的网站，因为它们用'/＆gt;'关闭了他们的元字段。

Answer 3

正如@StephenP所说

无法保证您访问的网站会有数据你希望以任何实际的方式暴露。您可以轻松检索使用Invoke-WebRequest和Invoke-RestMethod的网页然后你将需要解析返回的标头/数据

此外，无法保证网站不会试图阻止您正在做的事情。

下面是使用.NET HTML DOM进行解析的示例。 @ tim-aitken使用RegEx找到相同信息的例子，但他提到这将取决于使RegEx正确。另一方面，有时HTML DOM也无法解析页面。

# First retrieve the website
$result = Invoke-webrequest -Uri http://www.youtube.com/ -Method Get
$resultTable = @{}

# Get the title
$resultTable.title = $result.ParsedHtml.title

# Get the HTML Tag
$HtmlTag = $result.ParsedHtml.childNodes | Where-Object {$_.nodename -eq 'HTML'} 

# Get the HEAD Tag
$HeadTag = $HtmlTag.childNodes | Where-Object {$_.nodename -eq 'HEAD'}

# Get the Meta Tags
$MetaTags = $HeadTag.childNodes| Where-Object {$_.nodename -eq 'META'}

# You can view these using $metaTags | select outerhtml | fl 
# Get the value on content from the meta tag having the attribute with the name keywords
$resultTable.keywords = $metaTags  | Where-Object {$_.name -eq 'keywords'} | Select-Object -ExpandProperty content

# Do the same for description
$resultTable.description = $metaTags  | Where-Object {$_.name -eq 'description'} | Select-Object -ExpandProperty content

# Return the table we have built as an object
Write-Output New-Object -TypeName PSCustomObject -Property $resultTable

使用Powershell从给定网址获取网站元数据，例如标题，描述

3 个答案: