Question

让我先说一下，我知道当有这么多更好的工具时，sed中的网页抓取是一个愚蠢的想法。我试着告诉我的老师，但他坚持使用sed和awk。

我的目标是抓一个推特页面并返回所有推文。使用此代码：

sed -n '/<p class=\"TweetTextSize  js-tweet-text tweet-text\" lang=\"en\" data-aria-label-part=\"0\">/,/<\/p>/p' $targ | sed 's/<[^>]\+>/ /g;s/^[ \t]*//;/^$/d'

On this URL到目前为止，我完全无法获得我想要的东西。

以下是推文周围的代码：

Where is our universe headed? Answer to it's death, not to worry <a href="/PaulMattSutter" class="twitter-atreply pretty-link js-nav" dir="ltr" data-mentioned-user-id="556263627" ><s>@</s>PaulMattSutter</a> explains. <a href="https://t.co/ig8KtQzwOI" rel="nofollow" dir="ltr" data-expanded-url="http://oak.ctx.ly/r/42bnj" class="twitter-timeline-link" target="_blank" title="http://oak.ctx.ly/r/42bnj" >http://oak.ctx.ly/r/42bnj </a> <a href="https://t.co/kOy4jLCogk" class="twitter-timeline-link u-hidden" data-pre-embedded="true" dir="ltr" >pic.twitter.com/kOy4jLCogk</a>

现在，我想只从及其对应的之间的页面中抓取内容，然后我想对其进行过滤，以便中间的所有标记（包括打开和关闭）都被删除，所以它看起来像这样：

Where is our universe headed? Answer to it's death, not to worry @PaulMattSutter explains. http://oak.ctx.ly/r/42bnj pic.twitter.com/kOy4jLCogk

Answer 1

试试这个：

 sed -n '/<p class="TweetTextSize  js-tweet-text tweet-text" lang="en" data-aria-label-part="0">.*<\/p>/{s/<[^>]*>//g;p}' file

输出：

        You’ve never seen Pluto like this! @NASANewHorizons returns sharpest views yet! Take a look: http://go.nasa.gov/1NxLViz  pic.twitter.com/iK4OJKeLUE
        The world is filled with those who do nothing yet criticize. I love this @rickygervais motto: Go create! pic.twitter.com/m7zB8ucclT
        Can you see me waving? How to spot #Mars in the night sky: https://youtu.be/hv8hVvJlcJQ  https://amp.twimg.com/v/9b945202-5a89-4efa-80ab-514a0c17a965 …
        We knew the first high-res images of Pluto would be good, but we didn't expect THIS good: http://ow.ly/Vvrvm  pic.twitter.com/r8ZX4Z5tVt
        Toddler wears the most precious 'Monsters, Inc.' costume this world has ever seen http://on.mash.to/1HLrcMk  pic.twitter.com/2PyuDW9yms
        Are you on your way to your @TMobile store, @ATT customers?! GO!! #UncarrierUnwrapped http://t-mo.co/1SzhOfb  pic.twitter.com/0vNJbD2J4k
        1 year ago today, @USNavy @ussanchorage recovered Orion in Pacific ocean after successful flight. pic.twitter.com/DKW00LzlC3
        How Fallout 4 mastermind Todd Howard builds his epic dream worlds http://wrd.cm/1IILf8N  pic.twitter.com/WUV98e6U62
        23-year-old’s design collaboration tool Figma launches with $14M to fight Adobe http://tcrn.ch/1YPW8Pw  pic.twitter.com/wVrZbfgBOc
        Here’s a list of questions investors will ask themselves after you pitch a company to them. http://kpcb.cc/b466b5b  pic.twitter.com/zOJDOrq0na
        Blow your mind with this--ENJOY! (And check the SWEET video too)! http://www.nasa.gov/feature/new-horizons-returns-first-best-images-of-pluto … #PlutoFlyby pic.t
        Human gene editing is OK, but no designer babies—for now, summit concludes: http://ow.ly/VscSg  #GeneEditSummit pic.twitter.com/UW6DKtQX4h
        NASA releases the highest resolution images ever taken of Pluto http://www.theverge.com/2015/12/4/9851162/nasa-new-horizons-pluto-photo-new-high-resolution?utm_campai
        A drone that finds you a parking space and then hovers in it to stop anyone else taking it first.
        The ubiquity of guns poses a much greater threat to Americans than terrorism http://bit.ly/1lbvPoQ  By @Horganism pic.twitter.com/pdxvVqGS59
        Scientist disinvited from speaking at conference because of her pregnancy http://ow.ly/Vvyam  pic.twitter.com/FUAjaycIom
        Samsung agrees to pay Apple $548 million in settlement http://engt.co/1jEN28t  pic.twitter.com/3DmmccJUgW
        World's first in-human gene-editing treatment will tackle hemophilia http://www.engadget.com/2015/12/03/worlds-first-in-human-CRISPR-hemophilia/ … pic.twitte
        Where is our universe headed?  Answer to it's death, not to worry @PaulMattSutter explains. http://oak.ctx.ly/r/42bnj  pic.twitter.com/kOy4jLCogk
        New Horizons just sent back the sharpest images possible of Pluto's surface. Drink it in http://www.wired.com/2015/12/new-horizons-just-sent-its-highest-res-image
        Get people to do what you want with these 11 clever psychological tricks: http://lifehac.kr/GZaSWhN  pic.twitter.com/5rhWlVL9t0

使用SED进行WebScraping

1 个答案: