Question

我正在尝试从文本文件（file1.txt）中提取特定的字符串值
然后使用提取的字符串（URL地址）创建HTTP GET请求
HTTP响应应另存为新的HTML文件，目录。

我要提取的字符串是特定键的值。例如：“ display_url”：“ test.com”（提取“ test.com”，然后创建http请求）

file1.txt内容的结构可以是display_url的多个实例，因为它位于URL下的列表中。如果有多个值，则我想为每个值发出HTTP请求。

我的txt文件内容：

{"created_at":"Thu Nov 15 11:35:00 +0000 2018","id":15292802,"id_str":325802","text":"test8 https://test/ZtCsuk7Ek2 #osining","source":"\u003ca href=\"http://twitter.com\" rel=\"nofollow\"\u003eTwitter Web Client\u003c/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":961508561217052675,"id_str":"961508561217052675","name":"Online S","screen_name":"osectraining","location":"Israel","url":"https://www.test.co.il","description":"test","translator_type":"none","protected":false,"verified":false,"followers_count":2,"friends_count":51,"listed_count":0,"favourites_count":0,"statuses_count":7,"created_at":"Thu Feb 08 07:54:39 +0000 2018","utc_offset":null,"time_zone":null,"geo_enabled":false,"lang":"en","contributors_enabled":false,"is_translator":false,"profile_background_color":"000000","profile_background_image_url":"http://abs.twimg.com/images/themes/theme1/bg.png","profile_background_image_url_https":"https://abs.twimg.com/images/themes/theme1/bg.png","profile_background_tile":false,"profile_link_color":"1B95E0","profile_sidebar_border_color":"000000","profile_sidebar_fill_color":"000000","profile_text_color":"000000","profile_use_background_image":false,"profile_image_url":"http://pbs.twimg.com/profile_images/961510231346958336/d_KhBeTD_normal.jpg","profile_image_url_https":"https://pbs.twimg.com/profile_images/961510231346958336/d_KhBeTD_normal.jpg","profile_banner_url":"https://pbs.twimg.com/profile_banners/961508561217052675/1518076913","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[{"text":"osectraining","indices":[33,46]}],"urls":[{"url":"https://test/ZtCsuk7Ek2","expanded_url":"http://test.com","display_url":"test.com","indices":[7,30]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"possibly_sensitive":false,"filter_level":"low","lang":"en","timestamp_ms":"1542281700508"}

Answer 1

1）看起来您的文件不是有效的JSON文件，因此对于第1步，您必须执行以下操作：

url=${ cat /tmp/x.txt | grep -oP '(?<=display_url":")[^"]+' }

2＆3）现在，您可以执行以下操作：

curl $url -O /tmp/x.html

如果您有> 1个display_urls-必须使用循环，如下所示：

for url in $display_urls; do
    curl $url -O /tmp/$url.html
done

从文本文件中提取特定的字符串，并使用提取的字符串创建HTTP请求

1 个答案: