Question

我正在尝试从以下<iframes>属性中提取来源：

   iframes =  [<iframe frameborder="no" height="160px" scrolling="no" src="https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/308197184%3Fsecret_token%3Ds-VtArH&amp;auto_play=false&amp;hide_related=false&amp;show_comments=true&amp;show_user=true&amp;show_reposts=false&amp;visual=true" width="100%"></iframe>, <iframe allowtransparency="true" frameborder="0" scrolling="no" src="//www.facebook.com/plugins/likebox.php?href=https%3A%2F%2Fwww.facebook.com%2FPauseMusicale&amp;width=300&amp;height=62&amp;show_faces=false&amp;colorscheme=light&amp;stream=false&amp;show_border=false&amp;header=false" style="border:none; overflow:hidden; width:300px; height:62px;"></iframe>, <iframe allowfullscreen="" frameborder="0" height="169" src="//www.youtube.com/embed/videoseries?list=PLNKCTdT9YSESoQnj5tPP4P9kaIwBCx7F1" width="100%"></iframe>]

但是当我尝试提取它时：

for iframe in iframes:
    url = urllib2.urlopen(iframe.attrs['src'])
    print (url)

我收到以下错误：

   url = urllib2.urlopen(iframe.attrs['src'])
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 423, in open
    protocol = req.get_type()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 285, in get_type
    raise ValueError, "unknown url type: %s" % self.__original
ValueError: unknown url type: //www.facebook.com/plugins/likebox.php?href=https%3A%2F%2Fwww.facebook.com%2FPauseMusicale&width=300&height=62&show_faces=false&colorscheme=light&stream=false&show_border=false&header=false

为什么我在http之前收到没有//www的网址？

这有解决方法吗？

Answer 1

为什么我在// www
之前没有http的网址

这是向用户代理指示在进行后续请求时应使用与当前页面相同的方案（http，https，ftp，文件等）的常用方法。因此，例如，如果您通过https加载当前页面，那么将使用https访问省略该方案的那些URL。

这有解决方法吗？

您可以使用urlparse模块在Python 2中处理此问题（因为这是您的Python版本）：

# from urllib.parse import urlparse, urlunparse    # Python 3
from urlparse import urlparse, urlunparse

for iframe in iframes:
    scheme, netloc, path, params, query, fragment = urlparse(iframe.attrs['src'])
    if not scheme:
        scheme = 'http'    # default scheme you used when getting the current page
    url = urlunparse((scheme, netloc, path, params, query, fragment))
    print('Fetching {}'.format(url))
    f = urllib2.urlopen(url)
#    print(f.read())    # dumps the response content

如果你运行上面的代码，你应该看到这个输出：

Fetching https://w.soundcloud.com/player/?url=https%3A//api.soundcloud.com/tracks/308197184%3Fsecret_token%3Ds-VtArH&auto_play=false&hide_related=false&show_comments=true&show_user=true&show_reposts=false&visual=true
Fetching http://www.facebook.com/plugins/likebox.php?href=https%3A%2F%2Fwww.facebook.com%2FPauseMusicale&width=300&height=62&show_faces=false&colorscheme=light&stream=false&show_border=false&header=false
Fetching http://www.youtube.com/embed/videoseries?list=PLNKCTdT9YSESoQnj5tPP4P9kaIwBCx7F1

表示默认方案已应用于URL。

Python - ValueError：未知的url类型

1 个答案: