Question

我从网址下载了一些文件。

我目前可以像这样访问我的文件：

import requests
from bs4 import BeautifulSoup
import os

prefix = 'https://n5eil01u.ecs.nsidc.org/MOST/MOD10A1.006/'
download_url = "https:/path_to_website"

s = requests.session()                                                         
soup = BeautifulSoup(s.get(download_url).text, "lxml")  

for a in soup.find_all('a', href=True):

     final_link = os.path.join(prefix, a['href'])
     result = s.get(final_link, stream = True)
     with open(a['href'], 'wb') as out_file:
          shutil.copyfileobj(result.raw, out_file)

这将下载文件并将其放入C：/ User。

的默认目录中

我想选择下载文件的位置。您可以选择路径与wget的位置，但我的方法会下载空文件，就好像它们没有被访问一样。

我用这样的wget尝试了这个：

out_path = "C:/my_path"
prefix = 'https://n5eil01u.ecs.nsidc.org/MOST/MOD10A1.006/'

s = requests.session()                                                         
soup = BeautifulSoup(s.get(download_url).text, "lxml")  

for a in page.find_all('a', href=True):

     final_link = os.path.join(prefix, a['href'])
     download = wget.download(final_link, out = out_path)

我认为wget无法正常工作，因为我正在访问带有身份验证的网站（未显示），当我加入最终链接时，我不再通过身份验证访问它。有没有办法用shutil指定outpath？

Answer 1

如何使用第一种方法替换用os.path.join(out_path, a['href'])打开的文件的路径？

import requests
from bs4 import BeautifulSoup
import os

out_path = "C:\\my_path"    
prefix = 'https://n5eil01u.ecs.nsidc.org/MOST/MOD10A1.006/'
download_url = "https:/path_to_website"

s = requests.session()                                                         
soup = BeautifulSoup(s.get(download_url).text, "lxml")  

for a in soup.find_all('a', href=True):
     final_link = os.path.join(prefix, a['href'])
     result = s.get(final_link, stream = True)
     new_file_path = os.path.join(out_path, a['href'])
     with open(new_file_path, 'wb') as out_file:    # this will create the new file at new_file_path
          shutil.copyfileobj(result.raw, out_file)

Answer 2

您可以创建如下所示的目标路径

target_path = r'c:\windows\temp'
with open(os.path.join(target_path, a['href']), 'wb') as out_file:
    shutil.copyfileobj(result.raw, out_file)

从URL

2 个答案: