这是我发现的一个脚本,用于使用grad cafe网站搜索不同程序的研究生入学结果。但是,当我运行它以查找“政治科学”的结果时,它表明我有以下错误
Traceback (most recent call last):
File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 57, in <module>
data = get_data()
File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 52, in get_data
pages = get_pages()
File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 45, in get_pages
n = find_n_pages()
File "C:/Users/lakna/OneDrive/Desktop/Spring 2018/Statistical Programming/Final Project/gradcafe_scraping.py", line 41, in find_n_pages
reg = re.search('over\s([\d]*)\spages',html)
File "C:\Users\lakna\AppData\Local\Programs\Python\Python36-32\lib\re.py", line 182, in search
return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object
我该如何解决这个问题?以下是我使用的代码
def get_page(i=0, keyword="Political Science"):
time.sleep(10)
if i==0:
#To change subjects, you want to change the keyword to say biostatistics,
#test it by searching on the site to make sure you get what you want.
url = "http://thegradcafe.com/survey/index.php?q="+keyword+"*&t=a&o=&pp=250"
else:
url="http://thegradcafe.com/survey/index.php?q="+keyword+"*&t=a&pp=250&o=&p="+str(i)
response = urlopen(url)
html = response.read().decode('utf-8')
return
def find_n_pages():
html = get_page()
reg = re.search('over\s([\d]*)\spages',html)
return int(reg.groups()[0])
def get_pages():
n = find_n_pages()
print ("Getting",n,"pages.")
pages = [get_page(i) for i in range(1,n+1)]
return pages
def get_data():
data=[]
pages = get_pages()
for page in pages:
data=get_data_from_page(page,data)
return data
答案 0 :(得分:1)
您的get_page
函数未返回html,它返回无。
def get_page(i=0, keyword="Political Science"):
...
html = response.read().decode('utf-8')
return # this is equivalent to return None (or not having this line at all)
应为:
def get_page(i=0, keyword="Political Science"):
...
return response.read().decode('utf-8')
因此错误:
In [11]: re.search("", None)
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-11-1858f7517272> in <module>()
----> 1 re.search("", None)
/Users/andy/.miniconda3/lib/python3.6/re.py in search(pattern, string, flags)
180 """Scan through string looking for a match to the pattern, returning
181 a match object, or None if no match was found."""
--> 182 return _compile(pattern, flags).search(string)
183
184 def sub(pattern, repl, string, count=0, flags=0):
TypeError: expected string or bytes-like object