从数据库中提取数据导致错误堆栈溢出

时间:2018-03-07 19:39:38

标签: python database python-3.x stack-overflow

我正在运行一个代码,该代码从S& P 500的500家公司获取信息。每次到达特定公司ANDV时,我都会收到堆栈溢出致命错误。有关如何解决此问题的任何建议?我添加了time.sleep(2)来对抗任何潜在的API调用限制。

以下是代码:

import bs4 as bs
import pickle
import requests, time
import datetime as dt
import os
import pandas as pd
import pandas_datareader.data as web

def save_sp500_tickers():
    resp = requests.get('https://en.wikipedia.org/wiki/List_of_S%26P_500_companies')
    soup = bs.BeautifulSoup(resp.text, "lxml")
    table = soup.find('table', {'class':'wikitable sortable'})
    tickers = []
    for row in table.findAll('tr')[1:]:
        ticker = row.findAll('td')[0].text
        tickers.append(ticker)

    with open("sp500tickers.pickle", "wb") as f:
        pickle.dump(tickers, f)

    print(tickers)

    return  tickers

#save_sp500_tickers()

def get_data_from_morningstar(reload_sp500=False):
    if reload_sp500:
        tickers = save_sp500_tickers()
    else:
        with open("sp500tickers.pickle","rb") as f:
            tickers = pickle.load(f)
    if not os.path.exists('stock_dfs'):
        os.makedirs('stock_dfs')

    start = dt.datetime(2000,1,1)
    end = dt.datetime(2016,12,31)

    for ticker in tickers:
        print(ticker)
        time.sleep(2)
        if not os.path.exists('stock_dfs/{}.csv'.format(ticker)):
            df = web.DataReader(ticker, 'morningstar', start, end)
            df.to_csv('stock_dfs/{}.csv'.format(ticker))
        else:
            print('Already have {}'.format(ticker))

get_data_from_morningstar()

错误,显着缩短文件线程,是这样的:

ANDV
adding ANDV to retry list
Fatal Python error: Cannot recover from stack overflow.

Current thread 0x00001690 (most recent call first):
  File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\enum.py", line 519 in __new__
  File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\enum.py", line 291 in __call__
  File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\socket.py", line 103 in _intenum_converter    
  File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\socket.py", line 747 in getaddrinfo
  File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\util\connection.py", line 60 in create_connection
  File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connection.py", line 141 in _new_conn
  File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3\connection.py", line 166 in connect
  File "C:\Users\Aaron Mazie\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pandas_datareader\mstar\daily.py", line 137 in _dl_mult_symbols
  ...

谢谢!

0 个答案:

没有答案