我正在尝试在Google App Engine上构建一个数据存储区,以便为一群公司从StockTwits中收集一些流数据。我基本上复制了我用Twitter做的一个,但它给了我一个HTTPException:其中一个URL无效和/或缺少SSL证书错误。我更改了URL以查看其他公司,但得到了相同的结果。
以下是提取数据的代码:
class StreamHandler(webapp2.RequestHandler):
def get(self):
tickers = ['AAPL','GOOG', 'IBM', 'BAC', 'INTC',
'DELL', 'C', 'JPM', 'WFM', 'WMT',
'AMZN', 'HOT', 'SPG', 'SWY', 'HTSI',
'DUK', 'CEG', 'XOM', 'F', 'WFC',
'CSCO', 'UAL', 'LUV', 'DAL', 'COST', 'YUM',
'TLT', 'HYG', 'JNK', 'LQD', 'MSFT',
'GE', 'LVS', 'MGM', 'TWX', 'DIS', 'CMCSA',
'TWC', 'ORCL', 'WPO', 'NYT', 'GM', 'JCP',
'LNKD', 'OPEN', 'NFLX', 'SBUX', 'GMCR',
'SPLS', 'BBY', 'BBBY', 'YHOO', 'MAR',
'L', 'LOW', 'HD', 'HOV', 'TOL', 'NVR', 'RYL',
'GIS', 'K', 'POST', 'KRFT', 'CHK', 'GGP',
'RSE', 'RWT', 'AIG', 'CB', 'BRK.A', 'CAT']
for i in set(tickers):
urlst = 'https://api.stocktwits.com/api/2/streams/symbol/'
tickerstringst = urlst + i + '.json'
tickurlst = urllib2.Request(tickerstringst)
sttweets = urllib2.urlopen(tickurlst)
stcode = sttweets.getcode()
if stcode == 200:
stresults = json.load(sttweets, 'utf-8')
if "messages" in stresults:
stentries = stresults["messages"]
for stentry in stentries:
sttweet = streamdata()
stcreated = stentry['created_at']
sttweetid = str(stentry['id'])
sttweettxt = stentry['body']
sttweet.ticker = i
sttweet.created_at = stcreated
sttweet.tweet_id = sttweetid
sttweet.text = sttweettxt
sttweet.source = "StockTwits"
sttweet.put()
这是显示错误的日志文件。我在本地Python开发服务器上运行它,顺便说一下:
WARNING 2012-12-06 23:20:12,993 dev_appserver.py:3655] Could not initialize images API; you are likely missing the Python "PIL" module. ImportError: No module named _imaging
INFO 2012-12-06 23:20:13,017 dev_appserver_multiprocess.py:655] Running application dev~jibdantestv2 on port 8088: http://localhost:8088
INFO 2012-12-06 23:20:13,017 dev_appserver_multiprocess.py:657] Admin console is available at: http://localhost:8088/_ah/admin
INFO 2012-12-06 23:20:54,776 dev_appserver.py:3092] "GET /_ah/admin HTTP/1.1" 302 -
INFO 2012-12-06 23:20:54,953 dev_appserver.py:3092] "GET /_ah/admin/datastore HTTP/1.1" 200 -
INFO 2012-12-06 23:20:55,280 dev_appserver.py:3092] "GET /_ah/admin/images/google.gif HTTP/1.1" 200 -
INFO 2012-12-06 23:21:04,617 dev_appserver.py:3092] "GET /_ah/admin/cron HTTP/1.1" 200 -
INFO 2012-12-06 23:21:04,815 dev_appserver.py:3092] "GET /_ah/admin/images/google.gif HTTP/1.1" 200 -
WARNING 2012-12-06 23:21:07,392 urlfetch_stub.py:448] Stripped prohibited headers from URLFetch request: ['Host']
ERROR 2012-12-06 23:21:09,921 webapp2.py:1553] Invalid and/or missing SSL certificate for URL: https://api.stocktwits.com/api/2/streams/symbol/GIS.json
Traceback (most recent call last):
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2\webapp2.py", line 1536, in __call__
rv = self.handle_exception(request, response, e)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2\webapp2.py", line 1530, in __call__
rv = self.router.dispatch(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2\webapp2.py", line 1278, in default_dispatcher
return route.handler_adapter(request, response)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2\webapp2.py", line 1102, in __call__
return handler.dispatch()
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2\webapp2.py", line 572, in dispatch
return self.handle_exception(e, self.app.debug)
File "C:\Program Files (x86)\Google\google_appengine\lib\webapp2\webapp2.py", line 570, in dispatch
return method(*args, **kwargs)
File "C:\Users\Tank\Documents\Aptana Studio 3 Workspace\jibdantestv2\main.py", line 38, in get
sttweets = urllib2.urlopen(tickurlst)
File "C:\Python27\lib\urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "C:\Python27\lib\urllib2.py", line 400, in open
response = self._open(req, data)
File "C:\Python27\lib\urllib2.py", line 418, in _open
'_open', req)
File "C:\Python27\lib\urllib2.py", line 378, in _call_chain
result = func(*args)
File "C:\Python27\lib\urllib2.py", line 1215, in https_open
return self.do_open(httplib.HTTPSConnection, req)
File "C:\Python27\lib\urllib2.py", line 1180, in do_open
r = h.getresponse(buffering=True)
File "C:\Program Files (x86)\Google\google_appengine\google\appengine\dist27\httplib.py", line 502, in getresponse
raise HTTPException(str(e))
HTTPException: Invalid and/or missing SSL certificate for URL: https://api.stocktwits.com/api/2/streams/symbol/GIS.json
INFO 2012-12-06 23:21:09,937 dev_appserver.py:3092] "GET /add_data HTTP/1.1" 500 -
答案 0 :(得分:11)
我不知道GAE为什么会出现问题,但我注意到api.stocktwits.com返回的证书与其主题公用名称(即ssl2361.cloudflare.com)上的服务器名称不匹配,但仅限于其主题替代名称之一(“DNS Name = * .stocktwits.com”)。也许主题替代名称不受支持,或者不使用此处使用的通配符名称。 (这将是Google错误/缺失功能。)
我能够通过调用GAE urlfetch.fetch API来重现您的问题并找到解决方法。 (正如您所知,在GAE上,urllib2实现为urlfetch的包装器。)
从urllib2.Request
到jason.load
的行开始,替换为:
sttweets = urlfetch.fetch(tickerstringst, validate_certificate=False)
stcode = sttweets.status_code
if stcode == 200:
stresults = json.loads(sttweets.content, 'utf-8')
您的错误消失了,以及您实际使用真实网站的任何保证(尽管流量仍应加密)。
目前,urlfetch.fetch
GAE API文档说:
validate_certificate
底层实现当前默认为False,但在不久的将来默认为True。
嗯,欢迎来到未来,因为validate_certificate现在似乎默认为True
。
这可能是GAE urlfetch.fetch中的一个错误(或缺少功能,如果你想要善意的话),我鼓励你向Google报告。
答案 1 :(得分:-1)
我对GAE不是很熟悉,因此这可能是访问API端点的问题。此外,可能是您没有使用正确的Python库来执行SSL请求,但是因为您说您在Twitter请求中使用相同的代码,可能情况并非如此。
您可以尝试在本地或其他服务器上使用相同的代码,而不是GAE吗?
答案 2 :(得分:-1)
我遇到了同样的问题,我在validate_certificate
函数中将False
参数设置为urlfetch.fetch()
。
urlfetch.fetch(url, validate_certificate=false) #validates certificate
这没有做到这一点,我发现它是处理validate_certificate
参数的方式的内部错误。如果它被设置为False
,它将验证它,如果它是True
,则它不验证证书,或者至少这是它似乎正在做的事情。
urlfetch.fetch(url, validate_certificate=true) #does not validate certificate