通过Python Search API搜索间歇性地超时

时间:2018-01-08 13:43:42

标签: python google-app-engine google-api

我们有一个应用程序,它基本上只是一个表单提交,用于请求创建团队驱动器。它托管在Google App Engine上。

此超时错误来自表单中的单个字段,该字段只是为电子邮件地址预先输入。域中的所有名称都在数据存储区中编入索引,大约300k个实体 - 没有任何内容直接从目录api中提取。经过10秒的搜索(通过Python Google Search API),它将会超时。目前这种情况是断断续续的,但错误的频率却在增加。

Error: line 280, in get_result raise _ToSearchError(e) Timeout: Failed to complete request in 9975ms

基本上,加快搜索速度将会解决。我查看了代码,我不相信那里有任何改进的余地。我不确定增加实例类是否会改善这一点,它目前是F2。或者,如果可能有另一种方法来提高指数效率。我不完全确定如何做到这一点。任何想法都将不胜感激。

搜索代码:

class LookupUsersorGrpService(object):

    '''
    lookupUsersOrGrps accepts various params and performs  search

    ''' 
    def lookupUsersOrGrps(self,params):
        search_results_json = {}
        search_results = []
        directory_users_grps = GoogleDirectoryUsers()
        error_msg = 'Technical error'
        query = ''

        try:

            #Default few values if not present
            if ('offset' not in params) or (params['offset'] is None):      
                params['offset'] = 0
            else:
                params['offset'] = int(params['offset'])

            if ('limit' not in params) or (params['limit'] is None): 
                params['limit'] = 20
            else:
                params['limit'] = int(params['limit'])

            #Search related to field name
            query = self.appendQueryParam(q=query, p=params,  qname='search_name',  criteria=':',  pname='query', isExactMatch=True,splitString=True)

            #Search related to field email
            query = self.appendQueryParam(q=query, p=params,  qname='search_email',  criteria=':',  pname='query', isExactMatch=True, splitString=True)

            #Perform search
            log.info('Search initialized :\"{}\"'.format(query) )

            # sort results by name ascending
            expr_list = [search.SortExpression(expression='name', default_value='',direction=search.SortExpression.ASCENDING)]
            # construct the sort options
            sort_opts = search.SortOptions(expressions=expr_list)

            #Prepare the search index
            index = search.Index(name= "GoogleDirectoryUsers",namespace="1") 
            search_query = search.Query(
                    query_string=query.strip(),
                    options=search.QueryOptions(
                                limit=params['limit'],
                                offset=params['offset'],
                                sort_options=sort_opts,
                                returned_fields = directory_users_grps.get_search_doc_return_fields()
                                ))                
            #Execute the search query
            search_result = index.search(search_query)

            #Start collecting the values
            total_cnt = search_result.number_found
            params['limit'] = len(search_result.results)

            #Prepare the response object    
            for teamdriveDoc in search_result.results:
                teamdriveRecord = GoogleDirectoryUsers.query(GoogleDirectoryUsers.email==teamdriveDoc.doc_id).get()
                if teamdriveRecord:
                    if teamdriveRecord.suspended == False:
                        search_results.append(teamdriveRecord.to_dict())

            search_results_json.update({"users" : search_results})
            search_results_json.update({"limit" : params['limit'] if len(search_results)>0 else '0'})
            search_results_json.update({"total_count" : total_cnt if len(search_results)>0 else '0'})
            search_results_json.update({"status" : "success"})

        except Exception as e:
            log.exception("Error in performing search")
            search_results_json.update({"status":"failed"})
            search_results_json.update({"description":error_msg})             

        return search_results_json   



    ''' Retrieves the given param from dict and adds to query if exists    
    '''
    def appendQueryParam(self, q='', p=[], qname=None, criteria='=', pname=None, 
        isExactMatch = False, splitString = False, defaultValue=None):        

        if (pname in p) or (defaultValue is not None):
            if len(q) > 0:
                q += ' OR '
            q += qname 
            if criteria:
                q += criteria 

            if defaultValue is None:
                val = p[pname]
            else:
                val = defaultValue            

            if splitString:
                val = val.replace("", "~")[1: -1]

            #Helps to retain passed argument as it is, example email
            if isExactMatch:
                q +=  "\"" +val + "\""
            else:
                q +=  val

        return q

1 个答案:

答案 0 :(得分:0)

索引实例的search方法接受deadline参数,因此您可以使用该参数来增加您愿意等待搜索响应的时间:

search_result = index.search(search_query, deadline=30)

documentation没有为deadline指定可接受的值,但其他App Engine服务往往接受最多60秒的值。