Question

我不断收到类似的随机错误：

suspended generator _get_tasklet(context.py:329) raised ProtocolBufferDecodeError(corrupted)

或

suspended generator put(context.py:796) raised ValueError(Expecting , delimiter: line 1 column 440 (char 440))

或

suspended generator put(context.py:796) raised ValueError(Invalid \escape: line 1 column 18002 (char 18002))

或

suspended generator _get_tasklet(context.py:329) raised ProtocolBufferDecodeError(truncated)

直到几天前，一切都工作正常，我没有做任何改变。当我重新启动我的应用程序时，一切都很好，大约五分钟，直到我得到一个

suspended generator _get_tasklet(context.py:329) raised ProtocolBufferDecodeError(corrupted)

在那之后，我得到了每个数据库中的其他错误之一。导致错误的表和代码每次都不同。我不知道从哪里开始，因为错误每次都在一个新的地方。这些只是常规的数据库放置和获取，如

ndbstate = NdbStateJ.get_by_id(self.screen_name)

或

ndbstate.put()

谷歌搜索未能指出我的任何特定方向。有任何想法吗？错误

Expecting , delimiter: line 1 column 440 (char 440)

可能是因为某些表中的某些字段类型是JSON。但为什么突然间呢？

所以也许我没有在某个地方正确逃脱，比如使用r'{...}'，但如果某处有错误的条目，如果我无法查询，如何解决？为什么它会破坏所有查询的整个表格？为什么它是随机的。每次都不是同一个查询。

以下是表格的示例

class NdbStateJ(ndb.Model):
    last_id = ndb.IntegerProperty()
    last_search_id = ndb.IntegerProperty()
    last_geo_id = ndb.IntegerProperty()
    mytweet_num = ndb.IntegerProperty()
    mentions_processed = ndb.JsonProperty()
    previous_follower_responses = ndb.JsonProperty()
    my_tweets_tweeted = ndb.JsonProperty()
    responses_already_used = ndb.JsonProperty()
    num_followed_by_cyborg = ndb.IntegerProperty(default=0)
    num_did_not_follow_back = ndb.IntegerProperty(default=0)
    language_model_vector = ndb.FloatProperty(repeated=True)
    follow_wait_counter = ndb.IntegerProperty(default=0)

以下是创建表格的示例

ndbstate = NdbStateJ(id=screen_name,
last_id = 37397357946732541,
last_geo_id = 37397357946732541,
last_search_id = 0,
mytweet_num = 0,
mentions_processed = [],
previous_follower_responses = [],
my_tweets_tweeted = [],
responses_already_used= [],
language_model_vector = [])
ndbstate.put()

Answer 1

数据库中出现格式错误的JSON导致问题。我不知道为什么突然间问题开始发生在各处;也许谷歌方面有些变化，或者我没有充分检查，新用户能够输入格式错误的数据。谁知道呢。

为了解决这个问题，我从https://stackoverflow.com/users/1011633/nizz回复App Engine return JSON from JsonProperty，https://stackoverflow.com/users/1709587/mark-amery回复How to escape special characters in building a JSON string?和https://stackoverflow.com/users/1639625/tobias-k回复How do I automatically fix an invalid JSON string?获取灵感。

我将ndb.JsonProperty()替换为ExtendedJsonProperty，其中扩展版本与下面的代码类似。

import json
from google.appengine.ext import ndb 
import logging
logging.getLogger().setLevel(logging.DEBUG)
import re

class ExtendedJsonProperty(ndb.BlobProperty):
    # Inspired by https://stackoverflow.com/questions/18576556/app-engine-return-json-from-jsonproperty
    def _to_base_type(self, value):
        logging.debug('Dumping value '+str(value))
        try:
            return json.dumps(value) 
        except Exception as e:
            logging.warning(('trying to fix error dumping from database: ') +str(e))
            return fix_json(value,json.dumps)

    def _from_base_type(self, value):
        # originally return json.loads(value)
        logging.debug('Loading value '+str(value))
        try:
            return json.loads(value)
        except Exception as e:
            logging.warning(('trying to fix error loading from database: ') +str(e))
            return fix_json(value,json.loads)        

def fix_json(s,json_fun):
    for _i in range(len(s)):
        try:
            result = json_fun(s)   # try to parse...
            return result                    
        except Exception as e:  
            logging.debug('Exception for json loads: '+str(e))          
            if 'delimiter' in str(e):
                # E.g.: "Expecting , delimiter: line 34 column 54 (char 1158)"
                logging.debug('Escaping quote to fix.')
                s = escape_quote(s,e)
            elif 'escape' in str(e):
                # E.g.: "Invalid \escape: line 1 column 9 (char 9)"
                logging.debug('Removing invalid escape to fix.')
                s = remove_invalid_escape(s)
            else:
                break
    return json_fun('{}')

def remove_invalid_escape(value):
    # Inspired by https://stackoverflow.com/questions/19176024/how-to-escape-special-characters-in-building-a-json-string
    return re.sub(r'\\(?!["\\/bfnrt])', '', value)

def escape_quote(s,e):
    # Inspired by https://stackoverflow.com/questions/18514910/how-do-i-automatically-fix-an-invalid-json-string
    # "Expecting , delimiter: line 34 column 54 (char 1158)"
    # position of unexpected character after '"'
    unexp = int(re.findall(r'\(char (\d+)\)', str(e))[0])
    # position of unescaped '"' before that
    unesc = s.rfind(r'"', 0, unexp)
    s = s[:unesc] + r'\"' + s[unesc+1:]
    # position of corresponding closing '"' (+2 for inserted '\')
    closg = s.find(r'"', unesc + 2)
    if closg + 2 < len(s):
        print closg, len(s)
        s = s[:closg] + r'\"' + s[closg+1:]
    return s

Google App Engine NDB数据库损坏了吗？

1 个答案: