我有一个包含词典列表的文件,其中大多数都没有用引号标记。一个例子如下:
{game:Available,player:Available,location:"Chelsea, London, England",time:Available}
{"game":"Available","player":"Available","location":"Chelsea, London, England","time":"Available","date":"Available"}
如您所见,密钥也可能因字典而异。
我尝试用json模块或csv模块的DictReader来读取它,但是每次由于“”始终存在于位置值时遇到困难,但并不总是存在其他键或值。到目前为止,我看到了两种可能性:
PS:我的最后一点是能够格式化所有这些词典以创建一个SQL表,其中列是所有词典的联合,每行是我的词典之一,当缺少值时为空。
答案 0 :(得分:1)
如果它比你给出的例子更复杂,或者如果它必须更快,你应该考虑pyparsing。
否则你可以像这样编写更多hacky:
contentlines = ["""{"game":"Available","player":"Available","location":"Chelsea, London, England","time":"Available","date":"Available"}""", """{game:Available,player:Available,location:"Chelsea, London, England",time:Available}"""]
def get_dict(line):
keys = []
values = []
line = line.replace("{", "").replace("}", "")
contlist = line.split(":")
keys.append(contlist[0].strip('"').strip("'"))
for entry in contlist[1:-1]:
entry = entry.strip()
if entry[0] == "'" or entry[0] == '"':
endpos = entry[1:].find(entry[0]) + 2
else:
endpos = entry.find(",")
values.append(entry[0:endpos].strip('"').strip("'"))
keys.append(entry[endpos + 1:].strip('"').strip("'"))
values.append(contlist[-1].strip('"').strip("'"))
return dict(zip(keys, values))
for line in contentlines:
print get_dict(line)
答案 1 :(得分:1)
我认为这是一个非常完整的代码。
首先我创建了以下文件:
{surprise : "perturbating at start ", game:Available Universal Dices Game,
player:FTROE875574,location
:"Lakeview School, Kingsmere Boulevard, Saskatoon, Saskatchewan , Canada",time:15h18}
{"game":"Available"," player":"LOI4531",
"location": "Perth, Australia","time":"08h13","date":"Available"}
{"game":Available,player:PLLI874,location:"Chelsea, London, England",time:20h35}
{special:"midnight happening",game:"Available","player":YTR44,
"location":"Paris, France","time":"02h24"
,
"date":"Available"}
{game:Available,surprise:" hretyuuhuhu ",player:FT875,location
:,"time":11h22}
{"game":"Available","player":"LOI4531","location":
"Damas,Syria","time":"unavailable","date":"Available"}
{"surprise " : GARAMANANATALA Tower , game:Available Dices,player :
PuLuLu874,location:" Westminster, London, England ",time:20h01}
{"game":"Available",special:"overnight", "player":YTR44,"location":
"Madrid, Spain" , "time":
"12h33",
date:"Available"
}
然后,以下代码分两个阶段处理文件的内容:
首先,通过内容,收集所有词典中的所有干预键
扣除字典 posis ,为每个键提供其对应值必须连续占据的位置
其次,感谢另一个浏览文件,这些行是一个接一个地构建并收集在列表中行
顺便提一下,请注意与关键位置或“位置”相关联的值的条件得到遵守。
import re
dicreg = re.compile('(?<=\{)[^}]*}')
kvregx = re.compile('[ \r\n]*'
'(" *)?((location)|[^:]+?)(?(1) *")'
'[ \r\n]*'
':'
'[ \r\n]*'
'(?(3)|(" *)?)([^:]*?)(?(4) *")'
'[ \r\n]*(?:,(?=[^,]+?:)|\})')
checking_dict = {}
checking_list = []
filename = 'zzz.txt'
with open(filename) as f:
######## First part: to gather all the keys in all the dictionaries
prec,chunk = '','go'
ecr = []
while chunk:
chunk = f.read(120)
ss = ''.join((prec,chunk))
ecr.append('\n\n------------------------------------------------------------\nss == %r' %ss)
mat_dic = None
for mat_dic in dicreg.finditer(ss):
ecr.append('\nmmmmmmm dictionary found in ss mmmmmmmmmmmmmm')
for mat_kv in kvregx.finditer(mat_dic.group()):
k,v = mat_kv.group(2,5)
ecr.append('%s : %s' % (k,v))
if k in checking_list:
checking_dict[k] += 1
else:
checking_list.append(k)
checking_dict[k] = 1
if mat_dic:
prec = ss[mat_dic.end():]
else:
prec += chunk
print '\n'.join(ecr)
print '\n\n\nchecking_dict == %s\n\nchecking_list == %s' %(checking_dict,checking_list)
######## The keys are sorted in order that the less frequent ones are at the end
checking_list.sort(key=lambda k: checking_dict[k], reverse=True)
posis = dict((k,i) for i,k in enumerate(checking_list))
print '\nchecking_list sorted == %s\n\nposis == %s' % (checking_list,posis)
######## Now, the file is read again to build a list of rows
f.seek(0,0) # the file's pointer is move backed to the beginning of the file
prec,chunk = '','go'
base = [ '' for i in xrange(len(checking_list))]
rows = []
while chunk:
chunk = f.read(110)
ss = ''.join((prec,chunk))
mat_dic = None
for mat_dic in dicreg.finditer(ss):
li = base[:]
for mat_kv in kvregx.finditer(mat_dic.group()):
k,v = mat_kv.group(2,5)
li[posis[k]] = v
rows.append(li)
if mat_dic:
prec = ss[mat_dic.end():]
else:
prec += chunk
print '\n\n%s\n%s' % (checking_list,30*'___')
print '\n'.join(str(li) for li in rows)
结果
------------------------------------------------------------
ss == '{surprise : "perturbating at start ", game:Available Universal Dices Game,\n player:FTROE875574,location\n:"Lakeview S'
------------------------------------------------------------
ss == '{surprise : "perturbating at start ", game:Available Universal Dices Game,\n player:FTROE875574,location\n:"Lakeview School, Kingsmere Boulevard, Saskatoon, Saskatchewan , Canada",time:15h18}\n\n{"game":"Available"," player":"LOI4531",\n"l'
mmmmmmm dictionary found in ss mmmmmmmmmmmmmm
surprise : perturbating at start
game : Available Universal Dices Game
player : FTROE875574
location : "Lakeview School, Kingsmere Boulevard, Saskatoon, Saskatchewan , Canada"
time : 15h18
------------------------------------------------------------
ss == '\n\n{"game":"Available"," player":"LOI4531",\n"location": "Perth, Australia","time":"08h13","date":"Available"}\n\n{"game":Available,player:PLLI874,location:"Chelsea, Lo'
mmmmmmm dictionary found in ss mmmmmmmmmmmmmm
game : Available
player : LOI4531
location : "Perth, Australia"
time : 08h13
date : Available
------------------------------------------------------------
ss == '\n\n{"game":Available,player:PLLI874,location:"Chelsea, London, England",time:20h35}\n\n{special:"midnight happening",game:"Available","player":YTR44,\n"location":"Paris, France","t'
mmmmmmm dictionary found in ss mmmmmmmmmmmmmm
game : Available
player : PLLI874
location : "Chelsea, London, England"
time : 20h35
------------------------------------------------------------
ss == '\n\n{special:"midnight happening",game:"Available","player":YTR44,\n"location":"Paris, France","time":"02h24"\n,\n"date":"Available"}\n\n{game:Available,surprise:" hretyuuhuhu ",player:FT875,location\n:,"time":11h22}\n\n{"'
mmmmmmm dictionary found in ss mmmmmmmmmmmmmm
special : midnight happening
game : Available
player : YTR44
location : "Paris, France"
time : 02h24
date : Available
mmmmmmm dictionary found in ss mmmmmmmmmmmmmm
game : Available
surprise : hretyuuhuhu
player : FT875
location :
time : 11h22
------------------------------------------------------------
ss == '\n\n{"game":"Available","player":"LOI4531","location":\n"Damas,Syria","time":"unavailable","date":"Available"}\n\n{"surprise " '
mmmmmmm dictionary found in ss mmmmmmmmmmmmmm
game : Available
player : LOI4531
location : "Damas,Syria"
time : unavailable
date : Available
------------------------------------------------------------
ss == '\n\n{"surprise " : GARAMANANATALA Tower , game:Available Dices,player :\n PuLuLu874,location:" Westminster, London, England ",time:20'
------------------------------------------------------------
ss == '\n\n{"surprise " : GARAMANANATALA Tower , game:Available Dices,player :\n PuLuLu874,location:" Westminster, London, England ",time:20h01}\n\n{"game":"Available",special:"overnight", "player":YTR44,"location":\n"Madrid, Spain" , "time":\n"12h33",\nda'
mmmmmmm dictionary found in ss mmmmmmmmmmmmmm
surprise : GARAMANANATALA Tower
game : Available Dices
player : PuLuLu874
location : " Westminster, London, England "
time : 20h01
------------------------------------------------------------
ss == '\n\n{"game":"Available",special:"overnight", "player":YTR44,"location":\n"Madrid, Spain" , "time":\n"12h33",\ndate:"Available"\n}'
mmmmmmm dictionary found in ss mmmmmmmmmmmmmm
game : Available
special : overnight
player : YTR44
location : "Madrid, Spain"
time : 12h33
date : Available
------------------------------------------------------------
ss == ''
checking_dict == {'player': 8, 'game': 8, 'location': 8, 'time': 8, 'date': 4, 'surprise': 3, 'special': 2}
checking_list == ['surprise', 'game', 'player', 'location', 'time', 'date', 'special']
checking_list sorted == ['game', 'player', 'location', 'time', 'date', 'surprise', 'special']
posis == {'player': 1, 'game': 0, 'location': 2, 'time': 3, 'date': 4, 'surprise': 5, 'special': 6}
['game', 'player', 'location', 'time', 'date', 'surprise', 'special']
__________________________________________________________________________________________
['Available Universal Dices Game', 'FTROE875574', '"Lakeview School, Kingsmere Boulevard, Saskatoon, Saskatchewan , Canada"', '15h18', '', 'perturbating at start', '']
['Available', 'LOI4531', '"Perth, Australia"', '08h13', 'Available', '', '']
['Available', 'PLLI874', '"Chelsea, London, England"', '20h35', '', '', '']
['Available', 'YTR44', '"Paris, France"', '02h24', 'Available', '', 'midnight happening']
['Available', 'FT875', '', '11h22', '', 'hretyuuhuhu', '']
['Available', 'LOI4531', '"Damas,Syria"', 'unavailable', 'Available', '', '']
['Available Dices', 'PuLuLu874', '" Westminster, London, England "', '20h01', '', 'GARAMANANATALA Tower', '']
['Available', 'YTR44', '"Madrid, Spain"', '12h33', 'Available', '', 'overnight']
我写了上面的代码思考到一个无法完全读取的几GB的巨大文件:这样一个非常大的文件的处理必须在块之后完成。这就是为什么有指示:
while chunk:
chunk = f.read(120)
ss = ''.join((prec,chunk))
ecr.append('\n\n------------------------------------------------------------\nss == %r' %ss)
mat_dic = None
for mat_dic in dicreg.finditer(ss):
............
...............
if mat_dic:
prec = ss[mat_dic.end():]
else:
prec += chunk
但是,显然,如果文件不是太大,因此可以一次性读取,代码可以简化:
import re
dicreg = re.compile('(?<=\{)[^}]*}')
kvregx = re.compile('[ \r\n]*'
'(" *)?((location)|[^:]+?)(?(1) *")'
'[ \r\n]*'
':'
'[ \r\n]*'
'(?(3)|(" *)?)([^:]*?)(?(4) *")'
'[ \r\n]*(?:,(?=[^,]+?:)|\})')
checking_dict = {}
checking_list = []
filename = 'zzz.txt'
with open(filename) as f:
content = f.read()
######## First part: to gather all the keys in all the dictionaries
ecr = []
for mat_dic in dicreg.finditer(content):
ecr.append('\nmmmmmmm dictionary found in ss mmmmmmmmmmmmmm')
for mat_kv in kvregx.finditer(mat_dic.group()):
k,v = mat_kv.group(2,5)
ecr.append('%s : %s' % (k,v))
if k in checking_list:
checking_dict[k] += 1
else:
checking_list.append(k)
checking_dict[k] = 1
print '\n'.join(ecr)
print '\n\n\nchecking_dict == %s\n\nchecking_list == %s' %(checking_dict,checking_list)
######## The keys are sorted in order that the less frequent ones are at the end
checking_list.sort(key=lambda k: checking_dict[k], reverse=True)
posis = dict((k,i) for i,k in enumerate(checking_list))
print '\nchecking_list sorted == %s\n\nposis == %s' % (checking_list,posis)
######## Now, the file is read again to build a list of rows
base = [ '' for i in xrange(len(checking_list))]
rows = []
for mat_dic in dicreg.finditer(content):
li = base[:]
for mat_kv in kvregx.finditer(mat_dic.group()):
k,v = mat_kv.group(2,5)
li[posis[k]] = v
rows.append(li)
print '\n\n%s\n%s' % (checking_list,30*'___')
print '\n'.join(str(li) for li in rows)
答案 2 :(得分:0)
import re
text = """
{game:Available,player:Available,location:"Chelsea, London, England",time:Available}
{"game":"Available","player":"Available","location":"Chelsea, London, England","time":"Available","date":"Available"}
"""
dicts = re.findall(r"{.+?}", text) # Split the dicts
for dict_ in dicts:
dict_ = dict(re.findall(r'(\w+|".*?"):(\w+|".*?")', dict_)) # Get the elements
print dict_
>>>{'player': 'Available', 'game': 'Available', 'location': '"Chelsea, London, England"', 'time': 'Available'}
>>>{'"game"': '"Available"', '"time"': '"Available"', '"player"': '"Available"', '"date"': '"Available"', '"location"': '"Chelsea, London, England"'}
答案 3 :(得分:0)
希望这种pyparsing解决方案随着时间的推移更容易遵循和维护:
data = """\
{game:Available,player:Available,location:"Chelsea, London, England",time:Available}
{"game":"Available","player":"Available","location":"Chelsea, London, England","time":"Available","date":"Available"}"""
from pyparsing import Suppress, Word, alphas, alphanums, QuotedString, Group, Dict, delimitedList
LBRACE,RBRACE,COLON = map(Suppress, "{}:")
key = QuotedString('"') | Word(alphas)
value = QuotedString('"') | Word(alphanums+"_")
keyvalue = Group(key + COLON + value)
dictExpr = LBRACE + Dict(delimitedList(keyvalue)) + RBRACE
for d in dictExpr.searchString(data):
print d.asDict()
打印:
{'player': 'Available', 'game': 'Available', 'location': 'Chelsea, London, England', 'time': 'Available'}
{'date': 'Available', 'player': 'Available', 'game': 'Available', 'location': 'Chelsea, London, England', 'time': 'Available'}