I have an application which monitors some data from web and write files almost every second. This application runs 24/7, but sometimes I have to turn it off. I use a keyboard interrupt exception to stop the app when I want, but once in a while the application does not finish the file writing and I got a corrupted file. What make me fix the file so as to get the app working properly when I turn it on again.
This is a piece of the code:
while True:
try:
conns = [http.client.HTTPSConnection(C.REQUEST_HOST) for k in range(C.num_coins)]
for k in range(C.num_coins):
conns[k].request("GET", C.REQUEST_PATH_TICKER.format(C.coin_types[k]))
responses = [conns[k].getresponse().read() for k in range(C.num_coins)]
responses_json_new = [
json.loads(responses[k].decode(), object_pairs_hook=OrderedDict)
for k in range(C.num_coins)
]
responses_json_new = [responses_json_new[k]["ticker"] for k in range(C.num_coins)]
for k in range(C.num_coins):
if responses_json_new[k]["last"] != responses_json_old[k]["last"]:
date_ticker = datetime.datetime.fromtimestamp(
int(responses_json_new[k]["date"])
).strftime("%Y-%m-%d")
with open(files_path[k] + date_ticker + ".json", "a") as f:
json.dump(responses_json_new[k], f)
f.write("\n")
responses_json_old[k] = responses_json_new[k]
print(
out.format(
C.coin_types[k],
float(responses_json_new[k]["vol"]),
float(responses_json_new[k]["last"]),
datetime.datetime.fromtimestamp(
responses_json_new[k]["date"]
).strftime("%Y-%m-%d %H:%M:%S"),
)
)
currentDate, files_path = updateFilePath(currentDate, files_path)
time.sleep(1)
except KeyboardInterrupt:
print("\nInterrupted!\n")
for k in range(C.num_coins):
if conns[k]:
conns[k].close()
sys.exit()
except Exception as e:
print("\nERROR:", sys.exc_info()[0])
ut.exception_log(e, traceback.format_exc())
ut.close_waiting(conns, wait=30)
continue
finally:
for k in range(C.num_coins):
if conns[k]:
conns[k].close()
I need some suggestions on how can I handle this.
Thanks all!
答案 0 :(得分:0)
.dump()
(显然)可以适当地将输出分成几块。
您首先要使用.dumps()
编码为JSON字符串,然后使用.write()
编码为文件。
为证明输出可以写成多个块,让我们在Docker容器中启动strace
,看看会得到write()
d。
(host) $ docker run -it ubuntu:18.04 bash
# apt install strace python3
# cat > ex1.py
import json, sys
payload = [[x] * x for x in range(5, 10)] * 250
json.dump(payload, sys.stdout)
# strace python3 ex1.py >/dev/null
(uninteresting lines redacted)
write(1, "[[5, 5, 5, 5, 5], [6, 6, 6, 6, 6"..., 8194) = 8194
write(1, ", 6, 6], [7, 7, 7, 7, 7, 7, 7], "..., 8194) = 8194
write(1, "], [8, 8, 8, 8, 8, 8, 8, 8], [9,"..., 8194) = 8194
write(1, "[9, 9, 9, 9, 9, 9, 9, 9, 9], [5,"..., 4168) = 4168
# cat > ex2.py
import json, sys
payload = [[x] * x for x in range(5, 10)] * 250
pl2 = json.dumps(payload)
sys.stdout.write(pl2)
# strace python3 ex2.py
(uninteresting lines redacted)
write(1, "[[5, 5, 5, 5, 5], [6, 6, 6, 6, 6"..., 28750) = 28750