我有3个文件：2个.gz文件和1个.log文件。这些文件很大。下面有我的原始数据的样本副本。我想提取与过去24小时相对应的条目。
a.log.1.gz
2018/03/25-00:08:48.638553 508 7FF4A8F3D704 snononsonfvnosnovoosr
2018/03/25-10:08:48.985053 346K 7FE9D2D51706 ahelooa afoaona woom
2018/03/25-20:08:50.486601 1.5M 7FE9D3D41706 qojfcmqcacaeia
2018/03/25-24:08:50.980519 16K 7FE9BD1AF707 user: number is 93823004
2018/03/26-00:08:50.981908 1389 7FE9BDC2B707 user 7fb31ecfa700
2018/03/26-10:08:51.066967 0 7FE9BDC91700 Exit Status = 0x0
2018/03/26-15:08:51.066968 1 7FE9BDC91700 std:ZMD:
a.log.2.gz
2018/03/26-20:08:48.638553 508 7FF4A8F3D704 snononsonfvnosnovoosr
2018/03/26-24:08:48.985053 346K 7FE9D2D51706 ahelooa afoaona woom
2018/03/27-00:08:50.486601 1.5M 7FE9D3D41706 qojfcmqcacaeia
2018/03/27-10:08:50.980519 16K 7FE9BD1AF707 user: number is 93823004
2018/03/27-20:08:50.981908 1389 7FE9BDC2B707 user 7fb31ecfa700
2018/03/27-24:08:51.066967 0 7FE9BDC91700 Exit Status = 0x0
2018/03/28-00:08:51.066968 1 7FE9BDC91700 std:ZMD:
a.log
2018/03/28-10:08:48.638553 508 7FF4A8F3D704 snononsonfvnosnovoosr
2018/03/28-20:08:48.985053 346K 7FE9D2D51706 ahelooa afoaona woom
** Desired Result**
result.txt
2018/03/27-20:08:50.981908 1389 7FE9BDC2B707 user 7fb31ecfa700
2018/03/27-24:08:51.066967 0 7FE9BDC91700 Exit Status = 0x0
2018/03/28-00:08:51.066968 1 7FE9BDC91700 std:ZMD:
2018/03/28-10:08:48.638553 508 7FF4A8F3D704 snononsonfvnosnovoosr
2018/03/28-20:08:48.985053 346K 7FE9D2D51706 ahelooa afoaona woom
我不确定如何获取过去24小时内的条目。
答案 0 :(得分：1)
类似的事情应该起作用。
from datetime import datetime, timedelta
import glob
import gzip
from pathlib import Path
import shutil
def open_file(path):
if Path(path).suffix == '.gz':
return gzip.open(path, mode='rt', encoding='utf-8')
else:
return open(path, encoding='utf-8')
def parsed_entries(lines):
for line in lines:
yield line.split(' ', maxsplit=1)
def earlier():
return (datetime.now() - timedelta(hours=24)).strftime('%Y/%m/%d-%H:%M:%S')
def get_files():
return ['a.log'] + list(reversed(sorted(glob.glob('a.log.*'))))
output = open('output.log', 'w', encoding='utf-8')
files = get_files()
cutoff = earlier()
for i, path in enumerate(files):
with open_file(path) as f:
lines = parsed_entries(f)
# Assumes that your files are not empty
date, line = next(lines)
if cutoff <= date:
# Skip files that can just be appended to the output later
continue
for date, line in lines:
if cutoff <= date:
# We've reached the first entry of our file that should be
# included
output.write(line)
break
# Copies from the current position to the end of the file
shutil.copyfileobj(f, output)
break
else:
# In case ALL the files are within the last 24 hours
i = len(files)
for path in reversed(files[:i]):
with open_file(path) as f:
# Assumes that your files have trailing newlines.
shutil.copyfileobj(f, output)
# Cleanup, it would get closed anyway when garbage collected or process exits.
output.close()
然后，如果我们制作一些测试日志文件：
#!/bin/sh
echo "2019/01/15-00:00:00.000000 hi" > a.log.1
echo "2019/01/31-00:00:00.000000 hi2" > a.log.2
echo "2019/01/31-19:00:00.000000 hi3" > a.log
gzip a.log.1 a.log.2
并运行我们的脚本，它会输出预期的结果（对于该时间点）
2019/01/31-00:00:00.000000 hi2
2019/01/31-19:00:00.000000 hi3
答案 1 :(得分：0)
处理日志文件通常涉及大量数据，因此不希望按升序读取并且每次都读取所有内容，因为这会浪费大量资源。
立即想到的最快完成目标的方法（肯定会存在更好的方法）是一个非常简单的随机搜索：我们以相反的顺序搜索日志文件，因此从最新的开始。您无需随意访问所有行，只需随意选择一些stepsize，而仅查看每个stepsize的 some 行。这样，您可以在很短的时间内搜索千兆字节的数据。
此外，此方法不需要要求将文件的每一行存储在内存中，但只存储一些行和最终结果。
当a.log是当前日志文件时，我们从此处开始搜索：
with open("a.log", "rb+") as fh:
由于我们只对最近24小时感兴趣，因此我们首先跳到结尾并保存时间戳以作为格式字符串搜索：
timestamp = datetime.datetime.now() - datetime.timedelta(days=1) # last 24h
# jump to logfile's end
fh.seek(0, 2) # <-- '2': search relative to file's end
index = fh.tell() # current position in file; here: logfile's *last* byte
现在我们可以开始随机搜索了。您的行平均似乎长约65个字符，因此我们将其移动了倍数。
average_line_length = 65
stepsize = 1000
while True:
# we move a step back
fh.seek(index - average_line_length * stepsize, 2)
# save our current position in file
index = fh.tell()
# we try to read a "line" (multiply avg. line length times a number
# large enough to cover even large lines. Ignore largest lines here,
# since this is an edge cases ruining our runtime. We rather skip
# one iteration of the loop then)
r = fh.read(average_line_length * 10)
# our results now contains (on average) multiple lines, so we
# split first
lines = r.split(b"\n")
# now we check for our timestring
for l in lines:
# your timestamps are formatted like '2018/03/28-20:08:48.985053'
# I ignore minutes, seconds, ... here, just for the sake of simplicity
timestr = l.split(b":") # this gives us b'2018/03/28-20' in timestr[0]
# next we convert this to a datetime
found_time = datetime.datetime.strptime(timestr[0], "%Y/%m/%d-%H")
# finally, we compare if the found time is not inside our 24hour margin
if found_time < timestamp:
break
有了这段代码，只要我们过去24小时之内，就只会在每个stepsize（这里：1000行）中搜索几行。离开24小时后，我们知道，至多我们完全stepsize * average_line_length在文件中太远了。
然后过滤此“过分”变得非常容易：
# read in file's contents from current position to end
contents = fh.read()
# split for lines
lines_of_contents = contents.split(b"\n")
# helper function for removing all lines older than 24 hours
def check_line(line):
# split to extract datestr
tstr = line.split(b":")
# convert this to a datetime
ftime = datetime.datetime.strptime(tstr[0], "%Y/%m/%d-%H")
return ftime > timestamp
# remove all lines that are older than 24 hours
final_result = filter(check_line, lines_of_contents)
由于contents覆盖了文件的所有其余内容（以及lines所有行，在换行符contents处简单地\n拆分了），因此我们可以轻松地使用{ {1}}得到我们想要的结果。
filter中的每一行都将被馈送到lines，如果该行的时间为check_line并且True是我们的datetime对象，则该行将返回> timestamp timestamp。这意味着now - 1day将为所有早于check_line的行返回False，而timestamp将删除这些行。
显然，这远非最佳，但它易于理解，并且可以轻松扩展到几分钟，几秒钟，...
另外，覆盖多个文件也很容易：您只需要filter来查找所有可能的文件，从最新文件开始并添加另一个循环：您将搜索这些文件，直到第一个while循环失败时间，然后中断并读取当前文件中的所有剩余内容+之前访问过的所有文件中的所有内容。
大概是这样的：
glob.glob这样，如果所有行的使用时间都不超过24小时，则只需存储日志文件的所有行。如果循环在某个时间点中断，即我们找到了一个日志文件，并且确切的行龄超过24小时，则将final_lines = lst()
for file in logfiles:
# our while-loop
while True:
...
# if while-loop did not break all of the current logfile's content is
# <24 hours of age
with open(file, "rb+") as fh:
final_lines.extend(fh.readlines())
扩展final_lines，因为这将只覆盖年龄小于24小时的行。 / p>