有效地找到文本文件中的最后一行

Question

我需要从一些非常大（几百兆字节）的文本文件中提取最后一行来获取某些数据。目前，我使用python循环遍历所有行，直到文件为空，然后我处理返回的最后一行，但我确信有一种更有效的方法来执行此操作。

使用python检索文本文件的最后一行的最佳方法是什么？

Answer 1

不是直接的方式，但可能比简单的Python实现快得多：

line = subprocess.check_output(['tail', '-1', filename])

Answer 2

with open('output.txt', 'r') as f:
    lines = f.read().splitlines()
    last_line = lines[-1]
    print last_line

Answer 3

使用带有负偏移的文件seek方法和whence=os.SEEK_END来读取文件末尾的块。在该块中搜索最后一行结束字符并获取其后的所有字符。如果没有行结束，则备份更远并重复该过程。

def last_line(in_file, block_size=1024, ignore_ending_newline=False):
    suffix = ""
    in_file.seek(0, os.SEEK_END)
    in_file_length = in_file.tell()
    seek_offset = 0

    while(-seek_offset < in_file_length):
        # Read from end.
        seek_offset -= block_size
        if -seek_offset > in_file_length:
            # Limit if we ran out of file (can't seek backward from start).
            block_size -= -seek_offset - in_file_length
            if block_size == 0:
                break
            seek_offset = -in_file_length
        in_file.seek(seek_offset, os.SEEK_END)
        buf = in_file.read(block_size)

        # Search for line end.
        if ignore_ending_newline and seek_offset == -block_size and buf[-1] == '\n':
            buf = buf[:-1]
        pos = buf.rfind('\n')
        if pos != -1:
            # Found line end.
            return buf[pos+1:] + suffix

        suffix = buf + suffix

    # One-line file.
    return suffix

请注意，这不适用于不支持seek的内容，例如stdin或套接字。在这些情况下，你会被困在阅读整个事情（就像tail命令那样）。

Answer 4

如果你知道一条线的最大长度，你可以

def getLastLine(fname, maxLineLength=80):
    fp=file(fname, "rb")
    fp.seek(-maxLineLength-1, 2) # 2 means "from the end of the file"
    return fp.readlines()[-1]

这适用于我的Windows机器。但是我不知道如果你以二进制模式打开文本文件会在其他平台上发生什么。如果要使用seek（），则需要二进制模式。

Answer 5

寻找文件的末尾减去100个字节左右。读取并搜索换行符。如果这里没有换行符，请再追回100个字节左右。泡沫，冲洗，重复。最终你会找到换行符。最后一行在该换行符后立即开始。

最好的情况是你只读一个100字节。

Answer 6

如果您可以选择合理的最大行长度，您可以在开始阅读之前寻找接近文件的末尾。

myfile.seek(-max_line_length, os.SEEK_END)
line = myfile.readlines()[-1]

Answer 7

您可以将文件加载到mmap，然后使用mmap.rfind（string [，start [，end]]）查找文件中的第二个最后一个EOL字符吗？寻找文件中的那一点应该指向我想到的最后一行。

Answer 8

这里的效率低下并不是因为Python，而是因为文件的读取方式。找到最后一行的唯一方法是读取文件并找到行结尾。但是，搜索操作可用于跳过文件中的任何字节偏移。因此，您可以非常接近文件的末尾，并根据需要获取越来越大的块，直到找到最后一行结尾：

from os import SEEK_END

def get_last_line(file):
  CHUNK_SIZE = 1024 # Would be good to make this the chunk size of the filesystem

  last_line = ""

  while True:
    # We grab chunks from the end of the file towards the beginning until we 
    # get a new line
    file.seek(-len(last_line) - CHUNK_SIZE, SEEK_END)
    chunk = file.read(CHUNK_SIZE)

    if not chunk:
      # The whole file is one big line
      return last_line

    if not last_line and chunk.endswith('\n'):
      # Ignore the trailing newline at the end of the file (but include it 
      # in the output).
      last_line = '\n'
      chunk = chunk[:-1]

    nl_pos = chunk.rfind('\n')
    # What's being searched for will have to be modified if you are searching
    # files with non-unix line endings.

    last_line = chunk[nl_pos + 1:] + last_line

    if nl_pos == -1:
      # The whole chunk is part of the last line.
      continue

    return last_line

Answer 9

这是一个略有不同的解决方案。而不是多行，我专注于最后一行，而不是一个恒定的块大小，我有一个动态（加倍）块大小。有关详细信息，请参阅评论。

# Get last line of a text file using seek method.  Works with non-constant block size.  
# IDK if that speed things up, but it's good enough for us, 
# especially with constant line lengths in the file (provided by len_guess), 
# in which case the block size doubling is not performed much if at all.  Currently,
# we're using this on a textfile format with constant line lengths.
# Requires that the file is opened up in binary mode.  No nonzero end-rel seeks in text mode.
REL_FILE_END = 2
def lastTextFileLine(file, len_guess=1):
    file.seek(-1, REL_FILE_END)      # 1 => go back to position 0;  -1 => 1 char back from end of file
    text = file.read(1)
    tot_sz = 1              # store total size so we know where to seek to next rel file end
    if text != b'\n':        # if newline is the last character, we want the text right before it
        file.seek(0, REL_FILE_END)    # else, consider the text all the way at the end (after last newline)
        tot_sz = 0
    blocks = []           # For storing succesive search blocks, so that we don't end up searching in the already searched
    j = file.tell()          # j = end pos
    not_done = True
    block_sz = len_guess
    while not_done:
        if j < block_sz:   # in case our block doubling takes us past the start of the file (here j also = length of file remainder)
            block_sz = j
            not_done = False
        tot_sz += block_sz
        file.seek(-tot_sz, REL_FILE_END)         # Yes, seek() works with negative numbers for seeking backward from file end
        text = file.read(block_sz)
        i = text.rfind(b'\n')
        if i != -1:
            text = text[i+1:].join(reversed(blocks))
            return str(text)
        else:
            blocks.append(text)
            block_sz <<= 1    # double block size (converge with open ended binary search-like strategy)
            j = j - block_sz      # if this doesn't work, try using tmp j1 = file.tell() above
    return str(b''.join(reversed(blocks)))      # if newline was never found, return everything read

理想情况下，您将其包装在LastTextFileLine类中，并跟踪行长度的移动平均值。这可能会给你一个好的len_guess。

Answer 10

lines = file.readlines()
fileHandle.close()
last_line = lines[-1]

Answer 11

#!/usr/bin/python

count = 0

f = open('last_line1','r')

for line in f.readlines():

    line = line.strip()

    count = count + 1

    print line

print count

f.close()

count1 = 0

h = open('last_line1','r')

for line in h.readlines():

    line = line.strip()

    count1 = count1 + 1

    if count1 == count:

       print line         #-------------------- this is the last line

h.close()