Question

我试图逐行读取大文件，同时还要写入大文件，我想知道这样做的“最佳”方法。

我找到了reading a large file line by line的Stack Overflow帖子，并想知道将写入内容也合并到文件中的正确方法。有什么比嵌套第二个a + geom_errorbar(data=newdf,aes(x=xcol,ymin=ymincol,ymax=ymaxcol))

更好的了

我目前拥有的东西：

with open

Answer 1

您不必嵌套with语句。一个with语句可以使用多个上下文管理器。

with open(args.inPath + file, "r") as fpIn, open(args.outPath + file, "w") as fpOut:
    for line in fpIn:
       if re.match(some match): canWrite = True
       if re.match(some match 2): break
       if canWrite: fpOut.write(line)

比较干净。

Answer 2

yield是您最好的朋友：通过Lazy Method for Reading Big File in Python?

def read_in_chunks(file_object, chunk_size=1024):
  """Lazy function (generator) to read a file piece by piece.
  Default chunk size: 1k."""
  while True:
    data = file_object.read(chunk_size)
    if not data:
      break
    yield data

f = open(args.inPath + file, "r")

with open(args.outPath + file, "a") as fpOut:
  for chunk in read_in_chunks(f):
    if re.match(some match): canWrite = True
    if re.match(some match 2): break
    if canWrite: fpOut.write(chunk)

另请参见：https://www.pythoncentral.io/python-generators-and-yield-keyword/，https://www.geeksforgeeks.org/use-yield-keyword-instead-return-keyword-python/

这也会减轻您的内存占用。

用Python逐行读取一个大文件，同时还写入另一个大文件

2 个答案: