Question

在脚本中，我正在为文件写行，但有些行可能是重复的。所以我创建了一个临时的cStringIO类文件对象，我称之为“中间文件”。我首先将行写入中间文件，删除重复项，然后写入真实文件。

所以我编写了一个简单的for循环来迭代我的中间文件中的每一行并删除任何重复项。

def remove_duplicates(f_temp, dir_out):  # f_temp is the cStringIO object.
    """Function to remove duplicates from the intermediate file and write to physical file."""
    lines_seen = set()  # Define a set to hold lines already seen.
    f_out = define_outputs(dir_out)  # Create the real output file by calling function "define_outputs". Note: This function is not shown in my pasted code.

    cStringIO.OutputType.getvalue(f_temp)  # From: https://stackoverflow.com/a/40553378/8117081

    for line in f_temp:  # Iterate through the cStringIO file-like object.
        line = compute_md5(line)  # Function to compute the MD5 hash of each line. Note: This function is not shown in my pasted code.
        if line not in lines_seen:  # Not a duplicate line (based on MD5 hash, which is supposed to save memory).
            f_out.write(line)
            lines_seen.add(line)
    f_out.close()

我的问题是for循环永远不会被执行。我可以通过在调试器中输入断点来验证这一点;代码的那一行只是被跳过而且函数退出。我甚至阅读了this answer from this thread并插入了代码cStringIO.OutputType.getvalue(f_temp)，但这并没有解决我的问题。

我迷失了为什么我无法阅读和遍历我的文件类对象。

Answer 1

您引用的答案有点不完整。它告诉我们如何将cStringIO缓冲区作为字符串，但是你必须对该字符串做一些事情。你可以这样做：

def remove_duplicates(f_temp, dir_out):  # f_temp is the cStringIO object.
    """Function to remove duplicates from the intermediate file and write to physical file."""
    lines_seen = set()  # Define a set to hold lines already seen.
    f_out = define_outputs(dir_out)  # Create the real output file by calling function "define_outputs". Note: This function is not shown in my pasted code.

    # contents = cStringIO.OutputType.getvalue(f_temp)  # From: https://stackoverflow.com/a/40553378/8117081
    contents = f_temp.getvalue()     # simpler approach
    contents = contents.strip('\n')  # remove final newline to avoid adding an extra row
    lines = contents.split('\n')     # convert to iterable

    for line in lines:  # Iterate through the list of lines.
        line = compute_md5(line)  # Function to compute the MD5 hash of each line. Note: This function is not shown in my pasted code.
        if line not in lines_seen:  # Not a duplicate line (based on MD5 hash, which is supposed to save memory).
            f_out.write(line + '\n')
            lines_seen.add(line)
    f_out.close()

但最好在f_temp“文件句柄”上使用普通的IO操作，如下所示：

def remove_duplicates(f_temp, dir_out):  # f_temp is the cStringIO object.
    """Function to remove duplicates from the intermediate file and write to physical file."""
    lines_seen = set()  # Define a set to hold lines already seen.
    f_out = define_outputs(dir_out)  # Create the real output file by calling function "define_outputs". Note: This function is not shown in my pasted code.

    # move f_temp's pointer back to the start of the file, to allow reading
    f_temp.seek(0)

    for line in f_temp:  # Iterate through the cStringIO file-like object.
        line = compute_md5(line)  # Function to compute the MD5 hash of each line. Note: This function is not shown in my pasted code.
        if line not in lines_seen:  # Not a duplicate line (based on MD5 hash, which is supposed to save memory).
            f_out.write(line)
            lines_seen.add(line)
    f_out.close()

这是一个测试（有一个）：

import cStringIO, os

def define_outputs(dir_out):
    return open('/tmp/test.txt', 'w') 

def compute_md5(line):
    return line

f = cStringIO.StringIO()
f.write('string 1\n')
f.write('string 2\n')
f.write('string 1\n')
f.write('string 2\n')
f.write('string 3\n')

remove_duplicates(f, 'tmp')
with open('/tmp/test.txt', 'r') as f:
    print(str([row for row in f]))
# ['string 1\n', 'string 2\n', 'string 3\n']

无法迭代cStringIO

1 个答案: