使用python每x行处理一个文件块

时间:2011-07-19 17:29:14

标签: python text-extraction

我在这里要做的是从文件 DATA.txt 读取每行间隔 y z 行,然后执行功能找到就行了。即我想跳过第一个 y 行;阅读下一个 z 行;在刚刚读入的那些行上执行 find 功能;跳过下一个 y 行;并重复文件的长度(传入 sys.argv [1] )。

我在这里给了我一些变量的空行,我不知道为什么。如果需要,我可以提供查找功能,但我认为这样更简单。

如果有人想要提出完全不同的做法,只要我了解正在发生的事情,我就会像修复现有代码一样快乐。

编辑:我错过了几个parens但添加它们并没有解决问题。

import sys
import operator
import linecache
def find(arg)
    ...
x=0
while x<int(sys.argv[1]):
   x+=1 
   if mod(x, y)==0:
       for i in range(x,x+z):
           block=linecache.getline('DATA.txt', i)
           g = open('tmp','a+')
           g.write(block)
           linecache.clearcache()
           lines=g.read()
           find(lines)
           g.close()
   else:
       pass
g.close()
f.close()

4 个答案:

答案 0 :(得分:2)

编辑:尝试以下操作,我想我现在对您要做的事情有了更好的了解。

g = open('tmp','a+')
while x<int(sys.argv[1]):
   x+=1 
   if mod(x, y)==0:
       curr = g.tell()
       for i in range(x,x+z):
           block=linecache.getline('DATA.txt', i)
           g.write(block)
           linecache.clearcache()
       g.seek(curr)
       lines = g.read()
       find(lines)
   else:
       pass
g.close()

答案 1 :(得分:1)

Maimon,您的初始代码与索引有关。安德鲁的代码也是错误的,因为他把你的代码作为一个开始。

查看安德鲁代码的结果,其中我删除了关于 g 的行:

import sys
import operator
import linecache

x=0
y=7  # to skip
z=3  # to print

#g = open('tmp','a+')
while x<23:
    x+=1
    print 'x==',x
    if operator.mod(x, y)==0:
        #curr = g.tell()
        for i in range(x,x+z):
            block=linecache.getline('poem.txt', i)
            print 'block==',repr(block)
            #g.write(block)
            linecache.clearcache()
            #g.seek(curr)
            #lines = g.read()
            #find(lines)

    else:
        pass

#g.close()

应用于名为'poem.txt'的文件,其中包含24行:

1 In such a night, when every louder wind
2 Is to its distant cavern safe confined;
3 And only gentle Zephyr fans his wings,
4 And lonely Philomel, still waking, sings;
5 Or from some tree, famed for the owl's delight,
6 She, hollowing clear, directs the wand'rer right:
7 In such a night, when passing clouds give place,
8 Or thinly veil the heav'ns' mysterious face;
9 When in some river, overhung with green,
10 The waving moon and trembling leaves are seen;
11 When freshened grass now bears itself upright,
12 And makes cool banks to pleasing rest invite,
13 Whence springs the woodbind, and the bramble-rose,
14 And where the sleepy cowslip sheltered grows;
15 Whilst now a paler hue the foxglove takes,
16 Yet checkers still with red the dusky brakes
17 When scattered glow-worms, but in twilight fine,
18 Shew trivial beauties watch their hour to shine;
19 Whilst Salisb'ry stands the test of every light,
20 In perfect charms, and perfect virtue bright:
21 When odors, which declined repelling day,
22 Through temp'rate air uninterrupted stray;
23 When darkened groves their softest shadows wear,
24 And falling waters we distinctly hear;

结果是:

x== 1
x== 2
x== 3
x== 4
x== 5
x== 6
x== 7
block== '7 In such a night, when passing clouds give place,\n'
block== "8 Or thinly veil the heav'ns' mysterious face;\n"
block== '9 When in some river, overhung with green,\n'
x== 8
x== 9
x== 10
x== 11
x== 12
x== 13
x== 14
block== '14 And where the sleepy cowslip sheltered grows;\n'
block== '15 Whilst now a paler hue the foxglove takes,\n'
block== '16 Yet checkers still with red the dusky brakes\n'
x== 15
x== 16
x== 17
x== 18
x== 19
x== 20
x== 21
block== '21 When odors, which declined repelling day,\n'
block== "22 Through temp'rate air uninterrupted stray;\n"
block== '23 When darkened groves their softest shadows wear,\n'
x== 22
x== 23
x== 24
x== 25

我选择y = 7作为要跳过的行数,但是第7行被打印出来。

此外,计数继续8,9,10 ...打印3行7-8-9(选择z = 3)而不是继续10,11,12 ...然后下一个印刷的3行是14-15-16,而它应该是7 + 3第一行之后的行,即11-12-13行

事实上,如果你想跳过7行,然后打印3行,打印的行必须是:
8-9-10
18-19-20
28-29-30

我是对的吗?

编辑1

我的解决方案是:

def chunk_reading(filepath,y,z,x=0):
    # x : number of lines to skip before the periodical treatment
    # y : number of lines to periodically skip
    # z : number of lines to periodically print
    with open('poem.txt') as f:
        try:
            for sk in xrange(x):
                f.next()
            while True:
                try:
                    for i in xrange(y):
                        print 'i==',i
                        f.next()
                    for j in xrange(z):
                        print 'j==',j
                        print repr(f.next())
                except StopIteration:
                    break
        except StopIteration:
            print 'Not enough lines before the lines to print'


chunk_reading('poem.txt',7,3)

产生

i== 0
i== 1
i== 2
i== 3
i== 4
i== 5
i== 6
j== 0
"8 Or thinly veil the heav'ns' mysterious face;\n"
j== 1
'9 When in some river, overhung with green,\n'
j== 2
'10 The waving moon and trembling leaves are seen;\n'
i== 0
i== 1
i== 2
i== 3
i== 4
i== 5
i== 6
j== 0
'18 Shew trivial beauties watch their hour to shine;\n'
j== 1
"19 Whilst Salisb'ry stands the test of every light,\n"
j== 2
'20 In perfect charms, and perfect virtue bright:\n'
i== 0
i== 1
i== 2
i== 3
i== 4

编辑2

上述解决方案甚至可用于无法记录在RAM中的大文件。

以下内容适用于尺寸有限的文件:

def slice_reading(filepath,y,z,x=0):
    # x : number of lines to skip before the periodical treatment
    # y : number of lines to periodically skip
    # z : number of lines to periodically print
    with open('poem.txt') as f:
        lines = f.readlines()
        lgth = len(lines)

    if lgth > x+y:
        for a in xrange(x+y,lgth,y+z):
            print lines[a:a+z]
    else:
        print 'Not enough lines before lines to print'


slice_reading('poem.txt',7,3,5)

结果

['13 Whence springs the woodbind, and the bramble-rose,\n', '14 And where the sleepy cowslip sheltered grows;\n', '15 Whilst now a paler hue the foxglove takes,\n']
['23 When darkened groves their softest shadows wear,\n', '24 And falling waters we distinctly hear;']

答案 2 :(得分:0)

我认为您的问题可能就是lines=g.read行。它应该是lines=g.read()

答案 3 :(得分:0)

在“不同方法”类别中,我提供此(行号显然仅用于显示):

  1 """
  2 Reading line lines from DATA.txt, first skip 3 lines, then print 2 lines,
  3 then skip 3 more lines, etc.
  4 """
  5 
  6 def my_print(l):
  7     if (my_print.skip_counter > 0):
  8         my_print.skip_counter -= 1
  9     else:
 10         if (my_print.print_counter > 0):
 11             my_print.print_counter -= 1
 12             print l,
 13         else:
 14             my_print.skip_counter = my_print.skip_size
 15             my_print.print_counter = my_print.print_size
 16             my_print(l)
 17 
 18 my_print.skip_size = 3
 19 my_print.skip_counter = my_print.skip_size
 20 
 21 my_print.print_size = 2
 22 my_print.print_counter = my_print.print_size
 23 
 24 data = open('DATA.txt')
 25 for line in data:
 26     my_print(line)

改进这个的第一种方法是用一个类包装my_print()(用你的x和y作为成员变量)。如果你想要一些真正“pythonic”的东西,那么你可以充分利用发电机。