1）

Question

我想找到其全名（相对，虽然绝对也很好）的文件匹配给定的正则表达式（即，像glob模块，但是对于正则表达式匹配而不是shell通配符匹配）。使用find，可以做，例如：

find . -regex ./foo/\w+/bar/[0-9]+-\w+.dat

当然，我可以通过find或os.system(...)使用os.exec*(...)，但我正在寻找纯Python解决方案。以下代码将os.walk(...)与re模块正则表达式相结合是一种简单的Python解决方案。（它不健壮并且错过了许多（不那么角落）的角落情况，但是对于我的一次性目的来说已经足够了，为一次性数据库插入定位特定数据文件。）

import os
import re

def find(regex, top='.'):
    matcher = re.compile(regex)
    for dirpath, dirnames, filenames in os.walk(top):
        for f in filenames:
            f = os.path.relpath(os.path.join(dirpath, f), top)
            if matcher.match(f):
                yield f

if __name__=="__main__":
    top = "."
    regex = "foo/\w+/bar/\d+-\w+.dat"
    for f in find(regex, top):
        print f

但这效率低下。其内容与正则表达式不匹配的子树（例如，./foo/\w+/baz/，以继续上面的示例）被不必要地走了。理想情况下，这些子树应该从步行中删除;不应遍历路径名不是正则表达式的部分匹配的任何子目录。（我猜想GNU find实现了这样的优化，但我还没有通过测试或源代码细读来证实这一点。）

有没有人知道基于强大的基于正则表达式find的Python实现，理想情况下是使用子树修剪优化？我希望我只是错过了os.path模块或某个第三方模块中的方法。

Answer 1

来自help(os.walk)：

当topdown为true时，调用者可以就地修改dirnames列表（例如，通过del或slice赋值），walk将只递归到名称保留在dirnames中的子目录;这可以用来修剪搜索...

因此，一旦确定子目录（在dirnames中列出）不允许，就应该从dirnames中删除它。这将产生您正在寻找的子树修剪。（请务必先从尾端del dirnames import os import re def prune(regex,top='.'): sep=os.path.sep matcher = re.compile(regex) pieces=regex.split(sep) partial_matchers = map( re.compile, (sep.join(pieces[:i+1]) for i in range(len(pieces)))) for root, dirs, files in os.walk(top,topdown=True): for i in reversed(range(len(dirs))): dirname=os.path.relpath(os.path.join(root,dirs[i]), top) dirlevel=dirname.count(sep) # print(dirname,dirlevel,sep.join(pieces[:dirlevel+1])) if not partial_matchers[dirlevel].match(dirname): print('pruning {0}'.format( os.path.relpath(os.path.join(root,dirs[i]), top))) del dirs[i] for filename in files: filename=os.path.relpath(os.path.join(root,filename)) # print('checking {0}'.format(filename)) if matcher.match(filename): print(filename) if __name__=='__main__': prune(r'foo/\w+/bar/\d+-\w+.dat')项，这样就不会更改要删除的剩余项的索引。）

~/test% tree .
.
|-- foo
|   `-- baz
|       |-- bad
|       |   |-- bad1.txt
|       |   `-- badbad
|       |       `-- bad2.txt
|       `-- bar
|           |-- 1-good.dat
|           `-- 2-good.dat
`-- tmp
    |-- 000.png
    |-- 001.png
    `-- output.gif

使用如下目录结构运行脚本：

pruning tmp
pruning foo/baz/bad
foo/baz/bar/2-good.dat
foo/baz/bar/1-good.dat

产量

{{1}}

如果取消注释“检查”打印语句，很明显修剪后的目录不会被移动。

Answer 2

我编写了一个函数 select_walk（）来搜索和选择目录树中的文件。

在以下示例中，搜索的文件是名称与以下正则表达式模式匹配的目录中扩展名为.dat，.rtf，.jpeg的文件：

r'J:\\f[ruv]?o+\\\w+\\b[ae]r(\d+)?\\(?(1)TURI\1\d*|MONO\d+)

注意条件基本模式的存在：

(?(1)TURI\1\d*|MONO\d+)

将小组参考(1)和\1添加到基本模式中的数字匹配组（\ d +） b [ae] r（\ d +）< / em>的

1）

这是一个用于创建以例子为目的的目录树的代码：

（注意，它首先删除目录'foo \'，'fooo \'，'froooo \'，'faooo \'然后再创建它们）

import os from shutil import rmtree top = 'J:\\' for x in ('foo\\','fooo\\','froooo\\','faooo\\'): if os.path.isdir(top + x): rmtree(top + x) li = [('foo\\',('basil\\','poto%\\','tamata\\')), ('foo\\basil\\',('ber89','ber300')), ('foo\\basil\\ber89\\',('TURI850','TURI1023')), ('foo\\poto%\\',('ocean','earth')), ('foo\\tamata\\',('vahine',)), ('fooo\\',('york#\\','plain\\','atlantis\\')), ('fooo\\york#\\',('noto','nata')), ('fooo\\plain\\',('zx13ao','ws89rt','bar999')), ('fooo\\plain\\bar999\\',('TURI99905','TURI2227','MONO2')), ('fooo\\plain\\bar999\\TURI99905\\',('AERIAL','minidisc')), ('fooo\\plain\\bar999\\TURI99905\\AERIAL\\',('bumbum','corean')), ('fooo\\atlantis\\',('atlABC','atlDEFG')), ('fooo\\atlantis\\atlABC\\',('atlantis_sound','atlantis_image')), ('froooo\\',('one_dir\\','another_dir\\')), ('froooo\\one_dir\\',('bar25','ber')), ('froooo\\one_dir\\bar25\\',('TURI2501','TURI2502','TURI4813','MONO8')), ('froooo\\one_dir\\ber\\',('TURI30','TURI','MONO532')), ('froooo\\another_dir\\',('notseen','notseen2')), ('faooo\\',('somolo-\\','samala+\\'))] for rep,several in li: #print top + rep if os.path.isdir(top + rep) == False: os.mkdir(top + rep) for name in several: #print top + rep + name os.mkdir(top + rep + name) for filepath in (top + 'foo\\kalaomi.xls', top + 'foo\\basil\\ber89\\TURI850\\quetzal.jpeg', top + 'foo\\basil\\ber89\\TURI850\\tehoi.txt', top + 'foo\\poto%\\curcuma in poto%.txt', top + 'foo\\poto%\\ocean\\file in ocean.rtf', top + 'foo\\tamata\\vahine\\tahiti.jpeg', top + 'fooo\\york#\\yorkshire.jpeg', top + 'fooo\\plain\\bar999\\TURI99905\\galileo.jpeg', top + 'fooo\\plain\\bar999\\TURI99905\\polynesia.dat', top + 'fooo\\plain\\bar999\\TURI99905\\concrete.txt', top + 'fooo\\plain\\bar999\\TURI2227\\Monroe.jpeg', top + 'fooo\\plain\\bar999\\MONO2\\elastic.jpeg', top + 'froooo\\one_dir\\photo in one_dir.jpeg', top + 'froooo\\one_dir\\tabula.xls', top + 'froooo\\one_dir\\bar25\\TURI2501\\matallelo.jpeg', top + 'froooo\\one_dir\\bar25\\TURI2501\\italy.dat', top + 'froooo\\one_dir\\bar25\\TURI2501\\beretta.xls', top + 'froooo\\one_dir\\bar25\\TURI2501\\turi2501_ser.rtf', top + 'froooo\\one_dir\\bar25\\TURI4813\\boaf_inTURI4813.jpeg', top + 'froooo\\one_dir\\bar25\\TURI4813\\troui_in_TURI4813.txt', top + 'froooo\\one_dir\\bar25\\MONO8\\in_mono8.dat', top + 'froooo\\one_dir\\bar25\\MONO8\\in_mono8.rtf', top + 'froooo\\one_dir\\bar25\\MONO8\\in_mono8.xls', top + 'froooo\\one_dir\\bar25\\TURI2502\\adamante.jpeg', top + 'froooo\\one_dir\\bar25\\TURI2502\\egyptic.txt', top + 'froooo\\one_dir\\bar25\\TURI2502\\urubu.rtf', top + 'froooo\\one_dir\\ber\\MONO532\\bacillus.jpeg', top + 'froooo\\one_dir\\ber\\MONO532\\blueberry.dat', top + 'froooo\\one_dir\\ber\\MONO532\\Perfume.doc', top + 'faooo\\samala+\\kfaz.dat', top + 'faooo\\somolo-\\ytek.rtf', top + 'faooo\\123.txt', top + 'faooo\\458.rtf',): with open(filepath,'w') as f: pass

此代码创建以下树：

J: | |--foo | |--basil | |--ber89 | |--TURI850 | |--file quetzal.jpeg | |--file tehoi.txt | |--TURI1023 | |--ber300 | |--poto% | |--ocean | |--file in ocean.rtf | |--earth | |--file curcuma in poto%.txt | |--tamata | |--vahine | |--file tahiti.jpeg | |--file kalaomi.xls | |--fooo | |--york# | |--noto | |--nata | |---file yorkshire.jpeg | |--plain | |--zx13ao | |--ws89rt | |--bar999 | |--TURI99905 | |--AERIAL | |--bumbum | |--corean | |--minidisc | |--file galileo.jpeg | |--file polynesia.dat | |--file concrete.txt | |--TURI2227 | |--file Monroe.jpeg | |--MONO2 | |--file elastic.jpeg | |--atlantis | |--atlABC | |--atlantis_sound | |--atlantis_image | |--atlDEFG | |--froooo | |--one_dir | |--bar25 | |--TURI2501 | |--file matalello.jpeg | |--file italy.dat | |--file beretta.xls | |--file turi2501_ser.rtf | |--TURI2502 | |--file adamante.jpeg | |--file egyptic.txt | |--file urubu.rtf | |--TURI4813 | |--file boaf_inTURI4813.jpeg | |--file troui_inTURI4813.txt | |--MONO8 | |--file in_mono8.dat | |--file in_mono8.rtf | |--file in_mono8.xls | |--ber | |--TURI30 | |--TURI | |--MONO532 | |--file bacillus.jpeg | |--file blueberry.dat | |--file Perfume.doc | |--file photo in one_dir.jpeg | |--file tabula.xls | |--another_dir | |--notseen | |--notseen2 | |--faooo | |--somolo- | |--file ytek.rtf | |--samala+ | |file kfaz.dat | |--file 123.txt | |--file 458.rtf

与文件匹配的正则表达式的模式是：

r'J:\\f[ruv]?o+\\\w+\\b[ae]r(\d+)?\\(?(1)TURI\1\d*|MONO\d+)\\\w+\.(dat|rtf|jpeg)'

并且有选择地探索以搜索此类文件的目录将是以下内容：

'J:\\fooo\\plain\\bar999\\TURI99905' 'J:\\froooo\\one_dir\\bar25\\TURI2501' 'J:\\froooo\\one_dir\\bar25\\TURI2502' 'J:\\froooo\\one_dir\\ber\\MONO532'

2）

作为初步演示，这里的代码显示了 select_walk（）函数代码的一部分的功能，该代码构建了在树中迭代遍历期间仅探索所选目录所需的正则表达式。返回所选文件：

import re def compute_regexes(pat_file, displ = True): from os import sep splitted_pat = re.split(r'\\\\' if sep=='\\' else '/', pat_file) pat_parent_dir = (r'\\' if sep=='\\' else '/').join(splitted_pat[0:-1]) if displ: print ('IN FUNCTION compute_regexes() :' '\n\npat_file== %s' '\n\nsplitted_pat :\n%s' '\n\npat_parent_dir== %s\n') \ % (pat_file , '\n'.join(splitted_pat) , pat_parent_dir) dgr = {} for i,el in enumerate(splitted_pat): if re.search('\(.*?\)',el): dgr[len(dgr)+1] = i if displ: print 'dgr :' print '\n'.join('group(%s) is in splitted_pat[%s]' % (g,i) for g,i in dgr.iteritems()) def repl(mat, dgr = dgr): the = int(mat.group(1) if mat.group(1) else mat.group(2)) return str(the + dgr[the]) for i,el in enumerate(splitted_pat): splitted_pat[i] = re.sub(r'(?<=\(\?\()(\d+)(?=\))|(?<=\\)(\d+)',repl,el) pat_dirs = '' for x in splitted_pat[-2:0:-1]: pat_dirs = r'(?=\\|\Z)(\\%s%s)?' % (x,pat_dirs) pat_dirs = splitted_pat[0] + pat_dirs if displ: print '\npat_dirs==',pat_dirs return (re.compile(pat_file), re.compile(pat_dirs), re.compile(pat_parent_dir) ) pat_file = r'J:\\f[ruv]?o+\\\w+\\b[ae]r(\d+)?\\(?(1)TURI\1\d*|MONO\d+)\\\w+\.(dat|rtf|jpeg)' regx_file, regx_dirs, regx_parent_dir = compute_regexes(pat_file) print '\n\nEXAMPLES with regx_file :\n' print 'pat_file==',pat_file for filepath in ('J:\\fooo\\basil\\ber92\TURI9258\\beru.rtf ', 'J:\\froooooo\\ki_ki\\bar\MONO47\\madrid.jpeg '): print filepath,bool(regx_file.match(filepath)) print '\n\nEXAMPLES with regx_dirs :\n' for path in ('J:\\fooo', 'J:\\fooo\\basil', 'J:\\fooo\\basil\\ber92', 'J:\\fooo\\basil\\ber92\\TURI777', 'J:\\fooo\\basil\\ber92\\TURI9258', 'J:\\froooooo' 'J:\\froooooo\\ki_ki', 'J:\\froooooo\\ki_ki\\bar', 'J:\\froooooo\\ki=ki\\bar', 'J:\\froooooo\\ki_ki\\bar\MONO47'): print path,(" : ~~ this dir's name is OK ~~" if path==''.join(regx_dirs.match(path).group()) else " : ## this dir's name doesn't match ##")

函数 compute_regexes（）首先将原始 pat_file 正则表达式模式拆分为旨在匹配路径中目录名称的元素。

然后它计算：

正则表达式模式 pat_dirs ，以匹配所需文件的包含目录的不同级别的路径

匹配所需文件的任何直接父目录的正则表达式模式 pat_parent_dir

暗示 dgr 的处理和功能 repl（）是一种复杂性，允许函数 compute_regexes（）考虑到该组的引用（id est：特殊序列\ 1 \ 2等）并更改它们以获得 pat_dirs ，组的引用相对于为创建 pat_dirs 而引入的添加括号仍然正确。< / p>
此代码的结果：

IN FUNCTION compute_regexes() : pat_file== J:\\f[ruv]?o+\\\w+\\b[ae]r(\d+)?\\(?(1)TURI\1\d*|MONO\d+)\\\w+\.(dat|rtf|jpeg) splitted_pat : J: f[ruv]?o+ \w+ b[ae]r(\d+)? (?(1)TURI\1\d*|MONO\d+) \w+\.(dat|rtf|jpeg) pat_parent_dir== J:\\f[ruv]?o+\\\w+\\b[ae]r(\d+)?\\(?(1)TURI\1\d*|MONO\d+) dgr : group(1) is in splitted_pat[3] group(2) is in splitted_pat[4] group(3) is in splitted_pat[5] pat_dirs== J:(?=\\|\Z)(\\f[ruv]?o+(?=\\|\Z)(\\\w+(?=\\|\Z)(\\b[ae]r(\d+)?(?=\\|\Z)(\\(?(4)TURI\4\d*|MONO\d+))?)?)?)? EXAMPLES with regx_file : pat_file== J:\\f[ruv]?o+\\\w+\\b[ae]r(\d+)?\\(?(1)TURI\1\d*|MONO\d+)\\\w+\.(dat|rtf|jpeg) J:\fooo\basil\ber92\TURI9258\beru.rtf True J:\froooooo\ki_ki\bar\MONO47\madrid.jpeg True EXAMPLES with regx_dirs : J:\fooo : ~~ this dir's name is OK ~~ J:\fooo\basil : ~~ this dir's name is OK ~~ J:\fooo\basil\ber92 : ~~ this dir's name is OK ~~ J:\fooo\basil\ber92\TURI777 : ## this dir's name doesn't match ## J:\fooo\basil\ber92\TURI9258 : ~~ this dir's name is OK ~~ J:\frooooooJ:\froooooo\ki_ki : ## this dir's name doesn't match ## J:\froooooo\ki_ki\bar : ~~ this dir's name is OK ~~ J:\froooooo\ki=ki\bar : ## this dir's name doesn't match ## J:\froooooo\ki_ki\bar\MONO47 : ~~ this dir's name is OK ~~

3）

最后，这是函数

select_walk（）

执行搜索名称与某个正则表达式匹配的树中的文件的工作：
它会产生由内置的 os.walk（）函数返回的三元组（dirpath，dirnames，filenames），但只有那些目录 filenames 包含与 pat_file 匹配的正确文件名称。

当然，在迭代期间，函数 select_walk（）不会探索文件内容永远不会与密钥正则表达式模式 pat_file 匹配的目录，因为它们的（目录'）名称。

def select_walk(pat_file,start_dir): from os import sep splitted_pat = re.split(r'\\\\' if sep=='\\' else '/', pat_file) pat_parent_dir = (r'\\' if sep=='\\' else '/').join(splitted_pat[0:-1]) dgr = {} for i,el in enumerate(splitted_pat): if re.search('\(.*?\)',el): dgr[len(dgr)+1] = i def repl(mat, dgr = dgr): the = int(mat.group(1) if mat.group(1) else mat.group(2)) return str(the + dgr[the]) for i,el in enumerate(splitted_pat): splitted_pat[i] = re.sub(r'(?<=\(\?\()(\d+)(?=\))|(?<=\\)(\d+)',repl,el) pat_dirs = '' for x in splitted_pat[-2:0:-1]: pat_dirs = r'(?=\\|\Z)(\\%s%s)?' % (x,pat_dirs) pat_dirs = splitted_pat[0] + pat_dirs print 'pat_dirs==',pat_dirs regx_file = re.compile(pat_file) regx_dirs = re.compile(pat_dirs) regx_parent_dir = re.compile(pat_parent_dir) start_dir = start_dir.rstrip(sep) + sep print '\nstart_dir == '+start_dir for dirpath,dirnames,filenames in os.walk(start_dir): dirpath = dirpath.rstrip(sep) print '\n'.join(('explored dirpath : %s is_direct_parent: %s' \ % (dirpath,('NO','YES')[bool(regx_parent_dir.match(dirpath))]), ' dirnames : %s' % dirnames, ' filenames : %s' % filenames)) if regx_parent_dir.match(dirpath): filenames[:] = [filename for filename in filenames if regx_file.match(dirpath + sep + filename)] dirnames[:] = [] print '\n'.join((' dirnames : not to be explored ' , ' yielded filenames : %s\n' % filenames)) yield (dirpath,dirnames,filenames) else: dirnames[:] = [dirname for dirname in dirnames if regx_dirs.match(dirpath + sep + dirname).group()==dirpath + sep + dirname] print '\n'.join(('dirnames to explore : %s ' % dirnames, ' filenames : not to be yielded\n')) pat_file = r'J:\\f[ruv]?o+\\\w+\\b[ae]r(\d+)?\\(?(1)TURI\1\d*|MONO\d+)\\\w+\.(dat|rtf|jpeg)' print '\n\nSELECTED (dirpath, dirnames, filenames) :\n' + '\n'.join(map(repr, select_walk(pat_file,'J:\\')))

结果

pat_dirs== J:(?=\\|\Z)(\\f[ruv]?o+(?=\\|\Z)(\\\w+(?=\\|\Z)(\\b[ae]r(\d+)?(?=\\|\Z)(\\(?(4)TURI\4\d*|MONO\d+))?)?)?)? start_dir == J:\ explored dirpath : J: is_direct_parent: NO dirnames : ['Amazon', 'faooo', 'Favorites', 'foo', 'fooo', 'froooo', 'Python', 'RECYCLER', 'System Volume Information'] filenames : ['image00.pfm', 'rep.py'] dirnames to explore : ['foo', 'fooo', 'froooo'] filenames : not to be yielded explored dirpath : J:\foo is_direct_parent: NO dirnames : ['basil', 'poto%', 'tamata'] filenames : ['kalaomi.xls'] dirnames to explore : ['basil', 'tamata'] filenames : not to be yielded explored dirpath : J:\foo\basil is_direct_parent: NO dirnames : ['ber300', 'ber89'] filenames : [] dirnames to explore : ['ber300', 'ber89'] filenames : not to be yielded explored dirpath : J:\foo\basil\ber300 is_direct_parent: NO dirnames : [] filenames : [] dirnames to explore : [] filenames : not to be yielded explored dirpath : J:\foo\basil\ber89 is_direct_parent: NO dirnames : ['TURI1023', 'TURI850'] filenames : [] dirnames to explore : [] filenames : not to be yielded explored dirpath : J:\foo\tamata is_direct_parent: NO dirnames : ['vahine'] filenames : [] dirnames to explore : [] filenames : not to be yielded explored dirpath : J:\fooo is_direct_parent: NO dirnames : ['atlantis', 'plain', 'york#'] filenames : [] dirnames to explore : ['atlantis', 'plain'] filenames : not to be yielded explored dirpath : J:\fooo\atlantis is_direct_parent: NO dirnames : ['atlABC', 'atlDEFG'] filenames : [] dirnames to explore : [] filenames : not to be yielded explored dirpath : J:\fooo\plain is_direct_parent: NO dirnames : ['bar999', 'ws89rt', 'zx13ao'] filenames : [] dirnames to explore : ['bar999'] filenames : not to be yielded explored dirpath : J:\fooo\plain\bar999 is_direct_parent: NO dirnames : ['MONO2', 'TURI2227', 'TURI99905'] filenames : [] dirnames to explore : ['TURI99905'] filenames : not to be yielded explored dirpath : J:\fooo\plain\bar999\TURI99905 is_direct_parent: YES dirnames : ['AERIAL', 'minidisc'] filenames : ['concrete.txt', 'galileo.jpeg', 'polynesia.dat'] dirnames : not to be explored yielded filenames : ['galileo.jpeg', 'polynesia.dat'] explored dirpath : J:\froooo is_direct_parent: NO dirnames : ['another_dir', 'one_dir'] filenames : [] dirnames to explore : ['another_dir', 'one_dir'] filenames : not to be yielded explored dirpath : J:\froooo\another_dir is_direct_parent: NO dirnames : ['notseen', 'notseen2'] filenames : [] dirnames to explore : [] filenames : not to be yielded explored dirpath : J:\froooo\one_dir is_direct_parent: NO dirnames : ['bar25', 'ber'] filenames : ['photo in one_dir.jpeg', 'tabula.xls'] dirnames to explore : ['bar25', 'ber'] filenames : not to be yielded explored dirpath : J:\froooo\one_dir\bar25 is_direct_parent: NO dirnames : ['MONO8', 'TURI2501', 'TURI2502', 'TURI4813'] filenames : [] dirnames to explore : ['TURI2501', 'TURI2502'] filenames : not to be yielded explored dirpath : J:\froooo\one_dir\bar25\TURI2501 is_direct_parent: YES dirnames : [] filenames : ['beretta.xls', 'italy.dat', 'matallelo.jpeg', 'turi2501_ser.rtf'] dirnames : not to be explored yielded filenames : ['italy.dat', 'matallelo.jpeg', 'turi2501_ser.rtf'] explored dirpath : J:\froooo\one_dir\bar25\TURI2502 is_direct_parent: YES dirnames : [] filenames : ['adamante.jpeg', 'egyptic.txt', 'urubu.rtf'] dirnames : not to be explored yielded filenames : ['adamante.jpeg', 'urubu.rtf'] explored dirpath : J:\froooo\one_dir\ber is_direct_parent: NO dirnames : ['MONO532', 'TURI', 'TURI30'] filenames : [] dirnames to explore : ['MONO532'] filenames : not to be yielded explored dirpath : J:\froooo\one_dir\ber\MONO532 is_direct_parent: YES dirnames : [] filenames : ['bacillus.jpeg', 'blueberry.dat', 'Perfume.doc'] dirnames : not to be explored yielded filenames : ['bacillus.jpeg', 'blueberry.dat'] SELECTED (dirpath, dirnames, filenames) : ('J:\\fooo\\plain\\bar999\\TURI99905', [], ['galileo.jpeg', 'polynesia.dat']) ('J:\\froooo\\one_dir\\bar25\\TURI2501', [], ['italy.dat', 'matallelo.jpeg', 'turi2501_ser.rtf']) ('J:\\froooo\\one_dir\\bar25\\TURI2502', [], ['adamante.jpeg', 'urubu.rtf']) ('J:\\froooo\\one_dir\\ber\\MONO532', [], ['bacillus.jpeg', 'blueberry.dat'])

在Python中查找“-regex ...”或如何查找整个名称（路径+名称）与正则表达式匹配的文件？

2 个答案:

1）

2）

3）

select_walk（）