Question

我自10月以来一直在学习Python，以便从大约一百万个文本文件中提取数据。我一直试图用小巧谨慎的块来解决这个问题，这样我就不会为了一次性获得我想要的所有代码而感到不知所措。

对于我的第一个块，我想从文本文件中提取地址。到目前为止，我已经设法让我的代码一次只能处理一个文件，但由于我有超过100万个文件需要通过，我不认为手动执行此操作会有效。

为了简洁起见，我只包含了我的代码的第一部分，因为其余部分基本上是这样，但是寻找不同的关键字。

######
#Importing/creating modules
######
import os
import re
regex = re.compile('\d+')
######
#Creating N/A text for non-existing entries
######
CV2 = """N/A
N/A
"""


######
#Opening Database file 
######
target = open('C:/project/database.csv', 'a')



####
#Opening the source documents
####
big_file = open('C:/project/0000068100-99-000018.txt', 'r')




####
#Looking for central key
####
for line in big_file:
    if 'CENTRAL INDEX KEY:' in line:
        key = regex.findall(line)


####
#Looking for Conforming Period
####
big_file = open('C:/project/0000068100-99-000018.txt', 'r') 
for line in big_file:
    if 'CONFORMED PERIOD OF REPORT:' in line:
        conform = regex.findall(line)


#####
#Looking for File Type
#####       
big_file = open('C:/project/0000068100-99-000018.txt', 'r') 
for line in big_file:
    if 'CONFORMED SUBMISSION TYPE:' in line:
        type_temp = re.split('\s+',line) 
        type1 = type_temp [3]
        type = type1.split()



#####
#Looking for company name
#####   
big_file = open(''C:/project/0000068100-99-000018.txt', 'r')    
for line in big_file:
    if 'COMPANY CONFORMED NAME:' in line:
        name_temp = re.split('\:+',line) 
        name1 = name_temp [1]
        name2 = name1.split()
        name3 = ' '.join(name2)
        name = re.split('\::+',name3) 





####
#Looking for Street and Mail Addresses
####

big_file = open('C:/project/0000068100-99-000018.txt', 'r')
f = open('C:/Users/Martin/Thesis/address1.txt', 'w+')
for line in big_file:
    if 'STREET 1:' in line:
        f.write(line)



f = open('C:/project/address1.txt', 'w+')
a = open('C:/project/address1a.txt', 'w+')
b = os.path.getsize('C:/project/address1.txt')
########
#If empty, return NA, if not clean unnesseary tabs and formmating from line 
########
if b == 0:
    a.write(CV2)        
else:
    for line in f:
        type_temp = re.split('\:+',line) 
        add_temp = type_temp[1]
        add_temp_temp = ''.join(add_temp)
        add = re.split('\t+',add_temp_temp)
        tempo = "".join(add)
        a.write(tempo)
        #print tempo


a = open('C:/project/address1a.txt', 'w+')
lines=a.readlines()
bus1 = lines[0]
bus2 = re.split('\n+',bus1)
bus3 = bus2 [0]
business1 = re.split('\::+',bus3)
########
#Preping for inclusion for the Database entry 
########
mail1 = lines[1]
mail2 = re.split('\n+',mail1)
mail3 = mail2 [0]
mailad1 = re.split('\::+',mail3)


#####
#Formatting data tags
#####
company_name_temp = "Company Name"
company_name = re.split('\::+',company_name_temp)
bus_street1_temp = "Business Street 1"
bus_street1 = re.split('\::+',bus_street1_temp)
mstreet1_temp = key + conform + type + mail_street1 + mailad1
mstreet1 = ','.join(mstreet1_temp)




######
#Prepping for database
######
name_temp1 = key + conform + type + company_name + name
co_name = ','.join(name_temp1)

bstreet1_temp = key + conform + type + bus_street1 + business1
bstreet1 = ','.join(bstreet1_temp)

mstreet1_temp = key + conform + type + mail_street1 + mailad1
mstreet1 = ','.join(mstreet1_temp)



######
#Writing to database
######
target.write(co_name)
target.write("\n")
target.write(bstreet1)
target.write("\n")
target.write(mstreet1)
target.write("\n")

我尝试在顶部打开一次文件并多次调用该变量，但它没有用，我假设for循环看起来与此类似，但我不知道如何制作它有效。

for filename in os.listdir('C:/project'):
    bigfile = filename

由于

Answer 1

这是因为open返回实际流式传输的FileObjects，因此您无法多次访问它 - 实际上只能访问一次。你想在这里做的更像是这样：

for filename in os.listdir('C:\project'):
    bigfile = open(filename, 'r').read()
    # Now the file contents are saved within bigfile, and you can do as you please,
    #  accessing multiple times.

如果您的文件很大，则不建议这样做，因为Python需要花很多时间才能读入并保存整个文件。这就是open流式传输文件数据的确切原因，因此您可以根据需要随时访问它，而不是被迫一次性获取整个块。

顺便说一句，您应该查看csv模块。

通过多个文本文件循环到Python中的数据提取

1 个答案: