Question

我正在尝试使用具有相同布局的更改文件更新主文件。我想使用更改文件中的键替换/追加主文件中的行记录。两个输入文件都有重复项。需要使用匹配的记录更新主文件，并且新记录应附加到主文件中。

输入文件将具有相同的布局和＆＃34; |＆＃34;划分和巨大的（25-40GB）。你能帮帮我吗？

示例 -

主文件：

键1 | AAA | BBB | CCC

键1 | AAA | BBB | DDD

键1 | XXX | YYY | ZZZ

密钥2 | ZZZ | YYY | 123

密钥2 | EEE | FFF | RRR

密钥3 | RRR | EEE | GGG

密钥3 | SSS | TTT | GGG

更改文件：

键1 | 111 | 222 | 333

键1 | 222 | 333 | 444

键4 | 888 | 333 | 222

键4 | 888 | 777 | 222

输出文件：

键1 | 111 | 222 | 333

键1 | 222 | 333 | 444

密钥2 | ZZZ | YYY | 123

密钥2 | EEE | FFF | RRR

密钥3 | RRR | EEE | GGG

密钥3 | SSS | TTT | GGG

键4 | 888 | 333 | 222

键4 | 888 | 777 | 222

sample data in image format

Answer 1

所以我尝试了一下，因为这听起来很有趣，我正在学习python 请找到以下代码这适用于您的样品。但是，如果主文件中有更大的后续键孔，它会混合顺序。我无法修复它。

我真的遇到了很多问题，数据结构中有多个主键分布在多行上。

我不知道你在做什么，但我正在使用数据库，我可以告诉你，这种数据结构非常不寻常。如果您要重新构建数据集，可能会受益匪浅。

使用这些数据，您可能会将其存储在数据库中。如果你没有对它进行深入的学习算法。

示例：的这是混合订单的例子，但仍然有效

主文件

Key1|AAA|BBB|CCC
Key1|AAA|BBB|DDD
Key1|XXX|YYY|ZZZ
Key2|ZZZ|YYY|123
Key2|EEE|FFF|RRR
Key3|RRR|EEE|GGG
Key3|SSS|TTT|GGG
Key7|RRR|EEE|GGG
Key7|SSS|TTT|GGG

changefile

Key1|111|222|333
Key1|222|333|444
Key1|222|333|555
Key4|888|333|222
Key4|888|777|222
Key5|888|333|222
Key5|888|777|222
Key6|888|333|222
Key6|888|777|222
Key8|888|333|222
Key8|888|777|222
Key9|888|333|222
Key9|888|777|222

代码：

import fileinput

with open('changefile.txt') as infile:
    keyindex = []
    for line in infile:
        linelist = line.strip().split("|") ## split line by |
        key = linelist[0] ## assign the key
        keyid = linelist[0][3:] ## assign keyid
        keylist = [] ## assign keylist for loop

        ## finding duplicate keys in changefile and assign them to list
        if key not in keyindex: ## we need this because multiple keys in multiple lines
            with open('changefile.txt') as infile2:
                #spawning extra loop for each new key to open and search all duplicate keys and assign them to list
                for line2 in infile2:
                    if line2.startswith(key):
                        print(line2)
                        keylist.append(line2)
            ## Delete line with current key of loop from master file
            keyindex.append(key)
            print(keylist)
            for linem in fileinput.input('test.txt', inplace=True):
                if key in linem:
                    continue
                print(linem, end='')
            ## insert keys from keyindex

            for linei in fileinput.input('test.txt', inplace=1):

                    if 'Key'+str(int(keyid)+1) in linei: ## This statement is case sensitive
                        for item in keylist:
                            print(item, end='')
                        keylist = []
                    print(linei, end='')


# I had problems with not beeing able to go to next line at the beginning of this code if you fix this, this would be better then opening the file anew
##                    if last in linei and keylist:
##                        ##print('\n')
##                        for item in keylist:
##                            print(item, end='')
##                        keylist = []
##                        print('\n')

## this block may cause problem with memory you may can fix this with the comment block before this.
## this block is for adding left over keys from the end of change file to the end of master e.g. id 9 is in changefile, but masterfile is only going to key8
            with open("test.txt", "a") as myfile:
                if keylist:
                    for item in keylist:
                            myfile.write(item)
                    keylist = []
                else:
                    continue




        ## because we spawned a seperate loop each time we find a new key, we can skip the duplicate lines                    
        else:
            ## print('>>>key '+line+'already worked at! go to next line') # if you want to skip, uncomment continue and comment this 
            continue
    ##print all keyindexes that have been changed
    print('Following keys have been changed:'keyindex)

使用python中的键值使用更改文件更新主文件

1 个答案: