从python中的多个列表中查找重复项

Question

我想从单个列表中的多个列表中找到重复项（不是删除这些重复项但是提取这些重复值），例如：a有一个名为Chunks的列表，里面有13个列表。

我的数据如下

[[@TestRun
    And user set text "#Surname" on textbox name "surname"
    And user validate message on screen "Switch to paperless" 
    And user click on "Manage accounts" label 
    And user click link with label "View all online services" 
    And user waits for 10 seconds 
    Then page is successfully launched 
    And user click link with label "Go paperless for complete convenience" 
    Then page is successfully launched 
    And user validate message on screen "#EmailAddress" 
    And user clicks on the button "Confirm" 
    Then page is successfully launched 
    And user validate message on screen "#MessageValidate" 
    Then page is successfully launched 
    And user click on "menu open user preferences" label 
    And user clicks on the link "Statement and letter preferences" 
    Then page is successfully launched 
    And user validate "Switch to paperless" button is disabled 
    And user validate message on screen "Online only" 
    When user click on "Log out" label 
    Then page is successfully launched]

[@TestRun 
    And user click on link "Mobile site" 
    And user set text "#Surname" on textbox name "surname" 
    Then page is successfully launched 
    And user click on link "#Account" 
    Then page is successfully launched 
    And user verify message on screen "#Account" 
    And user verify message on screen "Manage statements" 
    And user verify message on screen "Step 1 of 3" 
    Then page is successfully launched 
    And user verify message on screen "Current format type"  
    And user verify message on screen "Online" 
    When user selects the radio button "Paper" ]


[@TestRun
 And user set text "#Surname" on textbox name "surname"
Then user wait for page load
And user click on button "Continue to Online Banking"
Then user wait for page load
    And user click on "menu open user preferences" label 
    And user clicks on the link "Statement and letter preferences" 
    Then page is successfully launched 
    And page is successfully launched 
    And user waits for 10 seconds ]
[ @TestRun
    And user set text "#Surname" on textbox name "surname"
    Then page is successfully launched 
    And user waits for 10 seconds 
    And user click checkbox "Telephone" 
    And user click checkbox "Post" 
    And user clicks on the button "Save" 
    Then page is successfully launched ]]

我已经在一个列表中提取了每个测试用例，即两个@testrun作为一个列表的行

 import itertools as it
import more_itertools as mit
import pandas as pd
## got seperated all test case in seprate list i.e 13 test cases in 13 lists
with open('cust_pref.txt', "r") as f1:
    lines_1 = f1.readlines()

    pred_1 = lambda x: x.startswith("@TestRun")      
    inv_pred_1 = lambda x: not pred_1(x)

    lines_1 = it.dropwhile(inv_pred_1, lines_1)         
    chunks_1 = list(mit.split_before(lines_1, pred_1))
##print the list of testcases
print(chunks_1)

现在我需要找出如何在所有这些列表中找到共同点，以及如何知道哪些列表是常见的

我尝试了以下

def get_duplicated_element(array):
    global result, checked_elements
    checked_elements = []
    result = -1
    def array_recursive_check(array):
        global result, checked_elements
        if result != -1: return
        for i in array:
            if type(i) == list:
                if i in checked_elements:
                    result = i
                    return
                checked_elements.append(i)
                array_recursive_check(i)
    array_recursive_check(array)
    return result

get_duplicated_element(chunks_1) ## this gives the answer as -1 , which is not expected

预期输出是：找到共同的值/行（在我的情况下），如果可能的话，哪些步骤出现在python中的列表编号

所需的输出是：

{  
    And user set text "#Surname" on textbox name "surname"
    Then page is successfully launched 
}

AS在每个列表中重复这些步骤，因此这些sholud是输出

我使用了以下内容来获取重复内容

def find_dupe(lists, target):
    seen = set()
    for lst in lists:
        for item in lst:
            if item == target and item in seen:
                return True
            seen.add(item)

seen, dups = set(), set()
for l in chunks:
    dups = dups.union(seen.intersection(set(l)))
    seen = seen.union(set(l))

我从中得到了一些重复但现在我的问题是我不知道哪一行来自哪个列表？有没有办法实现这个映射哪个值对应哪个列表

Answer 1

不是您想要的所需输出，但您可以获得进一步处理的提示。检查一下：

>>> data = [['@TestRun',
  '    And user set text "#Surname" on textbox name "surname"',
  '    And user validate message on screen "Switch to paperless" ',
  '    And user click on "Manage accounts" label ',
  '    And user click link with label "View all online services" ',
  '    And user waits for 10 seconds ',
  '    Then page is successfully launched ',
  '    And user click link with label "Go paperless for complete convenience" ',
  '    Then page is successfully launched ',
  '    And user validate message on screen "#EmailAddress" ',
  '    And user clicks on the button "Confirm" ',
  '    Then page is successfully launched ',
  '    And user validate message on screen "#MessageValidate" ',
  '    Then page is successfully launched ',
  '    And user click on "menu open user preferences" label ',
  '    And user clicks on the link "Statement and letter preferences" ',
  '    Then page is successfully launched ',
  '    And user validate "Switch to paperless" button is disabled ',
  '    And user validate message on screen "Online only" ',
  '    When user click on "Log out" label ',
  '    Then page is successfully launched'],
 ['@TestRun ',
  '    And user click on link "Mobile site" ',
  '    And user set text "#Surname" on textbox name "surname" ',
  '    Then page is successfully launched ',
  '    And user click on link "#Account" ',
  '    Then page is successfully launched ',
  '    And user verify message on screen "#Account" ',
  '    And user verify message on screen "Manage statements" ',
  '    And user verify message on screen "Step 1 of 3" ',
  '    Then page is successfully launched ',
  '    And user verify message on screen "Current format type"  ',
  '    And user verify message on screen "Online" ',
  '    When user selects the radio button "Paper"'],
 ['@TestRun',
  ' And user set text "#Surname" on textbox name "surname"',
  'Then user wait for page load',
  'And user click on button "Continue to Online Banking"',
  'Then user wait for page load',
  '    And user click on "menu open user preferences" label ',
  '    And user clicks on the link "Statement and letter preferences" ',
  '    Then page is successfully launched ',
  '    And page is successfully launched ',
  '    And user waits for 10 seconds']]
>>> data = [[line.strip() for line in test_set] for test_set in data]
>>> linewise_counts = {}
>>> for list_index,test_set in enumerate(pdata):
        for line in test_set:
            linewise_counts.setdefault(line,set()).add(list_index)


>>> duplicates = ["{} -> {}".format(line, in_list) for line,in_list in linewise_counts.items() if len(in_list)>1]
>>> duplicates
['And user clicks on the link "Statement and letter preferences" -> set([0, 2])',
 'And user waits for 10 seconds -> set([0, 2])',
 'Then page is successfully launched -> set([0, 1, 2])',
 '@TestRun -> set([0, 1, 2])',
 'And user set text "#Surname" on textbox name "surname" -> set([0, 1, 2])',
 'And user click on "menu open user preferences" label -> set([0, 2])']

Answer 2

您可以使用re和defaultdict

执行某些操作

def read_file(filehandle):
''' yields the chunks of the file, delimited by the `@TestRun`'''
    count = 0
    text = mmap.mmap(file.fileno(), 0)  # read all text in memory
    # https://stackoverflow.com/a/454589/1562285
    string_pattern = re.compile(rb'(?:\[\@TestRun(.+?)\].*?)*', re.DOTALL)
    for item in string_pattern.findall(text):
        if item:
            yield count, [i.strip() for i in item.decode('utf8').strip().split('\n')]
            count += 1

def parse_chunks(chunks):
""" puts the lines of these chunks into a dick, with the line as key and a list of the positions of this line `(chunk_no, line_no) as value`"""
    result = collections.defaultdict(list)
    for chunk_no, lines in chunks:
        for i, line in enumerate(lines):
            result[line].append((chunk_no, i))
    return dict(result)

然后你可以像这样使用

with open(file, 'r') as file
    chunks = read_file(file)
    result = parse_chunks(chunks)

{
    'And user set text "#Surname" on textbox name "surname"': [(0, 0), (1, 1), (2, 0), (3, 0)], 
    'And user validate message on screen "Switch to paperless"': [(0, 1)], 
    'And user click on "Manage accounts" label': [(0, 2)], 
    'And user click link with label "View all online services"': [(0, 3)],
    'And user waits for 10 seconds': [(0, 4), (2, 8), (3, 2)],
    'Then page is successfully launched': [(0, 5), (0, 7), (0, 10), (0, 12), (0, 15), (0, 19), (1, 2), (1, 4), (1, 8), (2, 6), (3, 1), (3, 6)],
    'And user click link with label "Go paperless for complete convenience"': [(0, 6)],
    'And user validate message on screen "#EmailAddress"': [(0, 8)],
    'And user clicks on the button "Confirm"': [(0, 9)],
    'And user validate message on screen "#MessageValidate"': [(0, 11)],
    'And user click on "menu open user preferences" label': [(0, 13), (2, 4)],
    'And user clicks on the link "Statement and letter preferences"': [(0, 14), (2, 5)],
    'And user validate "Switch to paperless" button is disabled': [(0, 16)],
    'And user validate message on screen "Online only"': [(0, 17)],
    'When user click on "Log out" label': [(0, 18)],
    'And user click on link "Mobile site"': [(1, 0)],
    'And user click on link "#Account"': [(1, 3)],
    'And user verify message on screen "#Account"': [(1, 5)],
    'And user verify message on screen "Manage statements"': [(1, 6)],
    'And user verify message on screen "Step 1 of 3"': [(1, 7)],
    'And user verify message on screen "Current format type"': [(1, 9)],
    'And user verify message on screen "Online"': [(1, 10)],
    'When user selects the radio button "Paper"': [(1, 11)],
    'Then user wait for page load': [(2, 1), (2, 3)],
    'And user click on button "Continue to Online Banking"': [(2, 2)],
    'And page is successfully launched': [(2, 7)],
    'And user click checkbox "Telephone"': [(3, 3)],
    'And user click checkbox "Post"': [(3, 4)],
    'And user clicks on the button "Save"': [(3, 5)]
}

您可以使用

过滤这些重复项

{key: value for key, value in result.items() if len(value)> 1}