Question

http://example.com/item/all-atv-quad.html,David,"Punjab",+123456789123
http://example.com/item/70cc-2014.html,Qubee,"Capital",+987654321987
http://example.com/item/quad-bike-zenith.html,Zenith,"UP",+123456789123

我有这个test.csv，我从某个网站上抓了几个项目，但事情是“数字”字段有冗余。所以我不知何故需要删除一个与以前相同的行。这只是示例文件，在实际文件中，一些数字重复超过50次。

import csv

with open('test.csv', newline='') as csvfile:
    csvreader = csv.reader(csvfile, delimiter=',')

    for column in csvreader:

        "Some logic here"

        if (column[3] == "+123456789123"):
            print (column[0])

            "or here"

我需要像这样重新格式化的csv：

http://example.com/item/all-atv-quad.html,David,"Punjab",+123456789123
http://example.com/item/70cc-2014.html,Qubee,"Capital",+987654321987

Answer 1

#!/usr/bin/env python
# -*- coding: utf-8 -*-


import pandas as pd


def direct():
    seen = set()
    with open("test.csv") as infile, open("formatted.csv", 'w') as outfile:
        for line in infile:
            parts = line.rstrip().split(',')
            number = parts[-1]
            if number not in seen:
                seen.add(number)
                outfile.write(line)


def using_pandas():
    """Alternatively, use Pandas"""
    df = pd.read_csv("test.csv", header=None)
    df = df.drop_duplicates(subset=[3])
    df.to_csv("formatted_pandas.csv", index=None, header=None)


def main():
    direct()
    using_pandas()


if __name__ == "__main__":
    main()

Answer 2

这会过滤掉重复项：

seen = set()
for line in csvreader:
    if line[3] in seen:
        continue
    seen.add(line[3])
    # write line to output file

csv读写逻辑：

with open('test.csv') as fobj_in, open('test_clean.csv', 'w') as fobj_out:
    csv_reader = csv.reader(fobj_in, delimiter=',')
    csv_writer = csv.writer(fobj_out, delimiter=',')
    seen = set()
    for line in csvreader:
        if line[3] in seen:
            continue
        seen.add(line[3])
        csv_writer.writerow(line)

使用python根据特定字段重新格式化CSV

2 个答案: