Question

我的日志文件包含每行中的一些信息，如下所示

Info1:NewOrder|key:123 |Info3:10|Info5:abc
Info3:10|Info1:OldOrder| key:456| Info6:xyz
Info1:NewOrder|key:007

我想将其更改为如下所示的csv（如果我将密钥，Info1，Info3更改为必需的标题）

key,Info1.Info3
123,NewOrder,10
456,OldOrder,10
007,NewOrder,

之前我使用awk获取字段值，但是日志记录可以更改连续打印的信息和密钥的顺序。所以我不能确定Info3总是会出现在某个特定列中。每次记录更改时，都需要更改脚本。

我打算在pandas dataframe中加载csv。所以python解决方案会更好。这更像是从日志文件生成csv的数据清理任务。

这是我在阅读答案后使用的内容

import csv
import sys
with open(sys.argv[1], 'r') as myLogfile:
        log=myLogfile.read().replace('\n', '')

requested_columns = ["OrderID", "TimeStamp", "ErrorCode"]

def wrangle(string, requested_columns):
        data = [dict([element.strip().split(":") for element in row.split("|")]) for row in string.split("\n")]
        body = [[row.get(column) for column in requested_columns] for row in data]
        return [requested_columns] + body

outpath = sys.argv[2]
open(outpath, "w", newline = "") with open(outpath, 'wb')
        writer = csv.writer(file)
        writer.writerows(wrangle(log, requested_columns))

示例logfile = https://ideone.com/cny805

Answer 1

它的大部分内容只是使用有用的字符串方法，如strip和split，以及列表推导。

import csv

string = """Info1=NewOrder|key=123 |Info3=10|Info5=abc
Info3=10|Info1=OldOrder| key=456| Info6=xyz
Info1=NewOrder|key=007"""

requested_columns = ["key", "Info1", "Info3"]

def wrangle(string, requested_columns):
    data = [dict([element.strip().split("=") for element in row.split("|")]) for row in string.split("\n")]
    body = [[row.get(column) for column in requested_columns] for row in data]
    return [requested_columns] + body

outpath = "whatever.csv"

with open(outpath, "w", newline = "") as file:
    writer = csv.writer(file)
    writer.writerows(wrangle(string, requested_columns))

Answer 2

您可以使用带有|分隔符的csv阅读器来帮助您入门，然后使用:进行拆分，为您提供每行字典，如下所示：

import csv

with open('input.csv', 'rb') as f_input, open('output.csv', 'wb') as f_output:
    csv_output = csv.writer(f_output)
    cols = ["OrderID", "TimeStamp", "ErrorCode"]
    csv_output.writerow(cols)

    for row in csv.reader(f_input, delimiter='|'):
        # Remove any entries that do not have a colon
        row = [c for c in row if c.find(':') != -1]
        # Convert remaining columns into a dictionary
        entries = {c.split(':')[0].strip() : c.split(':')[1].strip() for c in row}
        csv_output.writerow([entries.get(c, "") for c in cols])

给你一个输出文件：

OrderID,TimeStamp,ErrorCode
3000000,1488948188555841641,
3000000,1488948188556444675,0

直接将数据读入Pandas数据帧：

import pandas as pd
import csv

cols = ["OrderID", "TimeStamp", "ErrorCode"]
data = []

with open('input.csv', 'rb') as f_input:
    csv_output = csv.writer(f_output)

    for row in csv.reader(f_input, delimiter='|'):
        # Remove any entries that do not have a colon
        row = [c for c in row if c.find(':') != -1]
        # Convert remaining columns into a dictionary
        entries = {c.split(':')[0].strip() : c.split(':')[1].strip() for c in row}
        data.append([entries.get(c, "") for c in cols])

df = pd.DataFrame(data, columns=cols)
print df

给你：

   OrderID            TimeStamp ErrorCode
0  3000000  1488948188555841641          
1  3000000  1488948188556444675         0

从日志文件python创建csv头文件

2 个答案: