我是Python的新手，尤其是数据处理。这就是我想要实现的目标-
我在多台服务器上运行CIS测试，并为每台服务器生成一个CSV文件（文件名与服务器名相同）。所有服务器的输出文件都复制到中央服务器 产生的输出如下所示（截断的输出）-
File1: dc1pp1v01.co.uk.csv
Description,Outcome,Result
1.1 Database Placement,/var/lib/mysql,PASSED
1.2 Use dedicated least privilaged account,mysql,PASSED
1.3 Diable MySQL history,Not Found,PASSED
File2: dc1pp2v01.co.uk.csv
Description,Outcome,Result
1.1 Database Placement,/var/lib/mysql,PASSED
1.2 Use dedicated least privilaged account,mysql,PASSED
1.3 Diable MySQL history,Not Found,PASSED
File..n: dc1pp1v02.co.uk.csv
Description,Outcome,Result
1.1 Database Placement,/var/lib/mysql,PASSED
1.2 Use dedicated least privilaged account,mysql,PASSED
1.3 Diable MySQL history,Found,FAILED
我想要的是输出看起来像-
Description dc1pp1v01 dc1pp2v01 dc1pp1v02
0 1.1 Database Placement PASSED PASSED PASSED
1 1.2 Use dedicated least privilaged account PASSED PASSED PASSED
2 1.3 Diable MySQL history PASSED PASSED FAILED
要合并这些文件，我创建了另一个文件，其中仅包含Description字段，并且标题如下两栏-
file: cis_report.csv
Description,Result
1.1 Database Placement,
1.2 Use dedicated least privilaged account,
1.3 Diable MySQL history,
我已经编写了以下代码以进行基于列的合并-
import glob
import os
import pandas as pd
col_list = ["Description","Result"]
path = "/Users/Python/Data"
all_files = glob.glob(os.path.join(path, "dc*.csv"))
cis_df = pd.read_csv("/Users/Python/Data/cis_report.csv")
for fl in all_files:
d = pd.read_csv(fl, usecols=col_list)
f = cis_df.merge(d, on='Description')
cis_df = f.copy()
print(cis_df.head())
我得到的输出是-
Description Result_x Result_y Result_x Result_y
0 1.1 Database Placement NaN PASSED PASSED PASSED
1 1.2 Use dedicated least privilaged account NaN PASSED PASSED PASSED
2 1.3 Diable MySQL history NaN PASSED PASSED FAILED
在我的代码中，我不确定如何获取文件名作为结果的标题并摆脱NaN。
此外，是否有一种更好的方法可以在不使用虚拟文件（cis_report.csv）的情况下实现我正在寻找的输出？非常感谢您的帮助。
答案 0 :(得分：3)
您需要DataFrme.pivot()函数。
下面的代码得到了很好的注释，并且是一个可以正常工作的示例。根据需要进行更改
import os
import pandas as pd
# Get all file names in a directory
# Use . to use current working directory or replace it with
# e.g. r'C:\Users\Dames\Desktop\csv_files'
file_names = os.listdir('.')
# Filter out all non .csv files
# You can skip this if you know that only .csv files will be in that folder
csv_file_names = [fn for fn in file_names if fn[-4:] == '.csv']
# This Loads a csv file into a dataframe and sets the Server column
def load_csv(file_name):
df = pd.read_csv(file_name)
df['Server'] = file_name.split('.')[0]
return df
# Append all the csvfiles after being processed by load_csv
df = pd.DataFrame().append([load_csv(fn) for fn in csv_file_names])
# Turn DataFrame into Pivot Table
df = df.pivot('Description', 'Server', 'Result')
# Save DataFrame into CSV File
# If this script runs multiple times make sure that the final.csv is saved elsewhere!
# Or it will be read by the code above as an input file
df.to_csv('final.csv')
最终的DataFrame看起来像这样
Server dc1pp1v01 dc1pp1v02 dc1pp2v01
Description
1.1 Database Placement PASSED PASSED PASSED
1.2 Use dedicated least privilaged account PASSED PASSED PASSED
1.3 Diable MySQL history PASSED FAILED PASSED
还有这样的CSV文件
Description,dc1pp1v01,dc1pp1v02,dc1pp2v01
1.1 Database Placement,PASSED,PASSED,PASSED
1.2 Use dedicated least privilaged account,PASSED,PASSED,PASSED
1.3 Diable MySQL history,PASSED,FAILED,PASSED
答案 1 :(得分：0)
使用-
import glob
import os
import pandas as pd
col_list = ["Description","Result"]
path = "/Users/Python/Data"
all_files = glob.glob(os.path.join(path, "dc*.csv"))
cis_df = pd.read_csv("/Users/Python/Data/cis_report.csv")
from functools import reduce
df_final = reduce(lambda left,right: pd.merge(left,right,on='Description'), [cis_df]+[pd.read_csv(i, usecols=col_list) for i in all_files])
df_final.drop([i for i in df_final.columns if 'Outcome' in i], axis=1).rename(columns={i:j for i,j in zip([i for i in df_final.columns if 'Result' in i], [i.replace('.co.uk.csv','') for i in all_files])})
输出
Description dc1pp1v01 dc1pp2v01 dc1pp1v02
0 1.1 Database Placement PASSED PASSED PASSED
1 1.2 Use dedicated least privilaged account PASSED PASSED PASSED
2 1.3 Diable MySQL history PASSED PASSED FAILED
答案 2 :(得分：0)
最后，我设法自己完成了。下面的解决方案对我有用，但是我敢肯定还有更简洁的方法-
SELECT *
FROM
(
SELECT u.userid, u.photo, u.username, pc.created_date
FROM `last_activity` AS la
INNER JOIN `user` AS u ON la.userid = u.userid
LEFT JOIN private_chat pc ON u.userid = pc.sender_id
WHERE la.last_activity_date BETWEEN '2020-09-21 10:20:00' AND '2020-09-21 10:30:00'
AND u.username LIKE '%amit%'
UNION ALL
SELECT userid, photo, username, pc.created_date
FROM `user` u
LEFT JOIN private_chat pc ON u.userid = pc.sender_id
WHERE u.u_type = '2'
) a
ORDER BY a.created_date DESC;
我在描述保持文件中做了一些小的更改。删除了结果字段，并在每行的结尾处添加了“，”
文件：cis_report.csv
import glob
import os
import pandas as pd
from functools import reduce
col_list = ["Description","Result"]
path = "/Users/Python/Data"
all_files = glob.glob(os.path.join(path, "dc*.csv"))
final_cols = ['Description']
for j in all_files:
final_cols.append(os.path.basename(j).split('.',1)[0])
cis_df = pd.read_csv("/Users/Python/Data/cis_report.csv")
df_final = reduce(lambda left,right: pd.merge(left,right,on='Description'), [cis_df]+[pd.read_csv(i, usecols=col_list) for i in all_files])
df_final.rename(columns=dict(zip(df_final.columns,final_cols)),inplace=True)
print(df_final.head())
我得到的输出是-
Description
1.1 Database Placement
1.2 Use dedicated least privilaged account
1.3 Diable MySQL history
答案 3 :(得分：0)
尽管如此，您已经有了赢家：
import csv
from pathlib import Path
path = Path('/Users/Python/Data')
# Read the reports and store the results in a 2-dim list
results = []
for file in path.glob('dc*.co.uk.csv'):
with open(file, 'r') as fin:
results += [[file.name.split('.')[0]]
+ [row[2] for row in csv.reader(fin)][1:]]
# Read the row labels
with open(path / 'cis_report.csv', 'r') as fin:
labels = [row[0] for row in csv.reader(fin)]
# Prepare the output
output = [[label] + [result[i] for result in results]
for i, label in enumerate(labels)]
# Write the output
with open(path / 'cis_reports_merged.csv', 'w') as fout:
csv.writer(fout, delimiter='\t').writerows(output)