从mysql-slow.log文件中删除重复的行

Question

我的文件名为mysql-slow.log。

我想删除重复的查询。

搜索行是 SELECT帐户。，accounts_cstm。 FROM帐户LEFT JOIN accounts_cstm ON accounts.id = accounts_cstm.id_c WHERE customer_id_c =''AND deleted = 0 LIMIT 0,1; < /强>

以下是示例文件

# Time: 180110 11:31:06
# User@Host: root[root] @ localhost [127.0.0.1]
# Query_time: 2.224965  Lock_time: 0.000052 Rows_sent: 1  Rows_examined: 92610
SET timestamp=1515564066;
SELECT b_invoice.*,b_invoice_cstm.* FROM b_invoice  LEFT JOIN b_invoice_cstm ON b_invoice.id = b_invoice_cstm.id_c  WHERE order_id_c = '212959' AND deleted=0 LIMIT 0,1;
# Time: 180110 11:38:12
# User@Host: root[root] @ localhost [127.0.0.1]
# Query_time: 2.041713  Lock_time: 0.000048 Rows_sent: 0  Rows_examined: 101355
SET timestamp=1515564492;
SELECT accounts.*,accounts_cstm.* FROM accounts  LEFT JOIN accounts_cstm ON accounts.id = accounts_cstm.id_c  WHERE customer_id_c = '' AND deleted=0 LIMIT 0,1;
# Time: 180110 11:39:02
# User@Host: root[root] @ localhost [127.0.0.1]
# Query_time: 2.124880  Lock_time: 0.000037 Rows_sent: 0  Rows_examined: 101355
SET timestamp=1515564542;
SELECT accounts.*,accounts_cstm.* FROM accounts  LEFT JOIN accounts_cstm ON accounts.id = accounts_cstm.id_c  WHERE customer_id_c = '' AND deleted=0 LIMIT 0,1;
# Time: 180110 11:39:18
# User@Host: root[root] @ localhost [127.0.0.1]
# Query_time: 2.016269  Lock_time: 0.000026 Rows_sent: 0  Rows_examined: 101355
SET timestamp=1515564558;
SELECT accounts.*,accounts_cstm.* FROM accounts  LEFT JOIN accounts_cstm ON accounts.id = accounts_cstm.id_c  WHERE customer_id_c = '' AND deleted=0 LIMIT 0,1;
# Time: 180110 11:40:11
# User@Host: root[root] @ localhost [127.0.0.1]
# Query_time: 2.261057  Lock_time: 0.000040 Rows_sent: 0  Rows_examined: 101355
SET timestamp=1515564611;
SELECT accounts.*,accounts_cstm.* FROM accounts  LEFT JOIN accounts_cstm ON accounts.id = accounts_cstm.id_c  WHERE customer_id_c = '' AND deleted=0 LIMIT 0,1;
# Time: 180110 11:40:13
# User@Host: root[root] @ localhost [127.0.0.1]
# Query_time: 2.862533  Lock_time: 0.000050 Rows_sent: 0  Rows_examined: 101355
SET timestamp=1515564613;
SELECT accounts.*,accounts_cstm.* FROM accounts  LEFT JOIN accounts_cstm ON accounts.id = accounts_cstm.id_c  WHERE customer_id_c = '' AND deleted=0 LIMIT 0,1;

输出文件应为

#Time: 180110 11:31:06
# User@Host: root[root] @ localhost [127.0.0.1]
# Query_time: 2.224965  Lock_time: 0.000052 Rows_sent: 1  Rows_examined: 92610
SET timestamp=1515564066;
SELECT b_invoice.*,b_invoice_cstm.* FROM b_invoice  LEFT JOIN b_invoice_cstm ON b_invoice.id = b_invoice_cstm.id_c  WHERE order_id_c = '212959' AND deleted=0 LIMIT 0,1;
# Time: 180110 11:38:12
# User@Host: root[root] @ localhost [127.0.0.1]
# Query_time: 2.041713  Lock_time: 0.000048 Rows_sent: 0  Rows_examined: 101355
SET timestamp=1515564492;
SELECT accounts.*,accounts_cstm.* FROM accounts  LEFT JOIN accounts_cstm ON accounts.id = accounts_cstm.id_c  WHERE customer_id_c = '' AND deleted=0 LIMIT 0,1;

注意：同时在查询前删除上3行

#Time
#User
#Query_time

删除单行的mysql命令是

sudo sed -i '/^SELECT accounts.*,accounts_cstm.* FROM accounts  LEFT JOIN accounts_cstm ON accounts.id = accounts_cstm.id_c  WHERE customer_id_c = '' AND deleted=0 LIMIT 0,1;/d' mysql-slow.log

Answer 1

这个可以帮到你:)

awk '/^SET/{ s = $0; next }/^SELECT/&& !(a[$0]++){ print s; print $0 }' infile

输出：

SET timestamp=1515564066;
SELECT b_invoice.*,b_invoice_cstm.* FROM b_invoice  LEFT JOIN b_invoice_cstm ON b_invoice.id = b_invoice_cstm.id_c  WHERE order_id_c = '212959' AND deleted=0 LIMIT 0,1;
SET timestamp=1515564492;
SELECT accounts.*,accounts_cstm.* FROM accounts  LEFT JOIN accounts_cstm ON accounts.id = accounts_cstm.id_c  WHERE customer_id_c = '' AND deleted=0 LIMIT 0,1;

第一部分/^SET/{ s = $0; next }查找以SET开头的每一行。当一行以set开头时，整行（$ 0）存储在变量s中。第二部分/^SELECT/&& !(a[$0]++){ print s; print $0 }'正在查找以SELECT开头并且不在数组a中的行。如果一条线不在a中，它将被添加，并且将打印变量s和以SELECT开头的整行。