MySql对连接查询的计数非常慢

时间:2012-06-28 22:50:00

标签: mysql sql query-optimization

我有一个mysql的视图:

CREATE VIEW
    loggingquarantine_quarantine ( id, mail_id, partition_tag, content, rs, subject, sender, TIME,
    spam_level, size, sid, email ) AS
SELECT
    concat(CAST(`mr`.`mail_id` AS CHAR(255) charset utf8),CAST(`mr`.`partition_tag` AS CHAR(255)
    charset utf8))      AS `id`,
    `mr`.`mail_id`       AS `mail_id`,
    `mr`.`partition_tag` AS `partition_tag`,
    `mr`.`content`      AS `content`,
    `mr`.`rs`           AS `rs`,
    `m`.`subject`       AS `subject`,
    `m`.`from_addr`     AS `sender`,
    `m`.`time_num`      AS `TIME`,
    `m`.`spam_level`    AS `spam_level`,
    `m`.`size`          AS `size`,
    `m`.`sid`           AS `sid`,
    `maddr`.`email`          AS `email`
FROM
    (((`msgrcpt` `mr` JOIN `msgs` `m`
ON
    (
                `m`.`partition_tag` = `mr`.`partition_tag`
        AND
                `m`.`mail_id` = `mr`.`mail_id`
    )
)
JOIN `maddr` maddr
ON
    (
        `mr`.`rid` = `maddr`.`id`
    )
))

当我试图计算这个视图时,花费大约13分钟来获得250万的记录。那非常慢。所有字段都有索引。如果我指望每张桌子,它不会超过20秒。 这是mysql解释显示的内容:

mysql> explain SELECT COUNT(*) FROM `loggingquarantine_quarantine`;
+----+-------------+-------+--------+-----------------------------------------------------------------------+-------------------------+---------+-----------------------------------------------------------+---------+-------------+
| id | select_type | table | type   | possible_keys                                                         | key                     | key_len | ref                                                       | rows    | Extra       |
+----+-------------+-------+--------+-----------------------------------------------------------------------+-------------------------+---------+-----------------------------------------------------------+---------+-------------+
|  1 | SIMPLE      | maddr | index  | PRIMARY                                                               | maddr_partition_tag_idx | 5       | NULL                                                      | 1016497 | Using index |
|  1 | SIMPLE      | mr    | ref    | PRIMARY,msgrcpt_idx_rid,msgrcpt_mail_id_idx,msgrcpt_partition_tag_idx | msgrcpt_idx_rid         | 8       | mroute_logquar.maddr.id                                   |       2 | Using index |
|  1 | SIMPLE      | m     | eq_ref | PRIMARY,msgs_mail_id_idx,msgs_partition_tag_idx                       | PRIMARY                 | 22      | mroute_logquar.mr.partition_tag,mroute_logquar.mr.mail_id |       1 | Using index |
+----+-------------+-------+--------+-----------------------------------------------------------------------+-------------------------+---------+-----------------------------------------------------------+---------+-------------+

如何优化查询/视图,以便不需要13分钟进行计数..当前查询有什么问题?

更新。 如果我在没有视图的情况下直接在select上执行选择计数,它仍会获得相同的14分钟查询。

mysql> select count(1) FROM     (((`msgrcpt` `mr` JOIN `msgs` `m` ON     (                  `m`.`partition_tag` = `mr`.`partition_tag`         AND                 `m`.`mail_id` = `mr`.`mail_id`      ) ) JOIN `maddr` maddr ON     (         `mr`.`rid` = `maddr`.`id`     ) ));

+----------+
| count(1) |
+----------+
|  2582227 |
+----------+
1 row in set (14 min 28.96 sec)

当我在三个单独的查询中执行此操作时,这是计数的结果:

mysql> select count(1) from msgrcpt;
+----------+
| count(1) |
+----------+
|  2587307 |
+----------+
1 row in set (46.02 sec)

mysql> select count(1) from msgs;
+----------+
| count(1) |
+----------+
|  2421710 |
+----------+
1 row in set (7.77 sec)

mysql> select count(1) from maddr;
+----------+
| count(1) |
+----------+
|   994880 |
+----------+
1 row in set (0.23 sec)

更新2。

所有表格都是InnoDB。

mysql> SHOW status like 'key_%';                                                                                                                                                                               +------------------------+-------+
| Variable_name          | Value |
+------------------------+-------+
| Key_blocks_not_flushed | 0     |
| Key_blocks_unused      | 26792 |
| Key_blocks_used        | 0     |
| Key_read_requests      | 0     |
| Key_reads              | 0     |
| Key_write_requests     | 0     |
| Key_writes             | 0     |
+------------------------+-------+

msgs和msgrcpt表具有复合主键(msgsm的ms_和partition_tagmail_idrseqnum的mail_id,partation_tag)。 UPDATE 解释单表:

mysql> explain select count(1) from msgs;
+----+-------------+-------+-------+---------------+-------------------+---------+------+---------+-------------+
| id | select_type | table | type  | possible_keys | key               | key_len | ref  | rows    | Extra       |
+----+-------------+-------+-------+---------------+-------------------+---------+------+---------+-------------+
|  1 | SIMPLE      | msgs  | index | NULL          | msgs_idx_time_num | 4       | NULL | 2357360 | Using index |
+----+-------------+-------+-------+---------------+-------------------+---------+------+---------+-------------+
1 row in set (0.00 sec)

mysql> explain select count(1) from msgrcpt;
+----+-------------+---------+-------+---------------+----------------+---------+------+---------+-------------+
| id | select_type | table   | type  | possible_keys | key            | key_len | ref  | rows    | Extra       |
+----+-------------+---------+-------+---------------+----------------+---------+------+---------+-------------+
|  1 | SIMPLE      | msgrcpt | index | NULL          | msgrcpt_rs_idx | 3       | NULL | 2620758 | Using index |
+----+-------------+---------+-------+---------------+----------------+---------+------+---------+-------------+
1 row in set (0.00 sec)

mysql> explain select count(1) from maddr;
+----+-------------+-------+-------+---------------+-------------------------+---------+------+--------+-------------+
| id | select_type | table | type  | possible_keys | key                     | key_len | ref  | rows   | Extra       |
+----+-------------+-------+-------+---------------+-------------------------+---------+------+--------+-------------+
|  1 | SIMPLE      | maddr | index | NULL          | maddr_partition_tag_idx | 5       | NULL | 967058 | Using index |
+----+-------------+-------+-------+---------------+-------------------------+---------+------+--------+-------------+
1 row in set (0.00 sec)

更新。 为所有表创建表:

mysql> show create table msgrcpt;
| Table   | Create Table                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             
| msgrcpt | CREATE TABLE `msgrcpt` (
  `partition_tag` int(11) NOT NULL DEFAULT '0',
  `mail_id` varbinary(16) NOT NULL,
  `rseqnum` int(11) NOT NULL DEFAULT '0',
  `rid` bigint(20) unsigned NOT NULL,
  `is_local` char(1) NOT NULL DEFAULT '',
  `content` char(1) NOT NULL DEFAULT '',
  `ds` char(1) NOT NULL,
  `rs` char(1) NOT NULL,
  `bl` char(1) DEFAULT '',
  `wl` char(1) DEFAULT '',
  `bspam_level` float DEFAULT NULL,
  `smtp_resp` varchar(255) DEFAULT '',
  PRIMARY KEY (`partition_tag`,`mail_id`,`rseqnum`),
  KEY `msgrcpt_idx_rid` (`rid`),
  KEY `msgrcpt_mail_id_idx` (`mail_id`),
  KEY `msgrcpt_rs_idx` (`rs`),
  KEY `msgrcpt_ds_idx` (`ds`),
  KEY `msgrcpt_partition_tag_idx` (`partition_tag`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |

| msgs  | CREATE TABLE `msgs` (
  `partition_tag` int(11) NOT NULL DEFAULT '0',
  `mail_id` varbinary(16) NOT NULL,
  `secret_id` varbinary(16) DEFAULT '',
  `am_id` varchar(20) NOT NULL,
  `time_num` int(10) unsigned NOT NULL,
  `time_iso` char(16) NOT NULL,
  `sid` bigint(20) unsigned NOT NULL,
  `policy` varchar(255) DEFAULT '',
  `client_addr` varchar(255) DEFAULT '',
  `size` int(10) unsigned NOT NULL,
  `originating` char(1) NOT NULL DEFAULT '',
  `content` char(1) DEFAULT NULL,
  `quar_type` char(1) DEFAULT NULL,
  `quar_loc` varbinary(255) DEFAULT '',
  `dsn_sent` char(1) DEFAULT NULL,
  `spam_level` float DEFAULT NULL,
  `message_id` varchar(255) DEFAULT '',
  `from_addr` varchar(255) DEFAULT '',
  `subject` varchar(255) DEFAULT '',
  `host` varchar(255) NOT NULL,
  PRIMARY KEY (`partition_tag`,`mail_id`),
  KEY `msgs_idx_sid` (`sid`),
  KEY `msgs_idx_mess_id` (`message_id`),
  KEY `msgs_idx_time_num` (`time_num`),
  KEY `msgs_mail_id_idx` (`mail_id`),
  KEY `msgs_partition_tag_idx` (`partition_tag`),
  KEY `msgs_content_idx` (`content`),
  FULLTEXT KEY `ft_from_addr` (`from_addr`),
  FULLTEXT KEY `ft_subject` (`subject`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 |

| maddr | CREATE TABLE `maddr` (
  `partition_tag` int(11) DEFAULT '0',
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `email` varbinary(255) NOT NULL,
  `domain` varchar(255) NOT NULL,
  PRIMARY KEY (`id`),
  UNIQUE KEY `part_email` (`partition_tag`,`email`),
  KEY `maddr_email_idx` (`email`),
  KEY `maddr_partition_tag_idx` (`partition_tag`)
) ENGINE=InnoDB AUTO_INCREMENT=3373444 DEFAULT CHARSET=utf8 |

此查询包含个人资料:

mysql> SET PROFILING=1; SELECT
Query OK, 0 rows affected (0.00 sec)

    -> count(1)
    -> FROM
    ->     (((`msgrcpt` `mr` JOIN `msgs` `m`
    -> ON
    ->     (
    ->                 `m`.`partition_tag` = `mr`.`partition_tag`
    ->         AND
    ->                 `m`.`mail_id` = `mr`.`mail_id`
    ->     )
    -> )
    -> JOIN `maddr` maddr
    -> ON
    ->     (
    ->         `mr`.`rid` = `maddr`.`id`
    ->     )
    -> )); SHOW PROFILE ALL;

+----------+
| count(1) |
+----------+
|  4279394 |
+----------+
1 row in set (23 min 56.61 sec)

+----------------------+------------+-----------+------------+-------------------+---------------------+--------------+---------------+---------------+-------------------+-------------------+-------------------+-------+-----------------------+------------------+-------------+
| Status               | Duration   | CPU_user  | CPU_system | Context_voluntary | Context_involuntary | Block_ops_in | Block_ops_out | Messages_sent | Messages_received | Page_faults_major | Page_faults_minor | Swaps | Source_function       | Source_file      | Source_line |
+----------------------+------------+-----------+------------+-------------------+---------------------+--------------+---------------+---------------+-------------------+-------------------+-------------------+-------+-----------------------+------------------+-------------+
| starting             |   0.000161 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | NULL                  | NULL             |        NULL |
| checking permissions |   0.000030 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 1 |     0 | check_access          | sql_parse.cc     |        5043 |
| checking permissions |   0.000019 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | check_access          | sql_parse.cc     |        5043 |
| checking permissions |   0.000020 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | check_access          | sql_parse.cc     |        5043 |
| Opening tables       |   0.000039 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | open_tables           | sql_base.cc      |        5014 |
| System lock          |   0.000026 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | mysql_lock_tables     | lock.cc          |         304 |
| init                 |   0.000040 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | mysql_select          | sql_select.cc    |        1041 |
| optimizing           |   0.000030 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | optimize              | sql_optimizer.cc |         138 |
| statistics           |   0.000063 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | optimize              | sql_optimizer.cc |         358 |
| preparing            |   0.000032 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | optimize              | sql_optimizer.cc |         470 |
| executing            |   0.000021 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | exec                  | sql_executor.cc  |         137 |
| Sending data         | 999.999999 | 97.014251 |  10.376423 |            681167 |               25822 |      5157072 |       1951032 |             0 |                 0 |                 4 |               277 |     0 | execute               | sql_executor.cc  |         758 |
| end                  |   0.000106 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | mysql_select          | sql_select.cc    |        1071 |
| query end            |   0.000017 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | mysql_execute_command | sql_parse.cc     |        4761 |
| closing tables       |   0.000021 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | mysql_execute_command | sql_parse.cc     |        4809 |
| freeing items        |   0.000030 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | mysql_parse           | sql_parse.cc     |        5997 |
| logging slow query   |   0.000059 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             8 |             0 |                 0 |                 0 |                 0 |     0 | log_slow_statement    | sql_parse.cc     |        1720 |
| cleaning up          |   0.000019 |  0.000000 |   0.000000 |                 0 |                   0 |            0 |             0 |             0 |                 0 |                 0 |                 0 |     0 | dispatch_command      | sql_parse.cc     |        1654 |
+----------------------+------------+-----------+------------+-------------------+---------------------+--------------+---------------+---------------+-------------------+-------------------+-------------------+-------+-----------------------+------------------+-------------+
18 rows in set (0.02 sec)

表格索引:

mysql> show index from msgs;
+-------+------------+------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name               | Seq_in_index | Column_name   | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| msgs  |          0 | PRIMARY                |            1 | partition_tag | A         |          16 |     NULL | NULL   |      | BTREE      |         |               |
| msgs  |          0 | PRIMARY                |            2 | mail_id       | A         |     4174440 |     NULL | NULL   |      | BTREE      |         |               |
| msgs  |          1 | msgs_idx_sid           |            1 | sid           | A         |     2087220 |     NULL | NULL   |      | BTREE      |         |               |
| msgs  |          1 | msgs_idx_mess_id       |            1 | message_id    | A         |     4174440 |     NULL | NULL   | YES  | BTREE      |         |               |
| msgs  |          1 | msgs_idx_time_num      |            1 | time_num      | A         |     1391480 |     NULL | NULL   |      | BTREE      |         |               |
| msgs  |          1 | msgs_mail_id_idx       |            1 | mail_id       | A         |     4174440 |     NULL | NULL   |      | BTREE      |         |               |
| msgs  |          1 | msgs_partition_tag_idx |            1 | partition_tag | A         |          16 |     NULL | NULL   |      | BTREE      |         |               |
| msgs  |          1 | msgs_content_idx       |            1 | content       | A         |          16 |     NULL | NULL   | YES  | BTREE      |         |               |
| msgs  |          1 | ft_from_addr           |            1 | from_addr     | NULL      |     4174440 |     NULL | NULL   | YES  | FULLTEXT   |         |               |
| msgs  |          1 | ft_subject             |            1 | subject       | NULL      |     4174440 |     NULL | NULL   | YES  | FULLTEXT   |         |               |
+-------+------------+------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
10 rows in set (0.97 sec)

MSGRCPT

mysql> show index from msgrcpt;
+---------+------------+---------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table   | Non_unique | Key_name                  | Seq_in_index | Column_name   | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------+------------+---------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| msgrcpt |          0 | PRIMARY                   |            1 | partition_tag | A         |          29 |     NULL | NULL   |      | BTREE      |         |               |
| msgrcpt |          0 | PRIMARY                   |            2 | mail_id       | A         |     5218535 |     NULL | NULL   |      | BTREE      |         |               |
| msgrcpt |          0 | PRIMARY                   |            3 | rseqnum       | A         |     5218535 |     NULL | NULL   |      | BTREE      |         |               |
| msgrcpt |          1 | msgrcpt_idx_rid           |            1 | rid           | A         |      347902 |     NULL | NULL   |      | BTREE      |         |               |
| msgrcpt |          1 | msgrcpt_mail_id_idx       |            1 | mail_id       | A         |     5218535 |     NULL | NULL   |      | BTREE      |         |               |
| msgrcpt |          1 | msgrcpt_rs_idx            |            1 | rs            | A         |          29 |     NULL | NULL   |      | BTREE      |         |               |
| msgrcpt |          1 | msgrcpt_ds_idx            |            1 | ds            | A         |          29 |     NULL | NULL   |      | BTREE      |         |               |
| msgrcpt |          1 | msgrcpt_partition_tag_idx |            1 | partition_tag | A         |          29 |     NULL | NULL   |      | BTREE      |         |               |
+---------+------------+---------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
8 rows in set (0.70 sec)

MADDR:

mysql> show index from maddr;
+-------+------------+-------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name                | Seq_in_index | Column_name   | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+-------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| maddr |          0 | PRIMARY                 |            1 | id            | A         |     1653970 |     NULL | NULL   |      | BTREE      |         |               |
| maddr |          0 | part_email              |            1 | partition_tag | A         |          19 |     NULL | NULL   | YES  | BTREE      |         |               |
| maddr |          0 | part_email              |            2 | email         | A         |     1653970 |     NULL | NULL   |      | BTREE      |         |               |
| maddr |          1 | maddr_email_idx         |            1 | email         | A         |     1653970 |     NULL | NULL   |      | BTREE      |         |               |
| maddr |          1 | maddr_partition_tag_idx |            1 | partition_tag | A         |          19 |     NULL | NULL   | YES  | BTREE      |         |               |
+-------+------------+-------------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
5 rows in set (0.41 sec)

Inno db buffer size

mysql> SHOW VARIABLES LIKE 'innodb_buffer_pool_size';
+-------------------------+------------+
| Variable_name           | Value      |
+-------------------------+------------+
| innodb_buffer_pool_size | 2147483648 |
+-------------------------+------------+
1 row in set (0.02 sec)

9 个答案:

答案 0 :(得分:1)

您是否尝试过使用某些STRAIGHT_JOIN而不是简单的JOIN来改变JOIN中使用的表的顺序?有时,查询优化器不会为特定查询选择理想的顺序。换句话说,你对此有何看法?

SELECT
  concat(CAST(`mr`.`mail_id` AS CHAR(255) charset utf8),CAST(`mr`.`partition_tag` AS CHAR(255) charset utf8)) AS `id`,
  `mr`.`mail_id`       AS `mail_id`,
  `mr`.`partition_tag` AS `partition_tag`,
  `mr`.`content`      AS `content`,
  `mr`.`rs`           AS `rs`,
  `m`.`subject`       AS `subject`,
  `m`.`from_addr`     AS `sender`,
  `m`.`time_num`      AS `TIME`,
  `m`.`spam_level`    AS `spam_level`,
  `m`.`size`          AS `size`,
  `m`.`sid`           AS `sid`,
  `maddr`.`email`          AS `email`
FROM
  `msgrcpt` `mr`
    STRAIGHT_JOIN `msgs` `m`
      ON
        `m`.`partition_tag` = `mr`.`partition_tag` AND
        `m`.`mail_id` = `mr`.`mail_id`
    STRAIGHT_JOIN `maddr` maddr
      ON
        `mr`.`rid` = `maddr`.`id`

或者,因为你在msgrcpt表上使用了一些重型转换和转换,如果强制它在查询中最后一次加入会发生什么呢?

SELECT
  concat(CAST(`mr`.`mail_id` AS CHAR(255) charset utf8),CAST(`mr`.`partition_tag` AS CHAR(255) charset utf8)) AS `id`,
  `mr`.`mail_id`       AS `mail_id`,
  `mr`.`partition_tag` AS `partition_tag`,
  `mr`.`content`      AS `content`,
  `mr`.`rs`           AS `rs`,
  `m`.`subject`       AS `subject`,
  `m`.`from_addr`     AS `sender`,
  `m`.`time_num`      AS `TIME`,
  `m`.`spam_level`    AS `spam_level`,
  `m`.`size`          AS `size`,
  `m`.`sid`           AS `sid`,
  `maddr`.`email`          AS `email`
FROM
  `maddr` `maddr`
    STRAIGHT_JOIN `msgrcpt` `mr`
      ON
        `maddr`.`id` = `mr.`rid`
    STRAIGHT_JOIN `msgs` `m`
      ON
        `m`.`partition_tag` = `mr`.`partition_tag` AND
        `m`.`mail_id` = `mr`.`mail_id`

如果需要,您可以更多地使用订单,可能会有一个“魔术订单”导致您的查询压缩。例如,另一种变体可能是:

FROM
  `msgs` `m`
    STRAIGHT_JOIN `msgrcpt` `mr`
      ON
        `m`.`partition_tag` = `mr`.`partition_tag` AND
        `m`.`mail_id` = `mr`.`mail_id`
    STRAIGHT_JOIN `maddr`
      ON
        `maddr`.`id` = `mr`.`rid`

......等等。

此外,在你的SELECT列中,所有这些强制转换都会导致你的性能下降,因为无论你选择多少行都必须这样做,这可能会很多。您是否考虑过将一列添加到已经计算过的msgrcpt表中,以便它不必在查询中执行此操作?您可以向数据库添加触发器,以便有效地自动更新它,如果您不想在代码中考虑额外列,则无需使用计算值。通常情况下,使用这种优化有点过分,但在对数百万行进行SELECT查询时,它可能就是这样做的。

编辑:以下是我建议您更改msgrcpt表格的内容。它会使插入行的速度变慢很小,因为它运行一个触发器来计算新的列值,但我认为它会在加速你试图运行的查询时获得相当大的回报。标准免责声明适用 - 如果没有在测试副本上进行彻底测试,请不要在您的生产数据库上运行此文件!

-- The UPDATE command will probably take some time to run since it's updating
-- millions of rows. Be patient!
ALTER TABLE `msgrcpt` ADD COLUMN `friendly_id` TEXT NULL AFTER `rs`;
UPDATE `msgrcpt` SET
  `friendly_id` = CONCAT(CAST(`mail_id` AS CHAR(255) CHARSET utf8),
    CAST(`partition_tag` AS CHAR(255) CHARSET utf8));
DELIMITER $$
CREATE TRIGGER `trig_calc_id` BEFORE INSERT ON `msgrcpt`
  FOR EACH ROW BEGIN
    SET NEW.`friendly_id` =
      CONCAT(CAST(NEW.`mail_id` AS CHAR(255) CHARSET utf8),
        CAST(NEW.`partition_tag` AS CHAR(255) CHARSET utf8));
  END $$
CREATE TRIGGER `trig_update_id` BEFORE UPDATE ON `msgrcpt`
  FOR EACH ROW BEGIN
    SET NEW.`friendly_id` =
      CONCAT(CAST(NEW.`mail_id` AS CHAR(255) CHARSET utf8),
        CAST(NEW.`partition_tag` AS CHAR(255) CHARSET utf8));
  END $$
DELIMITER ;

现在不要选择那些乱糟糟的CONCATed CASTed东西,只需选择mr.friendly_id(或者你选择命名列的任何东西)。你的表现要好得多。

希望这会有所帮助,让我知道它是怎么回事!

答案 1 :(得分:0)

several methods数据库如何执行表连接。 MySQL正在使用 Nested-loops加入,如section on EXPLAIN output中所述。嵌套循环的工作方式是理解为什么这个查询花费这么多时间的关键。

根据您的EXPLAIN输出,MySQL首先扫描maddr表,因此您有1 016 497个执行循环。在每个执行数据库中,在两个小表上执行索引扫描。即使费用:

  • 启动新的索引扫描;
  • 在索引中搜索条目;
  • 完成扫描

非常小(因为数据很可能将被许多执行缓存),无论如何都需要几毫秒。然后乘以2msgrcptmsgs都在内部扫描。然后乘以外循环的执行次数,这给你很长的时间。

您的视图定义没有对返回的数据进行过滤,因此必须扫描和连接所有表。鉴于嵌套循环如何工作,很明显,当外部查询包含最少量的循环时,变量是性能最佳的循环。考虑到所有表的大小,索引扫描(根据EXPLAIN输出在索引上扫描所有3个表)将进行检查:

  • msgrcpt:log 2 (2587307)+ 1 = 22页,
  • msgs:log 2 (2421710)+ 1 = 22页
  • maddr:log 2 (994880)+ 1 =每次扫描21页
最坏情况场景中的

。这表明表扫描时间具有可比性,即使使用索引扫描也非常重要。尽管如此,maddr更小,这就是MySQL首先选择扫描它的原因。

正如我所看到的,数据库选择的计划是给定情况下的最佳计划。并且MySQL仅支持嵌套循环连接的事实没有提供优化可能性,因为外部循环已经是最小的。此处可能会使用Hash join来获得更好的效果。


那么,如何让事情在这种情况下表现呢?

我建议仅使用视图来查询带过滤器的数据,这将使MySQL有更多机会优化查询并快速返回数据。

要获得行数,我会去估算(如果这是可接受的),计算每个表的count(*)并取最大值。

另一种方法是重新设计模式,使其中一个表成为主模式,以便视图中的行数始终与主表中的行数相匹配。对不起,我不知道如何用给定的设计做到这一点。

如果没有一个选项可以接受,遗憾的是,我认为没有其他方法可以优化此查询。

答案 2 :(得分:0)

我同意Skippus国王关于两个要素...... STRAIGHT_JOIN帮助推翻优化器执行计划......以及为您保留预聚合的触发器。但是,如果您想要应用WHERE子句进行限制,则会被枪杀。

此外,如果您关心的只是一个COUNT,那么vyegorov评论抛出所有列将是最好的。通过让所有其他列强制从所有实际数据页中提取数据。

我会将查询呈现为

SELECT STRAIGHT_JOIN COUNT(*)
   FROM
      msgrcpt mr
         JOIN msgs m
            ON mr.mail_id = m.mail_id
           AND mr.partition_tag = m.partition_tag
         JOIN maddr
            ON mr.rid = maddr.id

我知道你提到索引,但我会 MsgRcpt表索引通过(mail_id,partition_tag) Msgs表索引通过相同的(mail_id,partition_tag) MAddr索引(id)

现在,为什么索引上的另一种方式与另一种方式相比?我会尝试在第一个索引字段中基于最小聚合。不确切知道您的卷基础是什么(邮件ID,分区标记),而是获取客户订单系统的样本。你可以让一个人在一天内订购一些东西,但对于一个给定的人,在某一天订购一两个订单。首先找到该人,将该个人日期与单日相比更容易找到一个人。这可能会对尝试首先通过LARGE列表的查询性能产生影响,然后是小型与小型/小型。

只需要一个COUNT(*),就可以得到一个计数。原始表中的其他列不是必需的,并且每个页面都被加载,因为它只需要索引的组件来限定结果中的记录。

如果你想要所有的数据,那么我会做同样的索引考虑因素,但查询为......

CREATE VIEW
    loggingquarantine_quarantine ( id, mail_id, partition_tag, content, rs, subject, sender, TIME,
    spam_level, size, sid, email ) AS

SELECT STRAIGHT_JOIN
      concat(CAST(mr.mail_id AS CHAR(255) charset utf8), 
             CAST(mr.partition_tag AS CHAR(255) charset utf8)) AS `id`,
      mr.mail_id,
      mr.partition_tag,
      mr.content,
      mr.rs,
      m.subject,
      m.from_addr AS sender,
      m.time_num AS `TIME`,
      m.spam_level,
      m.size,
      m.sid,
      maddr.email
   FROM
      msgrcpt mr
         JOIN msgs m
            ON mr.mail_id = m.mail_id
           AND mr.partition_tag = m.partition_tag
         JOIN maddr
            ON mr.rid = maddr.id

答案 3 :(得分:0)

视图查询itselt以及代码中视图的使用可能会影响性能。有关详细信息,请阅读文章 - MySQL VIEW as performance troublemaker

解决方法:

  • 执行简单查询,不要使用视图。
  • 如果您不想在另一个SELECT语句中将此视图用作数据集,则可以尝试创建存储过程而不是视图。

答案 4 :(得分:0)

尝试使用COUNT(*)insted COUNT(1),因为MySQL优化器非常错误 http://www.mysqlperformanceblog.com/2007/04/10/count-vs-countcol/

答案 5 :(得分:0)

对不起:我看到你的maddr表有一个partition_tag id,但是在组成VIEW的最后一个JOIN中没有使用该分区标记。这是应该的吗?

无论如何,在我看来,您的查询可能会受益于包含id和email的maddr索引。这应该允许直接从索引中提取具有较少I / O的电子邮件(这可能有利于评估各种索引的性能 - 在我看来,例如,单独的电子邮件字段中的索引很少会被使用,但它取决于你知道的工作流程,我不知道,所以我很可能会弄错。

答案 6 :(得分:0)

count()中写一个主键列,使其更快。像count(id)

答案 7 :(得分:0)

好吧:),所以一方面你说 - 我们有250万个条目,另一方面(根据你)计数返回4279394(略高:) :)。可能的原因(对不起,差不多凌晨3点:)) - 你没有指定msgrcpt的完整主键(rseqnum缺失),所以在优化器选择maddr或msgs作为主表的情况下 - 你可能会得到相同的结果msgrcpt的条目。同样基于msgrcpt和msgs的计数时间的差异,我强烈建议对msgrcpt进行碎片整理。我相信在对直接连接进行碎片整理和使用(已经推荐)之后,你的问题就会消失(如果没有 - 也许你可以找到一种方法来动态计算 - 无论你需要为rseqnum / spam_level计算什么,或者,如果spam_level是相同的完整的信息 - 也许你可以通过提供rseqnum的硬编码值来限制选择。

答案 8 :(得分:0)

加入需要BIG mysql加载。 在我的经验中, 创建临时表并从中选择数据, 提高数据库查询速度。

这是我的代码,代码2显示了100倍的快速速度。 您可以像代码2样式一样更改查询代码。 对于大型复杂的数据库查询,请使用临时表提升数据库查询速度。

  1. 使临时表简化查询并减少mysql db load。
  2. 从临时表中选择数据也需要更少的数据库负载。
  3. 代码1

        $sql = " select distinct wr_parent from $write_table where $sql_search ";
        $result = sql_query($sql);
        $total_count = mysql_num_rows($result);
    

    代码2

        $sql = " select wr_parent from $write_table where $sql_search ";
        $sql_tmp = " create TEMPORARY table list_tmp_count as $sql ";
        $sql_ord = " select distinct wr_parent from list_tmp_count ";
    
        @mysql_query($sql_tmp) or die("<p>$sql_tmp<p>" . mysql_errno() . " : " .  mysql_error() . "<p>error file : $_SERVER[PHP_SELF]");
        $result = @mysql_query($sql_ord) or die("<p>$sql_ord<p>" . mysql_errno() . " : " .  mysql_error() . "<p>error file : $_SERVER[PHP_SELF]");
        $total_count = mysql_num_rows($result);
    

    mysql temporary table

    如果临时表在tempfs(内存区域)中的位置速度也加倍。