为什么这个组会导致文件输出?

时间:2011-11-08 16:51:58

标签: mysql

我有一组高度关系的表格,我已经将其展平为两个InnoDB表格:

mysql> desc WPropertyCube;
+--------------------+---------------------+------+-----+---------+----------------+
| Field              | Type                | Null | Key | Default | Extra          |
+--------------------+---------------------+------+-----+---------+----------------+
| id                 | bigint(20) unsigned | NO   | PRI | NULL    | auto_increment |
| lineOfBusinessId   | bigint(20) unsigned | NO   | MUL | NULL    |                |
| txtProperty1       | varchar(125)        | YES  |     | NULL    |                |
| txtProperty2       | varchar(125)        | YES  |     | NULL    |                |
| txtProperty3       | varchar(125)        | YES  |     | NULL    |                |
...
| txtProperty20      | varchar(125)        | YES  |     | NULL    |                |
| lookupPropertyId1  | bigint(20) unsigned | YES  |     | NULL    |                |
| lookupPropertyId2  | bigint(20) unsigned | YES  |     | NULL    |                |
| lookupPropertyId3  | bigint(20) unsigned | YES  |     | NULL    |                |
...
| lookupPropertyId30 | bigint(20) unsigned | YES  |     | NULL    |                |
+--------------------+---------------------+------+-----+---------+----------------+
mysql> show indexes from WPropertyCube;
+---------------+------------+---------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table         | Non_unique | Key_name            | Seq_in_index | Column_name      | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+---------------+------------+---------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| WPropertyCube |          0 | PRIMARY             |            1 | id               | A         |      379383 |     NULL | NULL   |      | BTREE      |         |               |
| WPropertyCube |          1 | WPropertyCube_idx01 |            1 | lineOfBusinessId | A         |         204 |     NULL | NULL   |      | BTREE      |         |               |
+---------------+------------+---------------------+--------------+------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
mysql> desc WMeasureCube;
+----------------+---------------------+------+-----+---------+-------+
| Field          | Type                | Null | Key | Default | Extra |
+----------------+---------------------+------+-----+---------+-------+
| propertyCubeId | bigint(20) unsigned | NO   | PRI | NULL    |       |
| actionDate     | datetime            | NO   | PRI | NULL    |       |
| measure1       | decimal(15,6)       | YES  |     | NULL    |       |
| measure2       | decimal(15,6)       | YES  |     | NULL    |       |
| measure3       | decimal(15,6)       | YES  |     | NULL    |       |
...
| measure30      | decimal(15,6)       | YES  |     | NULL    |       |
+----------------+---------------------+------+-----+---------+-------+
mysql> show indexes from WMeasureCube;
+--------------+------------+-------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table        | Non_unique | Key_name          | Seq_in_index | Column_name    | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+--------------+------------+-------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| WMeasureCube |          0 | PRIMARY           |            1 | propertyCubeId | A         |          18 |     NULL | NULL   |      | BTREE      |         |               |
| WMeasureCube |          0 | PRIMARY           |            2 | actionDate     | A         |    81680372 |     NULL | NULL   |      | BTREE      |         |               |
| WMeasureCube |          1 | WMeasureCube_idx1 |            1 | actionDate     | A         |          18 |     NULL | NULL   |      | BTREE      |         |               |
| WMeasureCube |          1 | WMeasureCube_idx1 |            2 | propertyCubeId | A         |    81680372 |     NULL | NULL   |      | BTREE      |         |               |
+--------------+------------+-------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+

你可以看到我一直在玩索引,但还没有找到神奇的组合。 WPropertyCube有~100K记录,WMeasureCube有~80M。我的疑问是:

mysql> explain SELECT wmc.actionDate,
    ->        SUM(wmc.measure7),
    ->        SUM(wmc.measure8),
    ->        SUM(wmc.measure9)
    -> FROM   WMeasureCube wmc
    ->        INNER JOIN WPropertyCube wpc
    ->          ON wmc.propertyCubeId = wpc.id
    -> WHERE  wpc.lineOfBusinessId IN ( 1, 2, 3, 4 )
    ->        AND wmc.actionDate BETWEEN '2010-06-28' AND '2010-09-26'
    -> GROUP  BY wmc.actionDate
    -> ORDER  BY wmc.actionDate;
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+----------------------------------------------+
| id | select_type | table | type   | possible_keys               | key     | key_len | ref                       | rows     | Extra                                        |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+----------------------------------------------+
|  1 | SIMPLE      | wmc   | ALL    | PRIMARY,WMeasureCube_idx1   | NULL    | NULL    | NULL                      | 81680372 | Using where; Using temporary; Using filesort |
|  1 | SIMPLE      | wpc   | eq_ref | PRIMARY,WPropertyCube_idx01 | PRIMARY | 8       | db.wmc.propertyCubeId     |        1 | Using where                                  |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+----------------------------------------------+

此查询将~30K记录聚合到~1,000条记录中。请注意,没有索引用于访问WMeasureCube,更糟糕的是,有一个文件流是slooooowww。查询时间范围在30到150秒之间。令人困惑的是,如果我删除了GROUP BY

mysql> explain SELECT wmc.actionDate,
    ->        SUM(wmc.measure7),
    ->        SUM(wmc.measure8),
    ->        SUM(wmc.measure9)
    -> FROM   WMeasureCube wmc
    ->        INNER JOIN WPropertyCube wpc
    ->          ON wmc.propertyCubeId = wpc.id
    -> WHERE  wpc.lineOfBusinessId IN ( 1, 2, 3, 4 )
    ->        AND wmc.actionDate BETWEEN '2010-06-28' AND '2010-09-26'
    -> ORDER  BY wmc.actionDate;
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+-------------+
| id | select_type | table | type   | possible_keys               | key     | key_len | ref                       | rows     | Extra       |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+-------------+
|  1 | SIMPLE      | wmc   | ALL    | PRIMARY,WMeasureCube_idx1   | NULL    | NULL    | NULL                      | 81680372 | Using where |
|  1 | SIMPLE      | wpc   | eq_ref | PRIMARY,WPropertyCube_idx01 | PRIMARY | 8       | db.wmc.propertyCubeId     |        1 | Using where |
+----+-------------+-------+--------+-----------------------------+---------+---------+---------------------------+----------+-------------+

仍然没有索引使用,但至少没有文件输入。此查询在不到一秒的时间内始终运行。特别奇怪的是,在执行时间方面,按一组列排序应该是一致的,以便按同一组进行分组。

我该怎么做才能加快查询速度?

我试图添加所有服务器变量,但它们占用了太多空间。所以,我添加了我认为最有帮助的那些。

mysql> show variables;
+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+
| Variable_name                                     | Value                                                                                                                      |
+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+
| big_tables                                        | OFF                                                                                                                        |
| binlog_cache_size                                 | 4194304                                                                                                                    |
| binlog_direct_non_transactional_updates           | OFF                                                                                                                        |
| binlog_format                                     | STATEMENT                                                                                                                  |
| bulk_insert_buffer_size                           | 8388608                                                                                                                    |
| character_set_database                            | utf8                                                                                                                       |
| character_set_filesystem                          | binary                                                                                                                     |
| default_storage_engine                            | InnoDB                                                                                                                     |
| innodb_adaptive_flushing                          | ON                                                                                                                         |
| innodb_adaptive_hash_index                        | ON                                                                                                                         |
| innodb_additional_mem_pool_size                   | 16777216                                                                                                                   |
| innodb_autoextend_increment                       | 8                                                                                                                          |
| innodb_autoinc_lock_mode                          | 1                                                                                                                          |
| innodb_buffer_pool_instances                      | 1                                                                                                                          |
| innodb_buffer_pool_size                           | 6442450944                                                                                                                 |
| innodb_change_buffering                           | all                                                                                                                        |
| innodb_checksums                                  | ON                                                                                                                         |
| innodb_commit_concurrency                         | 0                                                                                                                          |
| innodb_concurrency_tickets                        | 500                                                                                                                        |
| innodb_data_file_path                             | ibdata1:10M:autoextend                                                                                                     |
| innodb_data_home_dir                              |                                                                                                                            |
| innodb_doublewrite                                | ON                                                                                                                         |
| innodb_fast_shutdown                              | 1                                                                                                                          |
| innodb_file_format                                | Barracuda                                                                                                                  |
| innodb_file_format_check                          | ON                                                                                                                         |
| innodb_file_format_max                            | Barracuda                                                                                                                  |
| innodb_file_per_table                             | ON                                                                                                                         |
| innodb_flush_log_at_trx_commit                    | 0                                                                                                                          |
| innodb_flush_method                               |                                                                                                                            |
| innodb_force_recovery                             | 0                                                                                                                          |
| innodb_io_capacity                                | 200                                                                                                                        |
| innodb_lock_wait_timeout                          | 120                                                                                                                        |
| innodb_locks_unsafe_for_binlog                    | OFF                                                                                                                        |
| innodb_log_buffer_size                            | 16777216                                                                                                                   |
| innodb_log_file_size                              | 268435456                                                                                                                  |
| innodb_log_files_in_group                         | 3                                                                                                                          |
| innodb_log_group_home_dir                         | ./                                                                                                                         |
| innodb_max_dirty_pages_pct                        | 90                                                                                                                         |
| innodb_max_purge_lag                              | 0                                                                                                                          |
| innodb_mirrored_log_groups                        | 1                                                                                                                          |
| innodb_old_blocks_pct                             | 37                                                                                                                         |
| innodb_old_blocks_time                            | 0                                                                                                                          |
| innodb_open_files                                 | 300                                                                                                                        |
| innodb_purge_batch_size                           | 20                                                                                                                         |
| innodb_purge_threads                              | 0                                                                                                                          |
| innodb_read_ahead_threshold                       | 56                                                                                                                         |
| innodb_read_io_threads                            | 8                                                                                                                          |
| innodb_replication_delay                          | 0                                                                                                                          |
| innodb_rollback_on_timeout                        | OFF                                                                                                                        |
| innodb_spin_wait_delay                            | 6                                                                                                                          |
| innodb_stats_on_metadata                          | ON                                                                                                                         |
| innodb_stats_sample_pages                         | 8                                                                                                                          |
| innodb_strict_mode                                | OFF                                                                                                                        |
| innodb_support_xa                                 | OFF                                                                                                                        |
| innodb_sync_spin_loops                            | 30                                                                                                                         |
| innodb_table_locks                                | ON                                                                                                                         |
| innodb_thread_concurrency                         | 0                                                                                                                          |
| innodb_thread_sleep_delay                         | 10000                                                                                                                      |
| innodb_use_native_aio                             | ON                                                                                                                         |
| innodb_use_sys_malloc                             | ON                                                                                                                         |
| innodb_version                                    | 1.1.2                                                                                                                      |
| innodb_write_io_threads                           | 8                                                                                                                          |
| join_buffer_size                                  | 8388608                                                                                                                    |
| key_buffer_size                                   | 8388608                                                                                                                    |
| large_files_support                               | ON                                                                                                                         |
| large_page_size                                   | 0                                                                                                                          |
| large_pages                                       | OFF                                                                                                                        |
| last_insert_id                                    | 0                                                                                                                          |
| max_binlog_cache_size                             | 18446744073709547520                                                                                                       |
| max_binlog_size                                   | 1073741824                                                                                                                 |
| max_heap_table_size                               | 536870912                                                                                                                  |
| max_join_size                                     | 18446744073709551615                                                                                                       |
| max_length_for_sort_data                          | 1024                                                                                                                       |
| max_sort_length                                   | 1024                                                                                                                       |
| max_sp_recursion_depth                            | 0                                                                                                                          |
| max_tmp_tables                                    | 32                                                                                                                         |
| optimizer_prune_level                             | 1                                                                                                                          |
| optimizer_search_depth                            | 62                                                                                                                         |
| optimizer_switch                                  | index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on     |
| preload_buffer_size                               | 32768                                                                                                                      |
| profiling                                         | OFF                                                                                                                        |
| profiling_history_size                            | 15                                                                                                                         |
| protocol_version                                  | 10                                                                                                                         |
| pseudo_thread_id                                  | 13                                                                                                                         |
| query_alloc_block_size                            | 8192                                                                                                                       |
| query_cache_limit                                 | 8388608                                                                                                                    |
| query_cache_min_res_unit                          | 4096                                                                                                                       |
| query_cache_size                                  | 134217728                                                                                                                  |
| query_cache_type                                  | ON                                                                                                                         |
| query_cache_wlock_invalidate                      | OFF                                                                                                                        |
| query_prealloc_size                               | 8192                                                                                                                       |
| rand_seed1                                        | 0                                                                                                                          |
| rand_seed2                                        | 0                                                                                                                          |
| range_alloc_block_size                            | 4096                                                                                                                       |
| read_buffer_size                                  | 8388608                                                                                                                    |
| sort_buffer_size                                  | 16777216                                                                                                                   |
| sql_big_selects                                   | ON                                                                                                                         |
| sql_big_tables                                    | OFF                                                                                                                        |
| sql_max_join_size                                 | 18446744073709551615                                                                                                       |
| storage_engine                                    | InnoDB                                                                                                                     |
| tmp_table_size                                    | 536870912                                                                                                                  |
| tmpdir                                            | /dev/shm                                                                                                                   |
| tx_isolation                                      | REPEATABLE-READ                                                                                                            |
| version                                           | 5.5.6-rc-log                                                                                                               |
| version_comment                                   | MySQL Community Server (GPL)                                                                                               |
| version_compile_machine                           | x86_64                                                                                                                     |
| version_compile_os                                | linux2.6                                                                                                                   |
+---------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------+

[编辑]感谢Daniel的建议,我已将DATETIME字段更改为DATE。令人高兴的是,该指数现在正在使用。不幸的是,我们仍然在进行文件归档。

+----+-------------+-------+-------+-----------------------------+---------------------+---------+---------------+--------+-----------------------------------------------------------+
| id | select_type | table | type  | possible_keys               | key                 | key_len | ref           | rows   | Extra                                                     |
+----+-------------+-------+-------+-----------------------------+---------------------+---------+---------------+--------+-----------------------------------------------------------+
|  1 | SIMPLE      | wpc   | index | PRIMARY,WPropertyCube_idx01 | WPropertyCube_idx01 | 8       | NULL          | 377500 | Using where; Using index; Using temporary; Using filesort |
|  1 | SIMPLE      | wmc   | ref   | PRIMARY,WMeasureCube_idx1   | PRIMARY             | 8       | db.wpc.id     |     49 | Using where                                               |
+----+-------------+-------+-------+-----------------------------+---------------------+---------+---------------+--------+-----------------------------------------------------------+

1 个答案:

答案 0 :(得分:0)

我认为你的问题是datetime列actiondate。如果您不需要时间信息,请尝试将此列设置为日期列!那么分组应该更快,你的索引应该可以工作。