我有一个大的(近10M记录)数据表,出于性能原因,它有一个辅助聚合伴随表。聚合表定期填充sofar unaggregated数据:
REPLACE INTO aggregate (channel_id, type, timestamp, value, count)
SELECT channel_id, 'day' AS type, MAX(timestamp) AS timestamp, SUM(value) AS value, COUNT(timestamp) AS count FROM data
WHERE timestamp < UNIX_TIMESTAMP(DATE_FORMAT(NOW(), "%Y-%m-%d")) * 1000
AND timestamp >= IFNULL((SELECT UNIX_TIMESTAMP(DATE_ADD(FROM_UNIXTIME(MAX(timestamp)/1000, "%Y-%m-%d"),
INTERVAL 1 day)) * 1000 FROM aggregate WHERE type = 'day'), 0)
GROUP BY channel_id, YEAR(FROM_UNIXTIME(timestamp/1000)), DAYOFYEAR(FROM_UNIXTIME(timestamp/1000));
我发现语句的SELECT
部分非常慢(快速PC上2秒以上),即使没有返回数据也是如此。由于聚合需要在嵌入式设备上运行,因此这是一个问题。这是计划:
id select_type table type key key_len rows Extra
1 PRIMARY data ALL 9184560 Using where; Using temporary; Using filesort
2 SUBQUERY aggregate index ts_uniq 22 1940 Using where; Using index
子查询本身是即时的。由于data
子句中的计算,显然channel_id/timestamp
不使用GROUP BY
索引:
CREATE TABLE `data` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`channel_id` int(11) DEFAULT NULL,
`timestamp` bigint(20) NOT NULL,
`value` double NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `ts_uniq` (`channel_id`,`timestamp`),
KEY `IDX_ADF3F36372F5A1AA` (`channel_id`)
) ENGINE=MyISAM AUTO_INCREMENT=10432870 DEFAULT CHARSET=latin1;
可以进一步优化查询吗?
更新:添加所需信息
SHOW INDEXES FROM data;
Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Null Index_type
data 0 PRIMARY 1 id A 9184560 BTREE
data 0 ts_uniq 1 channel_id A 164 YES BTREE
data 0 ts_uniq 2 timestamp A 9184560 BTREE
data 1 IDX_ADF3.. 1 channel_id A 164 YES BTREE
CREATE TABLE `aggregate` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`channel_id` int(11) NOT NULL,
`type` varchar(8) NOT NULL,
`timestamp` bigint(20) NOT NULL,
`value` double NOT NULL,
`count` int(11) NOT NULL,
PRIMARY KEY (`id`),
UNIQUE KEY `ts_uniq` (`channel_id`,`type`,`timestamp`)
) ENGINE=MyISAM AUTO_INCREMENT=1941 DEFAULT CHARSET=latin1;
我还注意到,当将GROUP BY更改为channel_id时间戳时,查询会立即生效。不幸的是,将数据计算添加为列是不可取的,因为分组是动态计算的。
当我甚至没有任何要分组的数据时,我无法理解为什么GROUP BY
索引应该是一个问题。我试过了
SELECT channel_id, 'day' AS type, MAX(timestamp) AS timestamp, SUM(value) AS value, COUNT(timestamp) AS count FROM data
WHERE timestamp < UNIX_TIMESTAMP(DATE_FORMAT(NOW(), "%Y-%m-%d")) * 1000
AND timestamp >= IFNULL((SELECT UNIX_TIMESTAMP(DATE_ADD(FROM_UNIXTIME(MAX(timestamp)/1000, "%Y-%m-%d"), INTERVAL 1 day)) * 1000
FROM aggregate WHERE type = 'day'), 0)
这同样慢,所以GROUP
似乎不是问题吗?
更新2
进一步向下挖掘道路
SELECT channel_id, 'day' AS type, timestamp, value, 1 FROM data
WHERE timestamp >= (SELECT UNIX_TIMESTAMP(DATE_ADD(FROM_UNIXTIME(MAX(timestamp)/1000, "%Y-%m-%d"),
INTERVAL 1 day)) * 1000 FROM aggregate WHERE type = 'day');
仍然很慢(1.4秒) - 所以根本不是GROUP BY
问题。
更新3
这仍然很慢:
SELECT channel_id, 'day' AS type, timestamp, value, 1 FROM data WHERE timestamp >= 1380837600000;
所以 - 问题是内部比较是针对时间戳的,它不能使用channel_id,timestamp索引,尽管这是GROUP BY
子句的一部分。
这导致了如何强制该指数的问题?
答案 0 :(得分:1)
将年和年日列添加到数据表,并在(channel_id,year,dayofyear)上设置索引。插入行时填充两个新列。