MySQL中的日期时间范围选择非常慢

时间:2017-11-05 10:27:26

标签: mysql sql database performance myisam

我的表posts有6.5M +记录。每个帖子都使用固定长度name表示。我使用MySQL Community 5.7,SSD磁盘大约10K-20K IOPS和1GB内存,key-buffer-size设置为512M(BTW主要是我使用默认MySQL配置驱动)。我的资源有限,因此我选择了MyISAM作为我的存储引擎。我的基准测试表明,在我的情况下,MyISAM更快。此外,我不太关心数据,因为它可以更新。

所以,这是我的计划信息:

+------------+--------+------------+
| TABLE_NAME | ENGINE | row_format |
+------------+--------+------------+
| posts      | MyISAM | Fixed      |
+------------+--------+------------+

+---------------------+---------------------+------+-----+---------+----------------+
| Field               | Type                | Null | Key | Default | Extra          |
+---------------------+---------------------+------+-----+---------+----------------+
| id                  | int(11) unsigned    | NO   | PRI | NULL    | auto_increment |
| name                | char(30)            | NO   | UNI | NULL    |                |
| worker_id           | tinyint(4) unsigned | NO   | MUL | NULL    |                |
| processing_priority | tinyint(4) unsigned | NO   | MUL | 0       |                |
| last_processed_at   | datetime            | YES  | MUL | NULL    |                |
| scraped_at          | datetime            | NO   | MUL | NULL    |                |
+---------------------+---------------------+------+-----+---------+----------------+

+-------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name            | Seq_in_index | Column_name         | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| posts |          0 | PRIMARY             |            1 | id                  | A         |     6579588 |     NULL | NULL   |      | BTREE      |         |               |
| posts |          0 | name                |            1 | name                | A         |     6579588 |     NULL | NULL   |      | BTREE      |         |               |
| posts |          1 | last_processed_at   |            1 | last_processed_at   | A         |     6579588 |     NULL | NULL   | YES  | BTREE      |         |               |
| posts |          1 | processing_priority |            1 | processing_priority | A         |           3 |     NULL | NULL   |      | BTREE      |         |               |
| posts |          1 | worker_id           |            1 | worker_id           | A         |          50 |     NULL | NULL   |      | BTREE      |         |               |
| posts |          1 | scraped_at          |            1 | scraped_at          | A         |      234985 |     NULL | NULL   |      | BTREE      |         |               |
+-------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+

我运行的查询:

SELECT COUNT(*) FROM `posts` WHERE `posts`.`worker_id` = 1 AND (last_processed_at >= '2017-11-04 22:20:27.203761')

MySQL需要3676.4ms才能执行此查询。

查询解释:

+----+-------------+-------+------------+------+-----------------------------+-----------+---------+-------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys               | key       | key_len | ref   | rows   | filtered | Extra       |
+----+-------------+-------+------------+------+-----------------------------+-----------+---------+-------+--------+----------+-------------+
|  1 | SIMPLE      | posts | NULL       | ref  | last_processed_at,worker_id | worker_id | 1       | const | 232621 |    37.45 | Using where |
+----+-------------+-------+------------+------+-----------------------------+-----------+---------+-------+--------+----------+-------------+

您对如何优化它有什么想法吗?

2 个答案:

答案 0 :(得分:3)

您可以使用worker_idlast_processed_at创建组合键,替换worker_id键。

答案 1 :(得分:0)

如果你的空间狭窄:

    对于可变长度>>> fmt = Account.objects.all()[0] >>> fmt.get_all_permissions() Traceback (most recent call last): File "<console>", line 1, in <module> File "C:\projects\Project\App\authentication\models.py", line 158, in get_all_permissions return _user_get_all_permissions(self, obj) File "C:\projects\python3-venv\lib\site-packages\django\contrib\auth\models.py", line 178, in _user_get_all_permissions permissions.update(backend.get_all_permissions(user, obj)) File "C:\projects\python3-venv\lib\site-packages\django\contrib\auth\backends.py", line 81, in get_all_permissions user_obj._perm_cache.update(self.get_group_permissions(user_obj)) File "C:\projects\python3-venv\lib\site-packages\django\contrib\auth\backends.py", line 74, in get_group_permissions return self._get_permissions(user_obj, obj, 'group') File "C:\projects\python3-venv\lib\site-packages\django\contrib\auth\backends.py", line 57, in _get_permissions perms = getattr(self, '_get_%s_permissions' % from_name)(user_obj) File "C:\projects\python3-venv\lib\site-packages\django\contrib\auth\backends.py", line 41, in _get_group_permissions return Permission.objects.filter(**{user_groups_query: user_obj}) File "C:\projects\python3-venv\lib\site-packages\django\db\models\manager.py", line 85, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "C:\projects\python3-venv\lib\site-packages\django\db\models\query.py", line 784, in filter return self._filter_or_exclude(False, *args, **kwargs) File "C:\projects\python3-venv\lib\site-packages\django\db\models\query.py", line 802, in _filter_or_exclude clone.query.add_q(Q(*args, **kwargs)) File "C:\projects\python3-venv\lib\site-packages\django\db\models\sql\query.py", line 1261, in add_q clause, _ = self._add_q(q_object, self.used_aliases) File "C:\projects\python3-venv\lib\site-packages\django\db\models\sql\query.py", line 1287, in _add_q allow_joins=allow_joins, split_subq=split_subq, File "C:\projects\python3-venv\lib\site-packages\django\db\models\sql\query.py", line 1190, in build_filter self.check_related_objects(field, value, opts) File "C:\projects\python3-venv\lib\site-packages\django\db\models\sql\query.py", line 1087, in check_related_objects self.check_query_object_type(value, opts, field) File "C:\projects\python3-venv\lib\site-packages\django\db\models\sql\query.py", line 1065, in check_query_object_type (value, opts.object_name)) ValueError: Cannot query "fmt@somewhere.com": Must be "Group" instance. >>> fmt.get_all_permissions() set()
  • ValueError可能会使您的.MYD文件比Ascii或latin1所需的大50%,或者3 所需的大小。 UTF8。
  • 不要盲目索引每一列;这会浪费.MYI文件中的空间。特别是,CHAR(30)可能从未使用过。

让我们看看names和一些查询,以进一步批评空间设置。

1GB的RAM,512MB的INDEX(processing_priority)非常糟糕。首先,key_buffer仅用于缓存索引块(每个1KB)。其次,您需要有空间来缓存数据,操作系统会这样做。第三,您需要代码和其他数据的空间。建议不超过SHOW CREATE TABLE

(至于你的实际问题,@ Turo给出了一个很好的答案。)但是你删除了现在多余的key_buffer吗? (这将节省大约70MB。)

有关索引创建的更多信息:http://mysql.rjweb.org/doc.php/index_cookbook_mysql