我的表posts
有6.5M +记录。每个帖子都使用固定长度name
表示。我使用MySQL Community 5.7,SSD磁盘大约10K-20K IOPS和1GB内存,key-buffer-size
设置为512M(BTW主要是我使用默认MySQL配置驱动)。我的资源有限,因此我选择了MyISAM作为我的存储引擎。我的基准测试表明,在我的情况下,MyISAM更快。此外,我不太关心数据,因为它可以更新。
所以,这是我的计划信息:
+------------+--------+------------+
| TABLE_NAME | ENGINE | row_format |
+------------+--------+------------+
| posts | MyISAM | Fixed |
+------------+--------+------------+
+---------------------+---------------------+------+-----+---------+----------------+
| Field | Type | Null | Key | Default | Extra |
+---------------------+---------------------+------+-----+---------+----------------+
| id | int(11) unsigned | NO | PRI | NULL | auto_increment |
| name | char(30) | NO | UNI | NULL | |
| worker_id | tinyint(4) unsigned | NO | MUL | NULL | |
| processing_priority | tinyint(4) unsigned | NO | MUL | 0 | |
| last_processed_at | datetime | YES | MUL | NULL | |
| scraped_at | datetime | NO | MUL | NULL | |
+---------------------+---------------------+------+-----+---------+----------------+
+-------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| posts | 0 | PRIMARY | 1 | id | A | 6579588 | NULL | NULL | | BTREE | | |
| posts | 0 | name | 1 | name | A | 6579588 | NULL | NULL | | BTREE | | |
| posts | 1 | last_processed_at | 1 | last_processed_at | A | 6579588 | NULL | NULL | YES | BTREE | | |
| posts | 1 | processing_priority | 1 | processing_priority | A | 3 | NULL | NULL | | BTREE | | |
| posts | 1 | worker_id | 1 | worker_id | A | 50 | NULL | NULL | | BTREE | | |
| posts | 1 | scraped_at | 1 | scraped_at | A | 234985 | NULL | NULL | | BTREE | | |
+-------+------------+---------------------+--------------+---------------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
我运行的查询:
SELECT COUNT(*) FROM `posts` WHERE `posts`.`worker_id` = 1 AND (last_processed_at >= '2017-11-04 22:20:27.203761')
MySQL需要3676.4ms才能执行此查询。
查询解释:
+----+-------------+-------+------------+------+-----------------------------+-----------+---------+-------+--------+----------+-------------+
| id | select_type | table | partitions | type | possible_keys | key | key_len | ref | rows | filtered | Extra |
+----+-------------+-------+------------+------+-----------------------------+-----------+---------+-------+--------+----------+-------------+
| 1 | SIMPLE | posts | NULL | ref | last_processed_at,worker_id | worker_id | 1 | const | 232621 | 37.45 | Using where |
+----+-------------+-------+------------+------+-----------------------------+-----------+---------+-------+--------+----------+-------------+
您对如何优化它有什么想法吗?
答案 0 :(得分:3)
您可以使用worker_id
和last_processed_at
创建组合键,替换worker_id
键。
答案 1 :(得分:0)
如果你的空间狭窄:
>>> fmt = Account.objects.all()[0]
>>> fmt.get_all_permissions()
Traceback (most recent call last):
File "<console>", line 1, in <module>
File "C:\projects\Project\App\authentication\models.py", line 158, in get_all_permissions
return _user_get_all_permissions(self, obj)
File "C:\projects\python3-venv\lib\site-packages\django\contrib\auth\models.py", line 178, in _user_get_all_permissions
permissions.update(backend.get_all_permissions(user, obj))
File "C:\projects\python3-venv\lib\site-packages\django\contrib\auth\backends.py", line 81, in get_all_permissions
user_obj._perm_cache.update(self.get_group_permissions(user_obj))
File "C:\projects\python3-venv\lib\site-packages\django\contrib\auth\backends.py", line 74, in get_group_permissions
return self._get_permissions(user_obj, obj, 'group')
File "C:\projects\python3-venv\lib\site-packages\django\contrib\auth\backends.py", line 57, in _get_permissions
perms = getattr(self, '_get_%s_permissions' % from_name)(user_obj)
File "C:\projects\python3-venv\lib\site-packages\django\contrib\auth\backends.py", line 41, in _get_group_permissions
return Permission.objects.filter(**{user_groups_query: user_obj})
File "C:\projects\python3-venv\lib\site-packages\django\db\models\manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "C:\projects\python3-venv\lib\site-packages\django\db\models\query.py", line 784, in filter
return self._filter_or_exclude(False, *args, **kwargs)
File "C:\projects\python3-venv\lib\site-packages\django\db\models\query.py", line 802, in _filter_or_exclude
clone.query.add_q(Q(*args, **kwargs))
File "C:\projects\python3-venv\lib\site-packages\django\db\models\sql\query.py", line 1261, in add_q
clause, _ = self._add_q(q_object, self.used_aliases)
File "C:\projects\python3-venv\lib\site-packages\django\db\models\sql\query.py", line 1287, in _add_q
allow_joins=allow_joins, split_subq=split_subq,
File "C:\projects\python3-venv\lib\site-packages\django\db\models\sql\query.py", line 1190, in build_filter
self.check_related_objects(field, value, opts)
File "C:\projects\python3-venv\lib\site-packages\django\db\models\sql\query.py", line 1087, in check_related_objects
self.check_query_object_type(value, opts, field)
File "C:\projects\python3-venv\lib\site-packages\django\db\models\sql\query.py", line 1065, in check_query_object_type
(value, opts.object_name))
ValueError: Cannot query "fmt@somewhere.com": Must be "Group" instance.
>>> fmt.get_all_permissions()
set()
,ValueError
可能会使您的.MYD文件比Ascii或latin1所需的大50%,或者3 次所需的大小。 UTF8。CHAR(30)
可能从未使用过。让我们看看names
和一些查询,以进一步批评空间设置。
1GB的RAM,512MB的INDEX(processing_priority)
非常糟糕。首先,key_buffer仅用于缓存索引块(每个1KB)。其次,您需要有空间来缓存数据,操作系统会这样做。第三,您需要代码和其他数据的空间。建议不超过SHOW CREATE TABLE
。
(至于你的实际问题,@ Turo给出了一个很好的答案。)但是你删除了现在多余的key_buffer
吗? (这将节省大约70MB。)
有关索引创建的更多信息:http://mysql.rjweb.org/doc.php/index_cookbook_mysql