可以在MySQL的BEFORE TRIGGER中安全地使用AUTO_INCREMENT

时间:2014-09-05 01:49:37

标签: mysql instagram auto-increment sharding triggers

Instagram的Postgres为Sharding实现自定义ID的方法很棒,但我需要在MySQL中实现。

所以,我转换了这个博客底部的方法,在这里:http://instagram-engineering.tumblr.com/post/10853187575/sharding-ids-at-instagram

MySQL版本:

CREATE TRIGGER shard_insert BEFORE INSERT ON tablename
FOR EACH ROW BEGIN

DECLARE seq_id BIGINT;
DECLARE now_millis BIGINT;
DECLARE our_epoch BIGINT DEFAULT 1314220021721;
DECLARE shard_id INT DEFAULT 1;

SET now_millis = (SELECT UNIX_TIMESTAMP(NOW(3)) * 1000);
SET seq_id = (SELECT AUTO_INCREMENT FROM information_schema.TABLES WHERE TABLE_SCHEMA = "dbname" AND TABLE_NAME = "tablename");
SET NEW.id = (SELECT ((now_millis - our_epoch) << 23) | (shard_id << 10) | (SELECT MOD(seq_id, 1024)));
END

该表大致如下:

CREATE TABLE tablename (
    id BIGINT AUTO_INCREMENT,
    ...
)

问题:

  1. 这里存在并发问题。当产生100个线程并运行插入时,我得到重复的序列值,这意味着两个触发器获得相同的auto_increment值。我该如何解决这个问题?
  2. 我尝试创建一个新表格,例如&#34; tablename_seq&#34;,有一行,一个用于存储我自己的auto_increment值的计数器,然后在TRIGGER中对该表进行更新,但问题是我无法在存储过程中锁定表(触发器),所以我有完全相同的问题,我不能保证计数器在触发器之间是唯一的:(。

    我很难过,真的很感激任何提示!

    可能的解决方案:

    1. MySQL 5.6具有UUID_SHORT(),它生成唯一的增量值,保证它们是唯一的。实际上,在调用它时,每次调用都会增加值+1。通过使用:SET seq_id =(SELECT UUID_SHORT());它似乎删除了并发问题。这样做的副作用是,现在(大致)整个系统中每毫秒不会超过1024次插入。如果做得更多,则可能出现DUPLICATE PRIMARY KEY错误。好消息是,在我的机器上的基准测试中,我得到〜3,000个插入/秒,或者没有触发器连接UUID_SHORT(),所以它似乎根本不会减慢它。

2 个答案:

答案 0 :(得分:2)

以下SQL Fiddle生成如下所示的输出:

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 45
Server version: 5.5.35-1

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> select `id` from `tablename`;
+-------------------+
| id                |
+-------------------+
| 11829806563853313 |
| 11829806563853314 |
| 11829806563853315 |
| 11829806563853316 |
| 11829806563853317 |
| 11829806563853318 |
| 11829806563853319 |
| 11829806563853320 |
| 11829806563853321 |
| 11829806563853322 |
| 11829806563853323 |
| 11829806563853324 |
| 11829806563853325 |
| 11829806563853326 |
| 11829806563853327 |
| 11829806563853328 |
| 11829806563853329 |
| 11829806563853330 |
| 11829806563853331 |
| 11829806563853332 |
| 11829806563853333 |
| 11829806563853334 |
| 11829806563853335 |
| 11829806563853336 |
| 11829806563853337 |
| 11829806563853338 |
| 11829806563853339 |
| 11829806563853340 |
| 11829806563853341 |
| 11829806563853342 |
| 11829806563853343 |
| 11829806563853344 |
| 11829806563853345 |
| 11829806563853346 |
| 11829806563853347 |
| 11829806563853348 |
| 11829806563853349 |
| 11829806563853350 |
| 11829806563853351 |
| 11829806563853352 |
+-------------------+
40 rows in set (0.01 sec)

如果能真正解决您的需求,请接受答案。

<强>更新

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 46
Server version: 5.5.35-1

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> use test;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Database changed
mysql> DELIMITER //

mysql> DROP FUNCTION IF EXISTS `nextval`//
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> DROP TRIGGER IF EXISTS `shard_insert`//
Query OK, 0 rows affected (0.00 sec)

mysql> DROP TABLE IF EXISTS `tablename_seq`, `tablename`;
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE TABLE `tablename_seq` (
    ->   `seq` BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY
    -> )//
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE TABLE `tablename` (
    ->   `id` BIGINT UNSIGNED PRIMARY KEY
    -> )//
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE FUNCTION `nextval`()
    -> RETURNS BIGINT UNSIGNED
    -> DETERMINISTIC
    -> BEGIN
    ->   DECLARE `_last_insert_id` BIGINT UNSIGNED;
    ->   INSERT INTO `tablename_seq` VALUES (NULL);
    ->   SET `_last_insert_id` := LAST_INSERT_ID();
    ->   DELETE FROM `tablename_seq`
    ->   WHERE `seq` = `_last_insert_id`;
    ->   RETURN `_last_insert_id`;
    -> END//
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE TRIGGER `shard_insert` BEFORE INSERT ON `tablename`
    -> FOR EACH ROW
    -> BEGIN
    ->   DECLARE `seq_id`, `now_millis` BIGINT UNSIGNED;
    ->   DECLARE `our_epoch` BIGINT UNSIGNED DEFAULT 1314220021721;
    ->   DECLARE `shard_id` INT UNSIGNED DEFAULT 1;
    ->   SET `now_millis` := `our_epoch` + UNIX_TIMESTAMP();
    ->   SET `seq_id` := `nextval`();
    ->   SET NEW.`id` := (SELECT (`now_millis` - `our_epoch`) << 23 |
    ->                            `shard_id` << 10 |
    ->                            MOD(`seq_id`, 1024)
    ->                   );
    -> END//
Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO `tablename`
    -> VALUES
    -> (0), (0), (0), (0), (0),
    -> (0), (0), (0), (0), (0),
    -> (0), (0), (0), (0), (0),
    -> (0), (0), (0), (0), (0),
    -> (0), (0), (0), (0), (0),
    -> (0), (0), (0), (0), (0),
    -> (0), (0), (0), (0), (0),
    -> (0), (0), (0), (0), (0)//
Query OK, 40 rows affected (0.00 sec)
Records: 40  Duplicates: 0  Warnings: 0

mysql> DELIMITER ;

mysql> SELECT `id` FROM `tablename`;
+-------------------+
| id                |
+-------------------+
| 12581084357198849 |
| 12581084357198850 |
| 12581084357198851 |
| 12581084357198852 |
| 12581084357198853 |
| 12581084357198854 |
| 12581084357198855 |
| 12581084357198856 |
| 12581084357198857 |
| 12581084357198858 |
| 12581084357198859 |
| 12581084357198860 |
| 12581084357198861 |
| 12581084357198862 |
| 12581084357198863 |
| 12581084357198864 |
| 12581084357198865 |
| 12581084357198866 |
| 12581084357198867 |
| 12581084357198868 |
| 12581084357198869 |
| 12581084357198870 |
| 12581084357198871 |
| 12581084357198872 |
| 12581084357198873 |
| 12581084357198874 |
| 12581084357198875 |
| 12581084357198876 |
| 12581084357198877 |
| 12581084357198878 |
| 12581084357198879 |
| 12581084357198880 |
| 12581084357198881 |
| 12581084357198882 |
| 12581084357198883 |
| 12581084357198884 |
| 12581084357198885 |
| 12581084357198886 |
| 12581084357198887 |
| 12581084357198888 |
+-------------------+
40 rows in set (0.00 sec)

请参阅db-fiddle

答案 1 :(得分:2)

另一种方法是获取自动增量数字块。如果将MySQL auto increment increment设置为1000,则进程可以在“序列”表中执行插入并获取自动增量值。然后,该进程知道它可以使用1000个连续数字,从该数字开始,将没有冲突。如果你要录制的只是一个数字,则无需在中央表中记录每个增量。

除了自动增量偏移之外,这在多个主设置中最常用。你也可以去多个主路线,并插入不同的主人。自动增量增量和偏移量将确保没有冲突。这需要扎实的MySQL复制知识。