Question

我需要重构我的MYSQL InnoDB数据库。

目前，我有一个customer表，其中包含3个产品名称。

我需要将这些名称提取到新的product表中。 product表应保存当前保存在customer表中的每个名称，并通过新的customer表链接到customer_product表。虽然产品名称可能不是唯一的，但它们彼此之间没有任何关系，这意味着每个customer都需要在product表中插入3个新条目和3个新条目进入customer_product表。

所以不要这样：

customer
| id | product_name_a | product_name_b | product_name_c |

我需要这个：

customer
| id |

customer_product
| customer_id | product_id | X3

product
| id | name | X3

我编写了以下有效的MYSQL程序：

BEGIN
  DECLARE nbr_of_customers BIGINT(20);
  DECLARE customer_count BIGINT(20);
  DECLARE product_id BIGINT(20);
  DECLARE customer_id BIGINT(20);
  DECLARE product_name_a VARCHAR(500);
  DECLARE product_name_b VARCHAR(500);
  DECLARE product_name_c VARCHAR(500);

  SELECT COUNT(*) FROM customer INTO nbr_of_customers;
  SET customer_count = 0;
  SET product_id = 1;

  WHILE customer_count < nbr_of_customers DO
    SELECT
      customer.id,
      customer.product_name_a,
      customer.product_name_b,
      customer.product_name_c
    INTO
      customer_id,
      product_name_a,
      product_name_b,
      product_name_c
    FROM customer
    LIMIT customer_count,1;

    INSERT INTO product(id, name)
      VALUES(product_id, product_name_a);
    INSERT INTO customer_product(customer_id, product_id)
      VALUES(customer_id, product_id);
    SET product_id = product_id + 1;

    INSERT INTO product(id, name)
      VALUES(product_id, product_name_b);
    INSERT INTO customer_product(customer_id, product_id)
      VALUES(customer_id, product_id);
    SET product_id = product_id + 1;

    INSERT INTO product(id, name)
      VALUES(product_id, product_name_c);
    INSERT INTO customer_product(customer_id, product_id)
      VALUES(customer_id, product_id);
    SET product_id = product_id + 1;

    SET customer_count = customer_count + 1;
  END WHILE;
END;

这太慢了。

我在本地运行并估计我的~15k客户需要大约1小时才能完成。而我的VPS服务器速度远远低于此，因此可能需要10小时才能完成。

问题似乎是插入需要很长时间。因此，我想在程序中存储所有插入，并在循环完成后批量执行它们，我知道要插入什么。

我有办法批量执行所有~100k插入以优化性能，还是有更好的方法来实现它？

最终编辑：

我标记了正确的解决方案，因为它很好地加速了这个过程，这是问题的主要焦点。最后，我最终使用修改后的生产代码（在Java中）执行迁移，因为解决方案有关不转义插入字符串的限制。

Answer 1

也许您可以在三个单独的插入（而不是~100K）中执行此操作，如下所示：

INSERT INTO customer_product (customer_id, product_id) 
SELECT customer.id as customer_id, product.id as product_id 
FROM customer 
  JOIN product on customer.product_name_a = product.name

INSERT INTO customer_product (customer_id, product_id) 
SELECT customer.id as customer_id, product.id as product_id 
FROM customer 
  JOIN product on customer.product_name_b = product.name


INSERT INTO customer_product (customer_id, product_id) 
SELECT customer.id as customer_id, product.id as product_id 
FROM customer 
  JOIN product on customer.product_name_c = product.name

当然，您必须提前设置product表，并且您希望事后从您的customer表中删除非规范化列。

如果您在customer.product_name_X列（可能还有product.name列上创建索引，可以进一步加快这一点，尽管它很少，如果它很重要则为idk）。 EXPLAIN可以提供帮助。

Answer 2

首先，使用游标处理单个查询的结果，而不是为每一行执行单独的查询。

然后将VALUES列表连接到您使用PREPARE和EXECUTE执行的字符串。

我的代码会批量插入100个客户，因为我希望查询的大小有限制。

BEGIN
  DECLARE product_id BIGINT(20);
  DECLARE customer_id BIGINT(20);
  DECLARE product_name_a VARCHAR(500);
  DECLARE product_name_b VARCHAR(500);
  DECLARE product_name_c VARCHAR(500);
  DECLARE done INT DEFAULT FALSE;
  DECLARE cur CURSOR FOR SELECT c.id, c.product_name_a, c.product_name_b, c.product_name_c FROM customer AS c;
  DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = TRUE;

  SET product_id = 1;

  OPEN cur;

  SET @product_values = '';
  SET @cp_values = '';

  read_loop: LOOP
    FETCH cur INTO customer_id, product_name_a, product_name_b, product_name_c;
    IF done THEN
      LEAVE read_loop;
    END IF;

    SET @product_values = CONCAT(@product_values, IF(@product_values != '', ',', ''), "(", product_id, ",'", product_name_a, "'), (", product_id + 1, ",'", product_name_b, "'), (", product_id + 2, ",'", product_name_c, "'), ");
    SET @cp_values = CONCAT(@cp_values, IF(@cp_values != '', ',', ''), "(", customer_id, ",", product_id, "), (", customer_id, ",", product_id + 1, "), (", customer_id, ",", product_id + 2, "),");

    SET product_id = product_id + 3;

    IF product_id % 300 = 1 -- insert every 100 customers
    THEN BEGIN
         SET @insert_product = CONCAT("INSERT INTO product(id, name) VALUES ", @product_values);
         PREPARE stmt1 FROM @insert_product;
         EXECUTE stmt1;

         SET @insert_cp = CONCAT("INSERT INTO customer_product(customer_id, product_id) VALUES ", @cp_values);
         PREPARE stmt2 FROM @insert_cp;
         EXECUTE stmt2;

         SET @product_values = '';
         SET @cp_values = '';
     END IF;

  END LOOP;

  IF @product_values != '' -- Process any remaining rows
  THEN BEGIN
       SET @insert_product = CONCAT("INSERT INTO product(id, name) VALUES ", @product_values);
       PREPARE stmt1 FROM @insert_product;
       EXECUTE stmt1;

       SET @insert_cp = CONCAT("INSERT INTO customer_product(customer_id, product_id) VALUES ", @cp_values);
       PREPARE stmt2 FROM @insert_cp;
       EXECUTE stmt2;

       SET @product_values = '';
       SET @cp_values = '';
   END IF;
END;

请注意，使用此解决方案，在插入之前，产品名称将无法正确转义。 如果任何产品名称包含特殊字符，例如单引号'，则此解决方案将无效。

批量MYSQL插入用于DB结构迁移中的性能

2 个答案: