将AUTO_INCREMENT id列添加到大型MySQL表时出现问题

时间：2015-05-25 21:39:25

标签： mysql database import

尝试失败1

我试图将一个巨大的SQL文件（包含600万条记录和239 GB）导入到具有不同模式的MySQL数据库中，以便获得一个带有“自动生成ID”的新列。

我在Windows上使用MySQL Workbench 6.2，但在导入数千万条记录后，它总是失败并显示错误消息“MySQL服务器已经消失”。

我使用“服务器 - ＆gt;数据导入”功能进行导入。

我尝试添加

wait_timeout = 288000
max_allowed_packet = 300G

进入MySQL Server文件夹内的my.ini，但发生了同样的错误。

错误日志如下：

14:16:37 Restoring C:\data.sql
Running: mysql.exe --defaults-file="c:\users\***\appdata\local\temp\5\tmpcfyyxk.cnf"  --protocol=tcp --host=127.0.0.1 --user=shodan --port=8080 --default-character-set=utf8 --comments --database=test2 < "C:\\data.sql"
ERROR 2006 (HY000) at line 13776162: MySQL server has gone away

Operation failed with exitcode 1
16:25:51 Import of C:\data.sql has finished with 1 errors

尝试失败2

如果我只是在原始表中添加一个新列，那么执行SQL查询只需要太长时间就不会成功。

如果导入问题无法解决，有没有办法直接将具有自动生成ID的新列添加到巨大的表中？

3 个答案:

答案 0 :(得分：2)

我建议使用这种方法，而不是你一直在尝试的解决方案：

  1. 创建一个复制旧表的新表+添加auto_increment id，
  2. 将旧表中的数据复制到新表中，然后
  3. 重命名表格，以便新表格替换旧表格。

    4. 我们假设您的 C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.6.2-x64-mingw32/lib/nokogiri .rb:29:in `require': cannot load such file -- nokogiri/nokogiri (LoadError) from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.6.2-x64-mingw32 /lib/nokogiri.rb:29:in `rescue in <top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/nokogiri-1.6.6.2-x64-mingw32 /lib/nokogiri.rb:25:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/loofah-2.0.2/lib/loofah.rb:3 :in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/loofah-2.0.2/lib/loofah.rb:3 :in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rails-html-sanitizer-1.0.2/l ib/rails-html-sanitizer.rb:2:in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/rails-html-sanitizer-1.0.2/l ib/rails-html-sanitizer.rb:2:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/sanitize_helper.rb:3:in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/sanitize_helper.rb:3:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/text_helper.rb:32:in `<module:TextHelper>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/text_helper.rb:29:in `<module:Helpers>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/text_helper.rb:6:in `<module:ActionView>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/text_helper.rb:4:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/form_tag_helper.rb:18:in `<module:FormTagHelper>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/form_tag_helper.rb:14:in `<module:Helpers>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/form_tag_helper.rb:8:in `<module:ActionView>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/form_tag_helper.rb:6:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/form_helper.rb:4:in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers/form_helper.rb:4:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers.rb:50:in `<module:Helpers>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers.rb:4:in `<module:ActionView>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/actionview-4.2.1/lib/action_ view/helpers.rb:3:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sprockets-rails-2.3.0/lib/sp rockets/rails/legacy_asset_tag_helper.rb:7:in `<module:LegacyAssetTagHelper>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sprockets-rails-2.3.0/lib/sp rockets/rails/legacy_asset_tag_helper.rb:6:in `<module:Rails>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sprockets-rails-2.3.0/lib/sp rockets/rails/legacy_asset_tag_helper.rb:4:in `<module:Sprockets>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sprockets-rails-2.3.0/lib/sp rockets/rails/legacy_asset_tag_helper.rb:3:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sprockets-rails-2.3.0/lib/sp rockets/rails/helper.rb:54:in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sprockets-rails-2.3.0/lib/sp rockets/rails/helper.rb:54:in `<module:Helper>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sprockets-rails-2.3.0/lib/sp rockets/rails/helper.rb:7:in `<module:Rails>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sprockets-rails-2.3.0/lib/sp rockets/rails/helper.rb:6:in `<module:Sprockets>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sprockets-rails-2.3.0/lib/sp rockets/rails/helper.rb:5:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sprockets-rails-2.3.0/lib/sp rockets/railtie.rb:6:in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sprockets-rails-2.3.0/lib/sp rockets/railtie.rb:6:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sass-rails-5.0.3/lib/sass/ra ils/railtie.rb:3:in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sass-rails-5.0.3/lib/sass/ra ils/railtie.rb:3:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sass-rails-5.0.3/lib/sass/ra ils.rb:11:in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sass-rails-5.0.3/lib/sass/ra ils.rb:11:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sass-rails-5.0.3/lib/sass-ra ils.rb:1:in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/sass-rails-5.0.3/lib/sass-ra ils.rb:1:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/bundler-1.9.6/lib/bundler/ru ntime.rb:76:in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/bundler-1.9.6/lib/bundler/ru ntime.rb:76:in `block (2 levels) in require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/bundler-1.9.6/lib/bundler/ru ntime.rb:72:in `each' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/bundler-1.9.6/lib/bundler/ru ntime.rb:72:in `block in require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/bundler-1.9.6/lib/bundler/ru ntime.rb:61:in `each' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/bundler-1.9.6/lib/bundler/ru ntime.rb:61:in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/bundler-1.9.6/lib/bundler.rb :134:in `require' from C:/RubyProjects/app/config/application.rb:7:in `<top (required)>' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/railties-4.2.1/lib/rails/com mands/commands_tasks.rb:78:in `require' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/railties-4.2.1/lib/rails/com mands/commands_tasks.rb:78:in `block in server' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/railties-4.2.1/lib/rails/com mands/commands_tasks.rb:75:in `tap' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/railties-4.2.1/lib/rails/com mands/commands_tasks.rb:75:in `server' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/railties-4.2.1/lib/rails/com mands/commands_tasks.rb:39:in `run_command!' from C:/Ruby22-x64/lib/ruby/gems/2.2.0/gems/railties-4.2.1/lib/rails/com mands.rb:17:in `<top (required)>' from bin/rails:4:in `require' from bin/rails:4:in `<main>' 的列current_table（当前是您的主键）和col1 VARCHAR(255) NOT NULL，这只是其他一些数据。

    col2 VARCHAR(255) NULL

    这应该比简单地添加列或导入到新数据库更好。它还将维护当前的表结构并创建替换，以便在出现问题时进行后备。当您验证新表结构按预期工作时，可以删除旧表。如果失败，您可以删除新表并回滚到旧表。

答案 1 :(得分：0)

检查MySQL错误日志以查看服务器是否因某些原因崩溃并重新启动： https://dev.mysql.com/doc/refman/5.6/en/error-log.html

如果不是这种情况，最可能的原因是max_allowed_pa​​cket。您希望在 服务器和客户端上将其更改为1G。你可以这样做。

  1. 更改服务器上的设置：

    SET GLOBAL max_allowed_pa​​cket_size = 1 * 1024 * 1024 * 1024;

  2. 将--max_allowed_pa​​cket = 1G添加到您的命令中：

    mysql.exe --defaults-file =＆＃34; c：\ users *** \ appdata \ local \ temp \ 5 \ tmpcfyyxk.cnf＆＃34; --protocol = tcp --max_allowed_pa​​cket = 1G --host = 127.0.0.1 --user = shodan --port = 8080 --default-character-set = utf8 --comments --database = test2＆lt; ＆＃34; C：\ data.sql＆＃34;

答案 2 :(得分：0)

最好使用 mysqldump ＆amp; mysql 命令行实用程序或

SELECT ... INTO OUTFILE

和

LOAD DATA INFILE

进行此类操作。只需在mysql服务器上直接执行，而不是远程执行。

https://dev.mysql.com/doc/refman/5.6/en/load-data.html

如果您只需要更改该表以添加autoinc列，请查看 pt-online-schema-change 。如果您不需要100％的正常运行时间，只需使用经典的ALTER TABLE语句即可完成，没有任何问题。你只需要足够的可用磁盘空间。

相关问题
最新问题