使用AWS Data Migration Service(DMS)从Heroku PostgreSQL迁移到AWS RDS无法正常工作

时间:2017-10-25 18:01:40

标签: postgresql heroku amazon-rds heroku-postgres amazon-dms

我正在尝试使用AWS DMS将数据从Heroku PostgreSQL数据库复制并复制到AWS RDS PostgreSQL实例,但它无法正常工作(下面有更多信息)。

在DMS日志中,我可以看到以下消息:

2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: RetCode: SQL_ERROR SqlState: 42501 NativeError: 1 Message: ERROR: must be superuser or replication role to use replication slots;, Error while executing the query [1022502] (ar_odbc_stmt.c:2421)

Heroku PostgreSQL的用户不是超级用户或具有复制角色,但我在2天前使用相同的源数据库进行了实验并且它正常工作(我停止了实验以创建更大的RDS实例目标)。我想我这次忘记配置了。

  • 资料来源:Heroku PostgreSQL standard-2(9.5.8)
  • 目标:位于sa-east-1地区的AWS RDS PostgreSQL(9.6.3),其中rds.logical_replication = 1
  • DMS:us-east-1地区的实例

完整的Cloudwatch日志:

2017-10-25T17:43:27 [TASK_MANAGER ]I: Task 'WKJYBNDMNIRS3AEA77T5BP5RN4' running full load and CDC in resume mode after recoverable error, retry #3 (replicationtask.c:1078)
2017-10-25T17:43:27 [TASK_MANAGER ]I: Task Id: bb955197-648a-4177-8653-677b5daf5a6e (replicationtask.c:2682)
2017-10-25T17:43:30 [TASK_MANAGER ]I: Creating threads for all components (replicationtask.c:1664)
2017-10-25T17:43:31 [TASK_MANAGER ]I: Threads for all components were created (replicationtask.c:1832)
2017-10-25T17:43:31 [TASK_MANAGER ]I: Task initialization completed successfully (replicationtask.c:2741)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Going to connect to ODBC connection string: PROTOCOL=7.4-0;DRIVER={PostgreSQL};SERVER=XXXXX.rds.amazonaws.com;DATABASE=XXXXX;PORT=5432;sslmode=verify-ca;pqopt=sslrootcert=/rdsdbdata/rds-metadata/certificates/QQSFSDJTBLFXQLQ6VRVDDAYKCM/dms-ca-cert.pem;UID=USERID; (odbc_endpoint_imp.c:2236)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Target endpoint 'PostgreSQL' is using provider syntax 'PostgreSQLLinux' (provider_syntax_manager.c:589)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Transaction size is 10000. Array size is 1000. (odbc_endpoint_imp.c:837)
2017-10-25T17:43:31 [TARGET_APPLY ]I: CSV files will not be kept (odbc_endpoint_imp.c:1519)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Error CSV files will not be kept (odbc_endpoint_imp.c:1528)
2017-10-25T17:43:31 [TARGET_APPLY ]I: Working in transactional apply mode (endpointshell.c:1436)
2017-10-25T17:43:31 [SOURCE_CAPTURE ]I: Going to connect to ODBC connection string: Driver={PostgreSQL Unicode(x64)};Server=SERVER_AT_HEROKU.compute-1.amazonaws.com;Port=5432;Database=XXXX;UseDeclareFetch=1;Fetch=10000;Uid=USERID;Pwd=PASSWORD;sslmode=require (postgres_endpoint_util.c:252)
2017-10-25T17:43:31 [SOURCE_CAPTURE ]I: Source endpoint 'PostgreSQL Source' is using provider syntax '<default>' (provider_syntax_manager.c:583)
2017-10-25T17:43:34 [TASK_MANAGER ]I: All stream components were initialized (replicationtask.c:2547)
2017-10-25T17:43:34 [SOURCE_CAPTURE ]I: Initial position is set to 'now' (postgres_endpoint_util.c:343)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]I: Replication slot 'wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e' does NOT exist. (postgres_endpoint_capture.c:293)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]I: Initial positioning requested is 'now' (postgres_endpoint_capture.c:313)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Failed to execute statement: 'SELECT xlog_position FROM pg_create_logical_replication_slot('wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e', 'test_decoding')' [1022502] (ar_odbc_stmt.c:2415)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: RetCode: SQL_ERROR SqlState: 42501 NativeError: 1 Message: ERROR: must be superuser or replication role to use replication slots;, Error while executing the query [1022502] (ar_odbc_stmt.c:2421)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Creating replication slot 'wkjybndmnirs3aea_00016385_bb955197_648a_4177_8653_677b5daf5a6e' failed. [1022502] (postgres_endpoint_capture.c:334)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Error executing source loop [1022502] (streamcomponent.c:1581)
2017-10-25T17:43:35 [TASK_MANAGER ]E: Task error notification received from subtask 0, thread 0 [1022502] (replicationtask.c:2375)
2017-10-25T17:43:35 [TASK_MANAGER ]E: Stream component failed at subtask 0, component st_0_QVUF6NET6KSB26SCIIRH4UQZAQ [1022502] (subtask.c:1350)
2017-10-25T17:43:35 [TASK_MANAGER ]W: Task 'WKJYBNDMNIRS3AEA77T5BP5RN4' encountered a recoverable error (repository.c:4749)
2017-10-25T17:43:35 [SOURCE_CAPTURE ]E: Stream component 'st_0_QVUF6NET6KSB26SCIIRH4UQZAQ' terminated [1022502] (subtask.c:1513)
2017-10-25T17:43:40 [TASK_MANAGER ]I: Subtask #0 ended (replicationtask_util.c:937)
2017-10-25T17:43:40 [TASK_MANAGER ]I: Task management thread terminated (replicationtask.c:3210)

2 个答案:

答案 0 :(得分:4)

The prereqs for DMS表示源数据库的用户需要具有超级用户属性; Heroku不允许这样做。

https://devcenter.heroku.com/articles/heroku-postgresql

答案 1 :(得分:0)

也许你可以在安装了heroku cli的docker容器中创建一个lambda,你可以添加:

heroku pg:backups:capture
heroku pg:backups:download

到lambda脚本。然后,您可以将zip上传到s3,然后触发脚本以将数据加载到RDS