Question

我已成功重构以下脚本以从服务器1上的表中选择记录，然后连接到服务器2并将缺少的记录插入/忽略到那里的克隆表中。

这可行，但需要大约1.5分钟才能运行。我希望有人能以更快，更有效的方式提供帮助，因为它成功但价格昂贵。

我没有选择进行联合存储或复制，因此必须使用脚本完成此操作。我以前通过使用源表的最大ID来做到这一点，但在插入后我每天最多丢失15条记录。

这是脚本：

$source_data = mysqli_query($conn, 
    "select * from `cdrdb`.`session` where ts >= now() - INTERVAL 1 DAY");

while($source = $source_data->fetch_assoc()) {
    //Insert new rows into ambition.session
    $stmt = $conn2->prepare(
        "INSERT IGNORE INTO ambition.session (SESSIONID, 
         SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,
         DIALPLANNAME,TERMINATIONREASONCODE,ISCLEARINGLEGORIGINATING,
         CREATIONTIMESTAMP,ALERTINGTIMESTAMP,CONNECTTIMESTAMP,DISCONNECTTIMESTAMP,
         HOLDTIMESECS,LEGTYPE1,LEGTYPE2,INTERNALPARTYTYPE1,INTERNALPARTYTYPE2
         ,SERVICETYPEID1,SERVICETYPEID2,EXTENSIONID1,EXTENSIONID2,
         LOCATION1,LOCATION2,TRUNKGROUPNAME1,TRUNKGROUPNAME2,SESSIONIDTRANSFEREDFROM
         ,SESSIONIDTRANSFEREDTO,ISTRANSFERINITIATEDBYLEG1,
         SERVICEEXTENSION1,SERVICEEXTENSION2,SERVICENAME1,
         SERVICENAME2,MISSEDUSERID2,ISEMERGENCYCALL,NOTABLECALLID,
         RESPONSIBLEUSEREXTENSIONID,
         ORIGINALLYCALLEDPARTYNO,ACCOUNTCODE,ACCOUNTCLIENT,ORIGINATINGLEGID
         ,SYSTEMRESTARTNO,PATTERN,HOLDCOUNT,AUXSESSIONTYPE,
         DEVICEID1,DEVICEID2,ISLEG1ORIGINATING,ISLEG2ORIGINATING,
         GLOBALCALLID,CADTEMPLATEID,CADTEMPLATEID2,ts,INITIATOR,
         ACCOUNTNAME,APPNAME,CALLID,CHRTYPE,CALLERNAME,serviceid1,serviceid2)

        VALUES ( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")
    or die(mysqli_error($conn2)) ;

    mysqli_stmt_bind_param($stmt,
        "iisssiissssiiiiiiiiissssiiissssiiiisssiisiiiiiiiiisisssisii"
         ,$source['SESSIONID']
         ,$source['SESSIONTYPE']
         ,$source['CALLINGPARTYNO']
         //omitting other columns for sake of space
       );
          $stmt->execute() or die(mysqli_error($conn2));
                                    }

Answer 1

一个简单的改进是将调用移至prepare()到循环之前。由于每次循环准备好的语句都是相同的，因此每次都不需要联系DB服务器。

您也可以将调用移到循环外的bind_param()，因为每次变量都相同。 bind_param绑定到引用，因此更新变量将更改调用execute()时插入的内容。

但是，这些可能只会产生很小的差异。提高INSERT查询速度的最有效方法之一是一次插入多行。使用PDO比使用mysqli更容易，因为您可以在调用$stmt->execute()时提供一系列参数。代码看起来像：

$params = array();
$count = 0;
$batch_size = 100;
$placeholders = implode(", ", array_fill(0, $batch_size, "(SESSIONID, 
     SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,
     DIALPLANNAME,TERMINATIONREASONCODE,ISCLEARINGLEGORIGINATING,
     CREATIONTIMESTAMP,ALERTINGTIMESTAMP,CONNECTTIMESTAMP,DISCONNECTTIMESTAMP,
     HOLDTIMESECS,LEGTYPE1,LEGTYPE2,INTERNALPARTYTYPE1,INTERNALPARTYTYPE2
     ,SERVICETYPEID1,SERVICETYPEID2,EXTENSIONID1,EXTENSIONID2,
     LOCATION1,LOCATION2,TRUNKGROUPNAME1,TRUNKGROUPNAME2,SESSIONIDTRANSFEREDFROM
     ,SESSIONIDTRANSFEREDTO,ISTRANSFERINITIATEDBYLEG1,
     SERVICEEXTENSION1,SERVICEEXTENSION2,SERVICENAME1,
     SERVICENAME2,MISSEDUSERID2,ISEMERGENCYCALL,NOTABLECALLID,
     RESPONSIBLEUSEREXTENSIONID,
     ORIGINALLYCALLEDPARTYNO,ACCOUNTCODE,ACCOUNTCLIENT,ORIGINATINGLEGID
     ,SYSTEMRESTARTNO,PATTERN,HOLDCOUNT,AUXSESSIONTYPE,
     DEVICEID1,DEVICEID2,ISLEG1ORIGINATING,ISLEG2ORIGINATING,
     GLOBALCALLID,CADTEMPLATEID,CADTEMPLATEID2,ts,INITIATOR,
     ACCOUNTNAME,APPNAME,CALLID,CHRTYPE,CALLERNAME,serviceid1,serviceid2)"));
$stmt = $pdo->prepare("INSERT INTO ambition.sentence (<columns>) VALUES $placeholders");
while ($row = $source_data->fetch(PDO::FETCH_NUM)) {
    $params += $row; // Append this row to $params
    $count++;
    if ($count != $batch_size) {
        continue;
    }
    $stmt->execute($params);
    // Reset variables for next batch
    $params = array();
    $count = 0;
}
if ($count) { // Handle the last batch that isn't the full size
    $placeholders = implode(", ", array_fill(0, $count, "( ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"));
    $stmt = $pdo->prepare("INSERT INTO ambition.sentence (SESSIONID, 
         SESSIONTYPE,CALLINGPARTYNO,FINALLYCALLEDPARTYNO,
         DIALPLANNAME,TERMINATIONREASONCODE,ISCLEARINGLEGORIGINATING,
         CREATIONTIMESTAMP,ALERTINGTIMESTAMP,CONNECTTIMESTAMP,DISCONNECTTIMESTAMP,
         HOLDTIMESECS,LEGTYPE1,LEGTYPE2,INTERNALPARTYTYPE1,INTERNALPARTYTYPE2
         ,SERVICETYPEID1,SERVICETYPEID2,EXTENSIONID1,EXTENSIONID2,
         LOCATION1,LOCATION2,TRUNKGROUPNAME1,TRUNKGROUPNAME2,SESSIONIDTRANSFEREDFROM
         ,SESSIONIDTRANSFEREDTO,ISTRANSFERINITIATEDBYLEG1,
         SERVICEEXTENSION1,SERVICEEXTENSION2,SERVICENAME1,
         SERVICENAME2,MISSEDUSERID2,ISEMERGENCYCALL,NOTABLECALLID,
         RESPONSIBLEUSEREXTENSIONID,
         ORIGINALLYCALLEDPARTYNO,ACCOUNTCODE,ACCOUNTCLIENT,ORIGINATINGLEGID
         ,SYSTEMRESTARTNO,PATTERN,HOLDCOUNT,AUXSESSIONTYPE,
         DEVICEID1,DEVICEID2,ISLEG1ORIGINATING,ISLEG2ORIGINATING,
         GLOBALCALLID,CADTEMPLATEID,CADTEMPLATEID2,ts,INITIATOR,
         ACCOUNTNAME,APPNAME,CALLID,CHRTYPE,CALLERNAME,serviceid1,serviceid2) VALUES $placeholders");
    $stmt->execute($params);
}

为了使其正常工作，您需要确保SELECT查询返回的列与您要插入的列表的顺序相同。在执行此操作时请避免使用SELECT *，因此如果源表的架构发生更改，则不会有任何意外。

Answer 2

您现在可能知道，一次插入一行，每行一个事务，是加载数据的最慢方式。

我测试了不同的数据加载方式，我在Percona Live 2017上发表了演讲：

Load Data Fast!

TL; DR：使用LOAD DATA INFILE，即使您必须先将源数据转储到CSV文件中。

这是一个例子，虽然我还没有测试过：

$all_columns = "
  `SESSIONID`, `SESSIONTYPE`, `CALLINGPARTYNO`, `FINALLYCALLEDPARTYNO`,
  `DIALPLANNAME`, `TERMINATIONREASONCODE`, `ISCLEARINGLEGORIGINATING`,
  `CREATIONTIMESTAMP`, `ALERTINGTIMESTAMP`, `CONNECTTIMESTAMP`,
  `DISCONNECTTIMESTAMP`, `HOLDTIMESECS`, `LEGTYPE1`, `LEGTYPE2`,
  `INTERNALPARTYTYPE1`, `INTERNALPARTYTYPE2`, `SERVICETYPEID1`, 
  `SERVICETYPEID2`, `EXTENSIONID1`, `EXTENSIONID2`, `LOCATION1`, `LOCATION2`,
  `TRUNKGROUPNAME1`, `TRUNKGROUPNAME2`, `SESSIONIDTRANSFEREDFROM`,
  `SESSIONIDTRANSFEREDTO`, `ISTRANSFERINITIATEDBYLEG1`, `SERVICEEXTENSION1`,
  `SERVICEEXTENSION2`, `SERVICENAME1`, `SERVICENAME2`, `MISSEDUSERID2`,
  `ISEMERGENCYCALL`, `NOTABLECALLID`, `RESPONSIBLEUSEREXTENSIONID`,
  `ORIGINALLYCALLEDPARTYNO`, `ACCOUNTCODE`, `ACCOUNTCLIENT`, `ORIGINATINGLEGID`,
  `SYSTEMRESTARTNO`, `PATTERN`, `HOLDCOUNT`, `AUXSESSIONTYPE`, `DEVICEID1`,
  `DEVICEID2`, `ISLEG1ORIGINATING`, `ISLEG2ORIGINATING`, `GLOBALCALLID`,
  `CADTEMPLATEID`, `CADTEMPLATEID2`, `ts`, `INITIATOR`, `ACCOUNTNAME`, `APPNAME`,
  `CALLID`, `CHRTYPE`, `CALLERNAME`, `serviceid1`, `serviceid2`"

$select_sql = "
  SELECT $all_columns FROM `cdrdb`.`session` 
  WHERE ts >= NOW() - INTERVAL 1 DAY";

$source_data = mysqli_query($conn, $select_sql);

$tmpfilename = tempnam('/tmp', 'data');
fp = fopen($tmpfilename, 'w'); // do proper error handling and use 
while ($source = $source_data->fetch_assoc()) {
    fputcsv($fp, array_values($source));
}
fclose($fp);

$tmpfilename = mysqli_real_escape_string($conn2, $tmpfilename);

$load_sql = "
  LOAD DATA LOCAL INFILE '$tmpfilename' 
  IGNORE INTO TABLE `ambition`.`session`
  FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
  ( $all_columns )";

$conn2->query($load_sql);

加速数据库服务器连接之间的插入/忽略

2 个答案: