Question

我需要使用php（500.000行xml）将大型XML文件解析为mysql。但是使用以下代码，xml文件需要几个小时。如何优化？（我想也许可以让一个数组一次解析为mysql，而不是一次解析一个变量？）

foreach ($data as $dat) {
        $object = $dat;
        $UID = $object['id'];
        $test = mysql_query("SELECT * FROM reports WHERE UID = '$UID'");
        if ($test['UNIQUEID'] ==null) {

            $temp = array("MEDIA" => "{$name}");
            foreach ($object as $obj){
                mysql_query("INSERT INTO reports
                    (MEDIA, UID)
                    VALUES
                    ('$name', '$UID')");
                foreach ($obj as $ats) {
                    $attname = $ats['name'];

                    mysql_query("UPDATE reports 
                                SET $attname = '$ats'
                                WHERE UID = '$UID'
                                ");

                } 

            }
        }
        echo "Done";

    }

编辑： XML：

    <object id="382177">
    <attributes>
      <attribute kind="number" name="REVNO">1</attribute>
      <attribute kind="string" name="UNIQUEID">XXX</attribute>
      <attribute kind="number" name="EVENTVERSION">1</attribute>
      <attribute kind="string" name="EVENTASSOCID">4568190</attribute>
      <attribute kind="number" name="EVENTASSOCRELNO">2</attribute>
      <attribute kind="string" name="EVENTTYPE">PageFlow</attribute>
      <attribute format="%Y-%m-%d %H:%M:%S" kind="time" name="EVENTTIME">2014-02-09 09:40:52</attribute>
      <attribute kind="string" name="EVENTMSG">PageLocked=1</attribute>
      <attribute kind="string" name="EVENTUSER">XXX</attribute>
      <attribute kind="string" name="EVENTAPPL">XXX</attribute>
      <attribute kind="string" name="NAME">XXX</attribute>
      <attribute kind="string" name="NEWSROOM">XXX</attribute>
      <attribute kind="string" name="PRODUCT">XXX</attribute>
      <attribute kind="string" name="PUBDATE">11-02-2014</attribute>
      <attribute kind="string" name="ZONE">XXX</attribute>
      <attribute kind="string" name="EDITION">1</attribute>
      <attribute kind="string" name="PAGENAME">XXX</attribute>
      <attribute kind="number" name="PAGENO">1</attribute>
      <attribute kind="string" name="ARTICLE"></attribute>
     </attributes>
  </object>

编辑2：感谢Mike，这段代码大大提高了性能：

        foreach ($data as $key) {

        $lat = array();
        $lat = $key->attributes;

        $UID = $key['id'];

        mysql_query("INSERT INTO reports
                    (MEDIA, UID, REVNO, UNIQUEID, EVENTVERSION, EVENTASSOCID, EVENTASSOCRELNO, EVENTTYPE, EVENTTIME, EVENTMSG, EVENTUSER, EVENTAPPL, NAME, NEWSROOM, PRODUCT, PUBDATE, ZONE, EDITION, PAGENAME, PAGENO, ARTICLE, LAYOUTDESK, LAYOUTSTATE, RUNNINGPAGENO, SECTIONNAME, SECTIONNO, LASTOPERATOR, LASTREV, LASTDATAOPERATOR, LASTDATAREV, TYPE, SUBTYPE, LAYOUTTEMPLATE, EDITORIALSOURCEUID)
                    VALUES
                    ('$name', '$UID', '{$lat->attribute[0]}', '{$lat->attribute[1]}', '{$lat->attribute[2]}', '{$lat->attribute[3]}', '{$lat->attribute[4]}', '{$lat->attribute[5]}', '{$lat->attribute[6]}', '{$lat->attribute[7]}', '{$lat->attribute[8]}', '{$lat->attribute[9]}', '{$lat->attribute[10]}', '{$lat->attribute[11]}', '{$lat->attribute[12]}', '{$lat->attribute[13]}', '{$lat->attribute[14]}', '{$lat->attribute[15]}', '{$lat->attribute[16]}', '{$lat->attribute[17]}', '{$lat->attribute[18]}', '{$lat->attribute[19]}', '{$lat->attribute[20]}', '{$lat->attribute[21]}', '{$lat->attribute[22]}', '{$lat->attribute[23]}', '{$lat->attribute[24]}', '{$lat->attribute[25]}', '{$lat->attribute[26]}', '{$lat->attribute[27]}', '{$lat->attribute[28]}', '{$lat->attribute[29]}', '{$lat->attribute[30]}', '{$lat->attribute[31]}')");


    }

Answer 1

插入记录然后一次更新一个列会产生可怕的性能。您应该为每条记录解析XML中的所有数据，并将其全部插入一个INSERT。这将为您提供性能方面的量子改进

此外，如果您在插入之前检查每个插入的唯一性，则应使用唯一索引修改模式并使用“INSERT ... IGNORE ...”，检查数字受影响的行检查是否已插入每一行（如果需要检查）。

您可以使用INSERT的批处理形式一次性插入一批记录。

如果仍然不够，请先尝试将XML解析为CSV，然后使用LOAD DATA INFILE批量插入所有数据。

优化mysql查询 - foreach循环

1 个答案: