Question

我有一个csv文件，其中9个缺少数据（csv文件的大小为229mb）。我将csv文件拆分为较小的，即一个csv包含4000或5000个数据。我上传csv文件并从中读取数据并插入把它插入到数据库中。我将数据插入到虚拟表中并列出它。然后提交列表并插入到主表中。但它需要1小时以上。我的代码是，

$path = "csv1.csv";
if (($handle = fopen("$path", "r")) !== FALSE) {
$cntcnt=1;
   while (($info = fgetcsv($handle, 10000, ",")) !== FALSE) {
      foreach($info as $key=>$value) {
      $datas[$key]=$value;
                             if(isset($datas[3]))       $bname      = $datas[3];        else $bname     = "";
                            if(isset($datas[4]))        $address    = $datas[4];        else $address       = "";
                            if(isset($datas[5]))        $city       = $datas[5];        else $city          = "";
                            if(isset($datas[7]))        $zipcode    = $datas[7];        else $zipcode       = "";
                            if(isset($datas[8]))        $cname      = $datas[8];        else $cname         = "";
                            if(isset($datas[13]))       $phone      = $datas[13];       else $phone             = "";
                            if(isset($datas[14]))       $fax        = $datas[14];       else $fax           = "";
                            if(isset($datas[15]))       $email      = $datas[15];       else $email         = "";
                            if(isset($datas[16]))       $website    = $datas[16];       else $website           = "";
                            $bname= mysql_real_escape_string($bname); 
                            $cname= mysql_real_escape_string($cname);
                            $address =mysql_real_escape_string($address);
                            $city =mysql_real_escape_string($city);
$data=array('vendorid'=>$vendorid,'cid'=>$cid,'scid'=>$scid,'sscid'=>$sscid,'rid'=>$rid,'pid'=>$pid,'rcid'=>$cityid,'lid'=>$lid,'bname'=>$bname,'address'=>$address,'street'=>$street,'zipcode'=>$zipcode,'phone'=>$phone,'cell'=>$cell,'fax'=>$fax,'email'=>$email,'website'=>$website,'description'=>$description,'mapaddress'=>$mapaddress,'price'=>$price,'date'=>$date,'ref_code'=>$ref_code,'realestate_category'=>$realestate_category,'template'=>$template,'busname'=>$busname,'code'=>$code,'b_type'=>$b_type,'edit_date'=>$edit_date,'phone2'=>$phone2,'cname'=>$cname,'ssscid'=>$ssscid);
$this->db->insert('c_tbldummy',$data);
        }
        }

有没有办法加快这个过程？因为我的客户想要上传50000到75000个数据。任何一个请帮助我。

Answer 1

使用LOAD DATA LOCAL INFILE上传速度更快，并为表格添加索引

这里有LOAD DATA INFILE的样本

代码：

$query = "LOAD DATA LOCAL INFILE '$path_of_csv_file' INTO TABLE $table_name FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' IGNORE 1 LINES ($column_of_table);

您可以为不在csv

中的列添加“set”

代码：

$query = "LOAD DATA LOCAL INFILE '$path_of_csv_file' INTO TABLE $table_name FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' IGNORE 1 LINES ($column_of_table) set column1 = '$value1', column2= '$value2' ";

Answer 2

使用batch_insert可以减少你的时间......

$this->db->insert_batch();

在这里，你将每一行的数据插入一个表...所以，因为你有75000 +数据..这意味着75000+插入查询..我想这是花费时间....这可能不会明显加快但我相信这会减少插入时间..

批量插入会将查询减少为一个...并在一个查询中插入所有数据..

你可以尝试一下......

 while (($info = fgetcsv($handle, 10000, ",")) !== FALSE) {
  foreach($info as $key=>$value) {
  $datas[$key]=$value;
                         if(isset($datas[3]))       $bname      = $datas[3];        else $bname     = "";
                        if(isset($datas[4]))        $address    = $datas[4];        else $address       = "";
                        if(isset($datas[5]))        $city       = $datas[5];        else $city          = "";
                        if(isset($datas[7]))        $zipcode    = $datas[7];        else $zipcode       = "";
                        if(isset($datas[8]))        $cname      = $datas[8];        else $cname         = "";
                        if(isset($datas[13]))       $phone      = $datas[13];       else $phone             = "";
                        if(isset($datas[14]))       $fax        = $datas[14];       else $fax           = "";
                        if(isset($datas[15]))       $email      = $datas[15];       else $email         = "";
                        if(isset($datas[16]))       $website    = $datas[16];       else $website           = "";
                        $bname= mysql_real_escape_string($bname); 
                        $cname= mysql_real_escape_string($cname);
                        $address =mysql_real_escape_string($address);
                        $city =mysql_real_escape_string($city);
               $data[]=array('vendorid'=>$vendorid,'cid'=>$cid,'scid'=>$scid,'sscid'=>$sscid,'rid'=>$rid,'pid'=>$pid,'rcid'=>$cityid,'lid'=>$lid,'bname'=>$bname,'address'=>$address,'street'=>$street,'zipcode'=>$zipcode,'phone'=>$phone,'cell'=>$cell,'fax'=>$fax,'email'=>$email,'website'=>$website,'description'=>$description,'mapaddress'=>$mapaddress,'price'=>$price,'date'=>$date,'ref_code'=>$ref_code,'realestate_category'=>$realestate_category,'template'=>$template,'busname'=>$busname,'code'=>$code,'b_type'=>$b_type,'edit_date'=>$edit_date,'phone2'=>$phone2,'cname'=>$cname,'ssscid'=>$ssscid);

    }
    }

$this->db->insert_batch('c_tbldummy',$data);

对于insert_batch（）

，

CI doc

上传csv需要帮助

2 个答案: