php改善大型数据循环的性能

时间:2018-07-12 09:53:07

标签: php mysql excel

我在Excel(.xlsx)中有大约60000行的大量数据,我正在使用 Spout 库在MySQL中插入所有数据以读取Excel数据,并且从excel文件中读取数据后还有一些详细信息不是Unicode格式,也不是英语(当前数据使用印地语),因此我首先使用 Python 脚本将此非标准文本转换为Unicode,然后将Unicode数据印地语转换为使用 Google Translate API 使用英语。.一切正常,但是要花很多时间。在所有这些步骤之后,在我当前的脚本中将200行插入MySQL,这大约需要20分钟。.我的问题是如何改善当前脚本以提高性能 这是我当前的脚本:

// Convert local hindi font to unicode
function KridevToUnicode($k) {
    $myfile =  fopen("input.txt", "w");
    fwrite($myfile, $k);
    fclose($myfile);
    shell_exec('krutidev2unicode.py -i input.txt -o output.txt');
    $file = fopen("output.txt","r");
    return hindiToEnglish(fgets($file));
    fclose($file);
}
// Convert hindi to english
function hindiToEnglish($text) {
    $apiKey2 = 'XXXXXXXXXXXXXXXXXXXXX';
    $url = 'https://www.googleapis.com/language/translate/v2?key=' . $apiKey2 . '&q=' . rawurlencode($text) . '&source=hi&target=en';
    $handle = curl_init($url);
    curl_setopt($handle, CURLOPT_RETURNTRANSFER, true);
    $response = curl_exec($handle);                 
    $responseDecoded = json_decode($response, true);
    curl_close($handle);
    return $responseDecoded['data']['translations'][0]['translatedText'];
}
//Read data from Excel and insert in MySQL
require_once 'Spout/Autoloader/autoload.php';
use Box\Spout\Reader\ReaderFactory;
use Box\Spout\Common\Type;
$reader = ReaderFactory::create(Type::XLSX); 
$reader->open('a.xlsx');
foreach($reader->getSheetIterator() as $sheet) {
    foreach($sheet->getRowIterator() as $row) {
        ob_start();
        $i++;
        $membershipNumber = $row[1];
        $memberName = KridevToUnicode($row[3]);
        $fatherHusbandName = KridevToUnicode($row[4]);
        if(is_string($row[5])) {
            $dob = '';
        }else {
            $dob = $row[5]->format('d-m-Y');
            $dob = date('Y-m-d',strtotime($dob));
        }
        $permanentAddress = KridevToUnicode($row[6]);
        $currentAddress = KridevToUnicode($row[7]);
        $district = KridevToUnicode($row[8]);
        $state = KridevToUnicode($row[10]);
        $pincode = $row[11];
        $mobile = $row[13];
        $email = $row[14];
        $Shasan = KridevToUnicode($row[15]);
        $Occupation = KridevToUnicode($row[16]);
        $education = KridevToUnicode($row[17]);
        $Inspiration = KridevToUnicode($row[18]);
        $ReceiptNo = $row[19];
        if(is_string($row[20])) {
            $DateofReceipt = '';
        }else {
            $DateofReceipt = $row[20]->format('d-m-Y');
            $DateofReceipt = date('Y-m-d',strtotime($DateofReceipt));
        }
        $sql = "INSERT INTO users (membershipNumber, name, father_husband_name, dob, permanent_address, current_address, district, state, pin_code, phone, email, shasan, profession, education, inspiration, receiptNo, DateofReceipt)
        VALUES ('$membershipNumber', '$memberName', '$fatherHusbandName', '$dob', '$permanentAddress', '$currentAddress', '$district', '$state', '$pincode', '$mobile', '$email', '$Shasan', '$Occupation', '$education', '$Inspiration', '$ReceiptNo', '$DateofReceipt')";
        if ($conn->query($sql) === TRUE) {
        } else {
            echo "Error: " . $sql . "<br>" . $conn->error;
        }
        echo ob_get_contents();
        ob_end_flush();
    }
}
$reader->close();
$conn->close();

1 个答案:

答案 0 :(得分:0)

准备一个批量插入查询,并在完成创建查询后执行它。 您可以在插入时插入多个值,进行类似的查询并在完成后执行

Here就是