我收到内存耗尽错误,我不应该占用任何内存!
该应用程序位于Windows 8 Server / IIS i / PHP 5.5 / CodeIgniter / MS SQL Server
上错误如下:
[2014年5月23日10:56:57 America / New_York] PHP致命错误:允许 内存大小为134217728字节耗尽(试图分配1992年 C:\ inetpub \ wwwroot \ application \ models \ DW_import.php中的字节) 在第112行
[2014年5月23日11:07:34 America / New_York] PHP致命错误: 允许的内存大小为134217728字节耗尽(尝试分配 2438字节)in 第113行的C:\ inetpub \ wwwroot \ application \ models \ DW_import.php
该脚本在目录中查找几个不同的CSV文件以导入数据库。请记住,导入文件很大,有些高达4 Gigs的数据。据我所知,没有变量不断聚合可能导致此问题的数据。正在运行的脚本(模型)(此控制器没有视图,只有模型)如下:
DW_import.php
<?php
class dw_import extends CI_Model {
public function import(){
global $file,$errLogFile,$logFile,$tableName, $fieldList, $file, $count, $line, $query;
$this->load->database(); // init db connection
// map file types to database tables
$fileToDBArr = array(
'Customers' => 'customer',
'Customers_Historical' => 'customer_historical',
'Orders' => 'order',
'Customer_AR_Aggs' => 'customer_ar_aging_agg'
);
// extend timeout of this script
ini_set('max_execution_time', 3600);
// error handler to log errors and continue processing
function myErrorHandler($errno,$errstr,$errfile,$errline){
global $file,$errLogFile,$logFile,$tableName, $fieldList, $file, $count, $line, $query;
// error - store in DB
//echo "<br>[$errno $errstr $errfile $errline $tableName $file $count] $errLogFile<br>";
$err = "#$errno $errstr $errfile on line $errline :: Table $tableName File $file Row# $count Headers: $fieldList Data: $line";
echo $err;
file_put_contents($errLogFile,$err,FILE_APPEND);
};
set_error_handler("myErrorHandler");
// set temp error log file
$errLogFile = "C:/Data_Updates/logs/general." . date('YmdHis') . ".errLog";
// loop thru file types
foreach($fileToDBArr as $fileType=>$table){
// get the files for this import type
$fileArr = glob('C:/Data_Updates/'.$fileType.'.*');
sort($fileArr,SORT_STRING); // sort so earlier files (by date in file name) will process first
// loop thru files found
foreach($fileArr as $file){
// set log file paths specific to this import file
$errLogFile = str_replace('Data_Updates/','Data_Updates/logs/',$file) . "." . date('YmdHis') . ".errLog";
$logFile = str_replace('Data_Updates/','Data_Updates/logs/',$file) . "." . date('YmdHis') . ".log";
file_put_contents($logFile,"---BEGIN---",FILE_APPEND); // log
// lets get the file type and translate it into a table name
preg_match('/C:\/Data_Updates\/([^\.]+)/',$file,$matches);
$fileType = $matches[1];
$tableName = $fileToDBArr[$fileType];
// lets get the first row as a field list
$fp = fopen($file,'r');
//$fieldList = str_replace('"','',fgets($fp));
// counters to track status
$count = 0;
$startPoint = 0;
// see if continuation, set startPoint to last row imported from file
$query = "SELECT max(import_line) as maxline FROM $tableName WHERE import_file = '" . addslashes($file) . "'";
$result = $this->db->query($query);
foreach($result->result() as $row) $startPoint = $row->maxline+1; // set the startPoint if this is continuation
file_put_contents($logFile,"\nstartPoint $startPoint",FILE_APPEND); // log
// loop thru file lines
while (!feof($fp)) {
$line = fgets($fp);
// reformat those pesky dates from m/d/y to y-m-d
$line = preg_replace('/, ?(\d{1,2})\/(\d{1,2})\/(\d{4})/',',${3}-${1}-${2}',$line);
if(!$count){
// header row - set aside to use for column headers on insert statements
$fieldList = str_replace('"','',$line);
file_put_contents($logFile,"\nHeaders: $fieldList",FILE_APPEND); // log
} elseif($count >= $startPoint && trim($line)) {
// data row - insert into DB
$lineArr = str_getcsv($line); // turn this CSV line into an array
// build the insert query
$query = "INSERT INTO $tableName ($fieldList,import_date,import_file,import_line)
VALUES (";
foreach($lineArr as $k=>$v) $query .= ($v !== '') ? "'".addslashes(utf8_encode($v))."'," : " NULL,";
$query .= "now(),'" . addslashes($file). "',$count)
ON DUPLICATE KEY UPDATE ";
foreach(explode(',',$fieldList) as $k=>$v) $query .= "\n$v=" . (($lineArr[$k] !== '') ? "\"" . addslashes(utf8_encode($lineArr[$k])) . "\"" : "NULL") . ", ";
$query .= "import_date = now(),import_file='" . addslashes($file) . "',import_line = $count ";
if(!$this->db->query($query)) {
trigger_error('db error ' . $this->db->_error_number() . ' ' . $this->db->_error_message());
$status = 'error ';
} else {
$status = 'success ';
};
file_put_contents($logFile,"row: $count status: $status data: $line",FILE_APPEND); // log'
} else {
// skipped - this row was already imported from this file
// removed log to speed up
file_put_contents($logFile,"row: $count status: SKIPPED data: $line",FILE_APPEND); // log
}; // if $count
$count++;
}; // while $fp
fclose($fp);
// file complete - move file to archive
rename($file,str_replace('Data_Updates/','Data_Updates/archive/',$file));
file_put_contents($logFile,"-- END --",FILE_APPEND); // log
}; // each $fileArr
}; // each $globArr
} // end import function
} // end class
?>
任何帮助将不胜感激!
******** 编辑
根据几个人的建议,我添加了一些更改。这些更改仅影响循环逻辑的“数据行插入到DB”部分。您可以看到添加日志以跟踪memory_get_peak_usage,添加unset()和clearcachestat()。代码下面是一些日志数据:
file_put_contents($logFile,memory_get_peak_usage() . " line 1 \n\r",FILE_APPEND);
// data row - insert into DB
if(isset($lineArr)) unset($lineArr);
file_put_contents($logFile,memory_get_peak_usage() . " line 1.1 \n\r",FILE_APPEND);
$lineArr = str_getcsv($line); // turn this CSV line into an array
// build the insert query
file_put_contents($logFile,memory_get_peak_usage() . " line 2 lineArr size: " . strlen(implode(',',$lineArr)) . "\n\r",FILE_APPEND);
if(isset($query)) unset($query);
file_put_contents($logFile,memory_get_peak_usage() . " line 2.1 lineArr size: " . strlen(implode(',',$lineArr)) . "\n\r",FILE_APPEND);
$query = "INSERT INTO $tableName ($fieldList,import_date,import_file,import_line)
VALUES (";
file_put_contents($logFile,memory_get_peak_usage() . " line 2.2 lineArr size: " . strlen(implode(',',$lineArr)) . "\n\r",FILE_APPEND);
foreach($lineArr as $k=>$v) $query .= ($v !== '') ? "'".addslashes(utf8_encode($v))."'," : " NULL,";
$query .= "now(),'" . addslashes($file). "',$count)
ON DUPLICATE KEY UPDATE ";
file_put_contents($logFile,memory_get_peak_usage() . " line 2.3 lineArr size: " . strlen(implode(',',$lineArr)) . "\n\r",FILE_APPEND);
foreach(explode(',',$fieldList) as $k=>$v) $query .= "\n$v=" . (($lineArr[$k] !== '') ? "\"" . addslashes(utf8_encode($lineArr[$k])) . "\"" : "NULL") . ", ";
file_put_contents($logFile,memory_get_peak_usage() . " line 2.4 lineArr size: " . strlen(implode(',',$lineArr)) . "\n\r",FILE_APPEND);
$query .= "import_date = now(),import_file='" . addslashes($file) . "',import_line = $count ";
file_put_contents($logFile,memory_get_peak_usage() . " line 3 query size: " . strlen($query) . "\n\r",FILE_APPEND);
if(!$this->db->query($query)) {
trigger_error('db error ' . $this->db->_error_number() . ' ' . $this->db->_error_message());
$status = 'error ';
} else {
$status = 'success ';
};
clearstatcache();
日志数据:(最左边的数字是memory_get_peak_usage()调用的结果
2724960 line 1.1
2724960 line 2 lineArr size: 194
2724960 line 2.1 lineArr size: 194
2724960 line 2.2 lineArr size: 194
2724960 line 2.3 lineArr size: 194
2727392 line 2.4 lineArr size: 194
2727392 line 3 query size: 2346
2727392 line 1
2727392 line 1.1
2727392 line 2 lineArr size: 194
2727392 line 2.1 lineArr size: 194
2727392 line 2.2 lineArr size: 194
2727392 line 2.3 lineArr size: 194
2729944 line 2.4 lineArr size: 194
2729944 line 3 query size: 2346
2729944 line 1
2729944 line 1.1
2729944 line 2 lineArr size: 194
2729944 line 2.1 lineArr size: 194
2729944 line 2.2 lineArr size: 194
2729944 line 2.3 lineArr size: 194
2732448 line 2.4 lineArr size: 194
2732448 line 3 query size: 2346
2732448 line 1.1
2732448 line 2 lineArr size: 194
2732448 line 2.1 lineArr size: 194
2732448 line 2.2 lineArr size: 194
2732448 line 2.3 lineArr size: 194
2735088 line 2.4 lineArr size: 194
2735088 line 3 query size: 2346
请注意,内存在第2.3行和第2.4行之间仍在增长,这是以下代码行:
foreach(explode(',',$fieldList) as $k=>$v) $query .= "\n$v=" . (($lineArr[$k] !== '') ? "\"" . addslashes(utf8_encode($lineArr[$k])) . "\"" : "NULL") . ", ";
有什么想法吗?
答案 0 :(得分:6)
找到答案:
$this->load->database(); // init db connection, already in code
$this->db->save_queries = false; // ADD THIS LINE TO SOLVE ISSUE
这是CodeIgniter中一个可爱的未记录的设置。 CI显然默认保存查询,即使相对于插入/更新查询保存了一定数量的数据。由于在此导入过程中运行了大量插入,因此内存泄漏变得非常重要。将CI设置为不保存查询可以解决问题。
让我失望的是memory_get_peak_usage()
在运行插入查询之前报告内存使用量增加,而不是在它期间(PHP错误?)。
作为最终的现实检查,我删除了所有其他优化建议(unset
,clearstatcache
等),并确认它们对内存问题没有任何积极影响。
答案 1 :(得分:0)
尝试使用set_time_limit(0)
来定义进程的时间限制..在循环之间,您可以使用clearstatcache();