以下方法用于解析日志文件并在其中查找错误。它工作正常,直到错误数量达到数百万 - 3千万是具体的。然后服务器耗尽内存试图运行应用程序。我得到GC内存错误。我可以将参数更改为java,以便它可以使用更多内存,但我想先优化一下代码。
for (String logFile : fileList) {
Map<String, Long> messageErrorCount = new HashMap<String, Long>();
LOGGER.info("Mining data from: " + logFile);
BufferedReader reader = null;
try {
String hostName = getHostName(logFile);
reader = new BufferedReader(new FileReader(logFile));
long errorCountPerFile = 0;
String logLineStr;
long firstErrorOccurance = 1;
while ((logLineStr = reader.readLine()) != null) {
String errorResult = isErrorLine(logLineStr);
if (errorResult != null) {
errorCountPerFile++;
if (messageErrorCount.containsKey(errorResult)) {
messageErrorCount.put(errorResult, messageErrorCount.get(errorResult) + 1);
} else {
messageErrorCount.put(errorResult, firstErrorOccurance);
}
Pattern pattern = Pattern.compile("^(.*?)\\s-");
Matcher matcher = pattern.matcher(errorResult);
if (matcher.find()) {
String formattedError = matcher.group(1);
if (totalErrorMessages.containsKey(formattedError)) {
totalErrorMessages.put(formattedError, totalErrorMessages.get(formattedError) + 1);
} else {
totalErrorMessages.put(formattedError, firstErrorOccurance);
}
}
}
}
// Need to add the file to list only if count is non zero.
if (errorCountPerFile > 0) {
String fileName = new File(logFile).getName();
fileErrorMaps.put(fileName, messageErrorCount);
putErrorLogRecord(hostName, fileName, errorCountPerFile, messageErrorCount);
}
} catch (IOException e) {
LOGGER.error("File operation failed: " + logFile);
throw new ParserException("File operation failed: " + logFile, e);
} finally {
try {
if (reader != null) {
reader.close();
}
} catch (IOException ex) {
LOGGER.error("File operation failed: " + logFile);
throw new ParserException("File operation failed: " + logFile, ex);
}
}
}
和isErrorLine
方法:
public String isErrorLine(String logString) {
if (isWhitelistedError(logString)) {
return null;
} else {
Matcher matcher = ULOG_LINE_PATTERN_QUOTES.matcher(logString);
if (matcher.find()) {
return (matcher.group(2) + " - " + matcher.group(3)).replace("\"", "");
}
matcher = ULOG_LINE_PATTERN_ENDLINE.matcher(logString);
if (matcher.find()) {
return (matcher.group(2) + " - " + matcher.group(3)).replace("\"", "");
}
}
return null;
}
这些行在每次使用时都会创建新对象:
Pattern pattern = Pattern.compile("^(.*?)\\s-");
Matcher matcher = pattern.matcher(errorResult);
有没有更好的方法来执行此方法,因此它使用更少的内存?