Question

我正以更好（更快）的方式寻找建议。我的问题是，当你增加＆＃34;主机的长度时，＃34;列出程序需要指数级更长才能完成，如果＆＃34;主机＆＃34;这个程序需要很长时间才能完成它似乎只是锁定。

＆＃34;主机＆＃34;是包含数万个项目的列表列表。当遍历＆＃34;主机＆＃34; i [0]将始终为IP地址，i [4]将始终为5位数字，i [7]将始终为多行字符串。
＆＃34; searchPatterns＆＃34;是从CSV文件读入的列表列表，其中元素i [0]到i [3]是正则表达式搜索模式（或字符串＆＃34; SKIP＆＃34;），i [6]是用于的唯一字符串识别模式匹配。

我目前的做法是使用CSV文件中的正则表达式模式来搜索＆＃34; hosts＆＃34;中包含的每个多行列表项。我[7]元素。有100个可能的匹配项，我需要识别与每个IP地址关联的所有匹配项，并从CSV文件中分配唯一字符串以标识所有模式匹配项。最后，我需要将这些信息放入＆＃34; fullMatchList＆＃34;以后再使用。

注意：即使每个列表项都在＆＃34; searchPatterns＆＃34;最多有4个模式，我只需要它来识别找到的第一个模式，然后它可以继续查找下一个列表项以继续查找该IP的匹配。

for i in hosts:
    if i[4] == "13579" or i[4] == "24680":
        for j in searchPatterns:
            for k in range(4):
                if j[k] == "SKIP":
                    continue
                else:
                    match = re.search(r'%s' % j[k], i[7], flags=re.DOTALL)
                    if match is not None:
                        if tempIP == "":
                            tempIP = i[0]
                            matchListPerIP.append(j[4])
                        elif tempIP == i[0]:
                            matchListPerIP.append(j[4])
                        elif tempIP != i[0]:
                            fullMatchList.append([tempIP, matchListPerIP])
                            tempIP = i[0]
                            matchListPerIP = []
                            matchListPerIP.append(j[4])
                        break
fullMatchList.append([tempIP, matchListPerIP])

以下是CSV文件中的正则表达式搜索模式示例：
(?!(.*?)\br2\b)cpe:/o:microsoft:windows_server_2008:

该模式旨在识别Windows Server 2008，并包含一个负向前瞻以避免匹配R2版本。

我是Python的新手，所以任何建议都值得赞赏！谢谢！

Answer 1

NIDS社区在针对一长串的正则表达式（防火墙规则）测试相同的字符串（网络数据包）方面做了很多工作。

我还没有读过任何文献，但是Coit等人的“寻求更快的字符串匹配以进行入侵检测或超过Snort的速度”似乎是一个不错的起点。

引言：

2019-03-12 14:21:14,272 main INFO sun.reflect.Reflection.getCallerClass is not supported. ReflectionUtil.getCallerClass will be much slower due to this. java.lang.ClassNotFoundException: sun.reflect.Reflection
at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
at org.apache.logging.log4j.util.LoaderUtil.loadClass(LoaderUtil.java:141)
at org.apache.logging.log4j.util.ReflectionUtil.<clinit>(ReflectionUtil.java:65)
at org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:136)
at org.apache.logging.log4j.core.impl.ThrowableProxy.<init>(ThrowableProxy.java:121)
at org.apache.logging.log4j.core.impl.MutableLogEvent.getThrownProxy(MutableLogEvent.java:329)
at org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(ExtendedThrowablePatternConverter.java:61)
at org.apache.logging.log4j.core.pattern.PatternFormatter.format(PatternFormatter.java:38)
at org.apache.logging.log4j.core.layout.PatternLayout$PatternSerializer.toSerializable(PatternLayout.java:294)
at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:195)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:180)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:57)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:176)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:169)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:160)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:156)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:129)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:120)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:447)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:432)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:416)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:402)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:146)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2091)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:1988)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1960)
at org.apache.logging.log4j.spi.AbstractLogger.error(AbstractLogger.java:733)
at ch.removonarx.chargingstations.data.importer.csv.CSVChargingStationImporter.parse(CSVChargingStationImporter.java:33)
at ch.removonarx.chargingstations.data.importer.csv.TestCSVChargingStationImporter.testChargingStationImport(TestCSVChargingStationImporter.java:27)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.maven.surefire.junit.PojoTestSet.executeTestMethod(PojoTestSet.java:151)
at org.apache.maven.surefire.junit.PojoTestSet.executeTestMethods(PojoTestSet.java:102)
at org.apache.maven.surefire.junit.PojoTestSet.execute(PojoTestSet.java:83)
at org.apache.maven.surefire.junit.JUnit3Provider.executeTestSet(JUnit3Provider.java:121)
at org.apache.maven.surefire.junit.JUnit3Provider.invoke(JUnit3Provider.java:98)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:189)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:165)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:85)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:115)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:75)

如何使用数百种模式在列表列表中搜索数万个项目

1 个答案: