Faster way to parse .pcap

Question

I am trying to parse huge .pcap files (~1-2GB each). I have tried to use scapy but so far it is much to slow. I have benchmarked timings using the following code with different values for count

from scapy.all import *
from scapy.layers.dns import DNSRR, DNS, DNSQR
import time
t0 = time.time()
pcap = 'output.pcap'
pkts = rdpcap(pcap, count=1000000)
t1 = time.time()
print(t1-t0)

It seems rdpcap() can only handle about 800 packets / second. Are there any optimizations for scapy, other tools, or good ways of scaling this process? Each packet is supposed to be DNS.

Answer 1

为了真正为您提供可靠的反馈，我需要您发布更大的代码块。但从我所看到的：

我的第一个想法是使用线程（如果可能的话：我不确定你的代码的其余部分是什么样的）

我的第二个想法是检查rdpcap（）函数的返回类型，并查看该变量类型的最大大小。可能是因为您尝试将更多数据放入缓冲区而不是可用空间。