在Docker容器中启动`pktgen`应用程序时vdev_probe（）失败

Question

我正在尝试使用Pktgen个虚拟端口在docker容器中启动一个openvswitch应用。我的理想系统的结构如下所示：

（1）构建dpdk-stable-17.11.4和ovs-2.9.0，并创建4个虚拟端口，如下所示：

Bridge "ovs-br0"
    Port "vhost-user2"
        Interface "vhost-user2"
            type: dpdkvhostuser
    Port "ovs-br0"
        Interface "ovs-br0"
            type: internal
    Port "vhost-user3"
        Interface "vhost-user3"
            type: dpdkvhostuser
    Port "vhost-user0"
        Interface "vhost-user0"
            type: dpdkvhostuser
    Port "vhost-user1"
        Interface "vhost-user1"
            type: dpdkvhostuser

当我查看创建虚拟端口的日志时，发现错误：

2019-01-27T15:59:23.346Z|00041|bridge|INFO|bridge ovs-br0: added interface ovs-br0 on port 65534
2019-01-27T15:59:23.375Z|00042|bridge|INFO|bridge ovs-br0: using datapath ID 000066b38a29f447
2019-01-27T15:59:23.375Z|00043|connmgr|INFO|ovs-br0: added service controller "punix:/usr/local/var/run/openvswitch/ovs-br0.mgmt"
2019-01-27T15:59:23.411Z|00044|dpdk|INFO|VHOST_CONFIG: vhost-user server: socket created, fd: 44
2019-01-27T15:59:23.411Z|00045|netdev_dpdk|INFO|Socket /usr/local/var/run/openvswitch/vhost-user0 created for vhost-user port vhost-user0
2019-01-27T15:59:23.411Z|00046|dpdk|INFO|VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/vhost-user0
2019-01-27T15:59:23.411Z|00047|netdev_dpdk|WARN|dpdkvhostuser ports are considered deprecated;  please migrate to dpdkvhostuserclient ports.
2019-01-27T15:59:23.411Z|00048|dpif_netdev|INFO|PMD thread on numa_id: 0, core id:  2 created.
2019-01-27T15:59:23.411Z|00049|dpif_netdev|INFO|There are 1 pmd threads on numa node 0
2019-01-27T15:59:23.412Z|00050|dpdk|ERR|RING: Cannot reserve memory for tailq
2019-01-27T15:59:23.424Z|00051|dpdk|ERR|RING: Cannot reserve memory for tailq
2019-01-27T15:59:23.434Z|00052|dpdk|ERR|RING: Cannot reserve memory for tailq
2019-01-27T15:59:23.443Z|00053|dpdk|ERR|RING: Cannot reserve memory for tailq
2019-01-27T15:59:23.458Z|00054|dpif_netdev|INFO|Core 2 on numa node 0 assigned port 'vhost-user0' rx queue 0 (measured processing cycles 0).
2019-01-27T15:59:23.458Z|00055|bridge|INFO|bridge ovs-br0: added interface vhost-user0 on port 1
2019-01-27T15:59:23.478Z|00056|dpdk|INFO|VHOST_CONFIG: vhost-user server: socket created, fd: 49
2019-01-27T15:59:23.478Z|00057|netdev_dpdk|INFO|Socket /usr/local/var/run/openvswitch/vhost-user1 created for vhost-user port vhost-user1
2019-01-27T15:59:23.478Z|00058|dpdk|INFO|VHOST_CONFIG: bind to /usr/local/var/run/openvswitch/vhost-user1
2019-01-27T15:59:23.478Z|00059|dpif_netdev|INFO|Core 2 on numa node 0 assigned port 'vhost-user0' rx queue 0 (measured processing cycles 0).
2019-01-27T15:59:23.479Z|00060|dpif_netdev|INFO|Core 2 on numa node 0 assigned port 'vhost-user1' rx queue 0 (measured processing cycles 0).

说RING: Cannot reserve memory for tailq。我不知道这个错误。我已经在DPDK中分配了一些hugepage。

（2）构建pktgen-3.5.0

（3）编写Dockerfile并启动docker。 Dockerfile：

FROM ubuntu:latest
RUN apt update -y
RUN apt-get install -y numactl libpcap-dev
WORKDIR /root/dpdk
COPY dpdk-stable-17.11.4 /root/dpdk/.
COPY pktgen-3.5.0 /root/pktgen/.
RUN ln -s /root/pktgen/app/x86_64-native-linuxapp-gcc/pktgen /usr/bin/pktgen
ENV PATH "$PATH:/root/dpdk/x86_64-native-linuxapp-gcc/app/"

构建图像：

sudo docker build -t pktgen-docker .

启动容器：

sudo docker run -ti --privileged --name=pktgen-docker \
    -v /mnt/huge:/mnt/huge -v /usr/local/var/run/openvswitch:/var/run/openswitch \
    pktgen-docker:latest

（4）启动pktgen应用。

app/x86_64-native-linuxapp-gcc/pktgen -l 0-1 -n 3 --file-prefix=pktgen-docker --no-pci --log-level=8\
    --vdev 'net_virtio_user0,mac=00:00:00:00:00:05,path=/var/run/openvswitch/vhost-user0' \
    --vdev 'net_virtio_user1,mac=00:00:00:00:00:01,path=/var/run/openvswitch/vhost-user1' \
    -- -T -P -m '1.[0-1]'

我只使用2个lcore，lcore-1用于2个虚拟端口的rx/tx。我使用--vdev端口（即ovs-dpdk，vhost_user0）设置了2个vhost_user1。

但是，错误来了：

Copyright (c) <2010-2017>, Intel Corporation. All rights reserved. Powered by DPDK
sh: 1: lspci: not found
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 3 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 4 lcore(s)
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Module /sys/module/vfio not found! error 2 (No such file or directory)
EAL: Setting up physically contiguous memory...
EAL: Trying to obtain current memory policy.
EAL: Hugepage /mnt/huge/pktgen-dockermap_1 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_2 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_3 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_4 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_5 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_6 is on socket 0
...
EAL: Hugepage /mnt/huge/pktgen-dockermap_167 is on socket 0
EAL: Hugepage /mnt/huge/pktgen-dockermap_0 is on socket 0
EAL: Ask a virtual area of 0x15000000 bytes
EAL: Virtual area found at 0x100000000 (size = 0x15000000)
EAL: Requesting 168 pages of size 2MB from socket 0
EAL: TSC frequency is ~2594018 KHz
EAL: Master lcore 0 is ready (tid=8f222900;cpuset=[0])
EAL: lcore 1 is ready (tid=8d5fd700;cpuset=[1])
vdev_probe(): failed to initialize net_virtio_user0 device

vdev_probe(): failed to initialize net_virtio_user1 device

EAL: Bus (vdev) probe failed.
Lua 5.3.4  Copyright (C) 1994-2017 Lua.org, PUC-Rio
   Copyright (c) <2010-2017>, Intel Corporation. All rights reserved.
   Pktgen created by: Keith Wiles -- >>> Powered by DPDK <<<

>>> Packet Burst 64, RX Desc 1024, TX Desc 2048, mbufs/port 16384, mbuf cache 2048
!PANIC!: *** Did not find any ports to use ***
PANIC in pktgen_config_ports():
*** Did not find any ports to use ***6: [app/x86_64-native-linuxapp-gcc/pktgen(_start+0x2a) [0x5592bd24d66a]]
5: [/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f5b8de20b97]]
4: [app/x86_64-native-linuxapp-gcc/pktgen(main+0xe67) [0x5592bd24a1e7]]
3: [app/x86_64-native-linuxapp-gcc/pktgen(pktgen_config_ports+0x170b) [0x5592bd277a4b]]
2: [app/x86_64-native-linuxapp-gcc/pktgen(__rte_panic+0xc5) [0x5592bd244bb7]]
1: [app/x86_64-native-linuxapp-gcc/pktgen(rte_dump_stack+0x2e) [0x5592bd2dcb3e]]
Aborted (core dumped)

• 我使用IGB_UIO而不是VFIO。
• 我没有设置--socket-mem，并且lcore1和lcore2都准备好了。
• --vdev无法初始化。我已检查/var/run/openvswitch，vhost-user0和vhost-user1存在。

感谢您分享您的想法。最好的祝福。

Answer 1

OVS在服务器模式下运行时，会在目录中创建virtio-user袜子。 pktgen也以virtio-user模式启动。由于两者都在服务器模式下运行，因此会发生冲突。

正确的方法是使用virtio_userx在客户端模式下启动pktgen。

注释：在注释中解释。这更多是使用问题