DPDK on SR-IOV

After successfully configuring SR-IOV on VMs, I had to set DPDK for the VMs for full performance.

1. Driver update

For VMs to use DPDK, driver updates were needed:

At host:

#Assume Driver Already downloaded and installed beforehand
#Install Driver (neohost-backend not working on x86_64/amd64)
sudo ./mlnxofedinstall --dpdk --upstream-libs --enable-sriov --hypervisor --force --without-neohost-backend

#reboot
sudo /etc/init.d/openibd restart
reboot

At VMs:

# Download Driver
wget https://content.mellanox.com/ofed/MLNX_OFED-4.9-6.0.6.0/MLNX_OFED_LINUX-4.9-6.0.6.0-ubuntu18.04-x86_64.iso

#Mount Driver
sudo mkdir /mnt/iso
mount -o loop /share/MLNX_OFED_LINUX-4.2-1.0.0.0-ubuntu16.04-x86_64.iso /mnt/iso
cd /mnt/iso

#Install Driver (neohost-backend not working on x86_64/amd64)
sudo ./mlnxofedinstall --dpdk --upstream-libs --guest --force --without-neohost-backend

#reboot
sudo /etc/init.d/openibd restart
reboot

*Note that compatability with DPDK <-> Pktgen is fragile. The below Steps were done using

DPDK - v.20.08
Pktgen - v.20.09

2. DPDK Build

First we need to download DPDK Pktgen and libbpf codes:

# Download and unzip DPDK
wget https://codeload.github.com/DPDK/dpdk/tar.gz/refs/tags/v20.08
tar xJf dpdk-<version>.tar.xz
cd dpdk-<version>

#Download & Install Meson/Ninja
sudo apt-get -y install python3-pip
pip3 install meson
pip3 install ninja
pip3 install pyelftools
reboot

#DPDK Config & Install
meson setup -Denable_kmods=true build
cd ./build/
sudo ninja && sudo ninja install && sudo ldconfig

#Change mlx4 setting
vim config/common_base
# Change CONFIG_RTE_LIBRTE_MLX4_PMD=y

#Hugepage setting
echo 1024 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepage
or
sudo sysctl -w vm.nr_hugepages=102400

3. Pktgen Install

#Get Pktgen
wget https://codeload.github.com/pktgen/Pktgen-DPDK/tar.gz/refs/tags/pktgen-20.09.0
tar -xvf pktgen-21.11.0.tar.gz 
cd Pktgen-DPDK-pktgen-21.11.0/

#Download Dependencies
sudo apt install cmake
sudo apt-get install -y pkg-config
sudo apt-get install -y libdpdk-dev
sudo apt-get install -y libnuma-dev
sudo apt-get install git libpcap-dev

#Install Pktgen
meson setup build
ninja -C build
sudo ninja -C build install

# Run Pktgen

4. Set DPDK

Due to unknown issues, pktgen was unavailable to find the ports. Thus troubleshooting was needed.

*This part has a lot of troubleshooting.
Hope that I could find a way that allows DPDK on SR-IOV VMs.
< Referred Links >
1. DPDK Linux Driver
MLX4 poll mode driver library
First, after having the link down, I need to bind the interface to DPDK.

4.1 vfio-pci Driver

It is said:

It is recommended that vfio-pci be used as the kernel module for DPDK-bound ports in all cases. If an IOMMU is unavailable, the vfio-pci can be used in no-iommu mode. If, for some reason, vfio is unavailable, then UIO-based modules, igb_uio and uio_pci_generic may be used. See section UIO for details.

Thus, I tried to add this driver via:

sudo modprobe vfio-pci

However, it was added, yet when trying to bind it stated:

4.2 igb_uio Driver

Then I found,

For some devices which lack support for legacy interrupts, e.g. virtual function (VF) devices, the igb_uio module may be needed in place of uio_pci_generic.

Because we're dealing with VFs with SR-IOV, I believed this was the right shot. To use this driver, I had to download and make:

# Install driver
git clone git://dpdk.org/dpdk-kmods
make dpdk-kmods/linux/igb_uio

# Set driver
sudo modprobe uio
sudo insmod igb_uio.ko

# Bind driver (at DPDK usertool dir)
./dpdk-devbind.py -b igb_uio 00:09.0

And with ./dpdk-devbing.py -s I was able to see it was successfully binded:

However, after trying the helloworld example, it aborted as below:

4.3 mlx4_core driver

Then, it worked using the original mlx4_core without bind/unbinding anything.

• One thing to notify is that enough hugepage was needed (atleast 1024) to operate the DPDK-Pktgen

5.Start Pktgen

Then, with the command as below, we can use the DPDK-Pktgen:

sudo /usr/local/bin/pktgen -l 0-8 -n 2 -- -T -P -m "[1-4:5-8].0"