I took this to our SR-IOV Guru, and he came back with this for you:
Problem description leads me to believe customer is unable to communicate between VM on a bridged network and VM with SR-IOV. I believe the packets from SR-IOV VM are dropped by the bridged interface. Workaround for this case was added to Linux kernel 3.4.x. Below is the excerpt of the actual scenario.
In SR-IOV enabled adapters, Physical Function (PF) does not work in bridge mode. When a bridge is created on the PF device, an emulation device in the Virtual Machine (VM) connects to this bridge cannot receive any unicast packets.
To avoid this from occurring, for each emulated device (virtio device in the VM) added to the bridge, the MAC address of the emulated device (virtio device in the VM) needs to be added manually to the forwarding database (FDB) filter table using the iproute2 package bridge tool. This can be done by executing the following command:
# bridge fdb add <MAC ADDR> dev <PF device interface>
To display if the mac address is added to the FDB table
# bridge fdb show dev < PF device interface >
The FDB entry when the emulated device is no longer in use or the guest to which it is assigned is moved to a different host must be deleted using this
command:
# bridge fdb del <MAC ADDR> dev <PF device interface>
Please instruct the customer to update the Linux kernel to version 3.4.x or newer and the commands above. I’d also recommend disabling VF Anti-spoofing if Linux Channel bonding driver is being used. Below is the command to disable it.
ip link set <nic interface name> vf <num> spoofchk off
Example: ip link set p5p2 vf 1 spoofchk off Disabling Spoofchk feature on VF # 1 for Ethernet interface P5P2.
Hope this helps
- Patrick