I found an issue and I've narrowed it down to the Intel driver.
Just to show the work I've put into this:
- I used iperf to test the bandwidth on all server types (Physical Linux install and inside a VM)
- I have an existing VMware support ticket and we've worked it down to the driver.
- I tested the issue with three different x540-T2 adapters and a x520-DA2 adapter.
Problem: When a server with the intel x540-T2 adapter is connected at 10Gbps and tries to transmit to another server that is connected at 1Gbps, All flows out of the 10Gbps adapter drops to a max of 1Gbps until the 1Gbps flows stop.
Let me clarify a bit:
Server A has an Intel x540-T2 adapter and is connected at 10Gbps.
Server B has an Intel x540-T2 adapter and is connected at 10Gbps.
Server C has a HP Server adapter and is connected at 1Gbps.
I start a Transmit flow from Server A to Server B and I see the iperf session reach full line rate (10Gbps)
I start a concurrent iperf flow From server A to Server C.
My expectation is that Server A will continue to transmit at 10Gbps (total), Server B will drop down 1Gbps to (8-9Gbps) to allow bandwidth for the 1Gbps flow to Server C.
BUT, this does not happen.
Server A will flop down to 1.5Gbps transmit and Server B and Server C will now receive 500-700Mbps until the flow to Server C stops then the flow from Server A to Server B goes back to 10Gbps.
Driver data:
If I use driver version 3.7.13.7 this problem does not exist! Everything works as expected and Server A continues to transmit at 10Gbps while Server B drops to 8Gbps+ and Server C receives at a full 1Gbps.
If I Update to any version after that, it has the transmit flop.
Driver Version 3.7.13.7 is the default driver used in the ESXi 5.5 build 1331820 install ISO.
Newer version tested: (these are listed as compatible with ESXi 5.5 on the VMware compatibility guide)
3.15.1.8
3.18.7
3.19.1
Some extra info from ESXi cli:
###### This is the working Driver:########
~ # vmkchdev -l | grep vmnic
0000:01:00.0 8086:1528 8086:001a vmkernel vmnic0
0000:01:00.1 8086:1528 8086:001a vmkernel vmnic1
~ # ethtool -i vmnic0
driver: ixgbe
version: 3.7.13.7.14iov-NAPI
firmware-version: 0x80000389
bus-info: 0000:01:00.0
~ # esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:01:00.00 ixgbe Up 10000Mbps Full a0:36:9f:01:b6:28 9000 Intel Corporation Ethernet Controller X540-AT2
###### This is one of the broken Driver versions:########
~ # vmkchdev -l | grep vmnic
0000:01:00.0 8086:1528 8086:0001 vmkernel vmnic0
0000:01:00.1 8086:1528 8086:0001 vmkernel vmnic1
~ # ethtool -i vmnic0
driver: ixgbe
version: 3.15.1.8iov
firmware-version: 0x80000389
bus-info: 0000:01:00.0
~ # esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:01:00.00 ixgbe Up 10000Mbps Full a0:36:9f:01:b6:28 9000 Intel Corporation Ethernet Controller X540-AT2
Things to note here are the VID and DID are still being detected correctly as 8086 and 1528.
Any thoughts or questions?
I have a ton of screen shots if needed to prove the case.