this post was submitted on 22 Nov 2024
5 points (100.0% liked)

homelab

6648 readers
21 users here now

founded 4 years ago
MODERATORS
CMonster@lemmy.ml
5
10GbE networking asymmetry in testing setup (hackertalks.com)
submitted 1 day ago* (last edited 1 day ago) by jet@hackertalks.com to c/homelab@lemmy.ml
9 comments fedilink hide all child comments
 

My testing setup is, all on the same subnet, ipv4

  • Windows Machine with Intel x520
    • Direct Connect 10Gbps cable
  • USW Aggregation switch (10Gbps)
    • Direct Connect 10Gbps cable
  • Synology NAS with Intel x520
    • SRVIO Connection
  • Debian VM

iperf3 Windows To Debian: 6Gbits/sec

.\iperf3.exe -c 192.168.11.57  --get-server-output
Connecting to host 192.168.11.57, port 5201
[  5] local 192.168.11.132 port 56855 connected to 192.168.11.57 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   817 MBytes  6.79 Gbits/sec
[  5]   1.01-2.00   sec   806 MBytes  6.82 Gbits/sec
[  5]   2.00-3.01   sec   822 MBytes  6.85 Gbits/sec
[  5]   3.01-4.00   sec   805 MBytes  6.81 Gbits/sec
[  5]   4.00-5.01   sec   818 MBytes  6.81 Gbits/sec
[  5]   5.01-6.00   sec   806 MBytes  6.82 Gbits/sec
[  5]   6.00-7.01   sec   821 MBytes  6.83 Gbits/sec
[  5]   7.01-8.00   sec   805 MBytes  6.80 Gbits/sec
[  5]   8.00-9.01   sec   820 MBytes  6.82 Gbits/sec
[  5]   9.01-10.00  sec   809 MBytes  6.84 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  7.94 GBytes  6.82 Gbits/sec                  sender
[  5]   0.00-10.00  sec  7.94 GBytes  6.82 Gbits/sec                  receiver

Server output:
-----------------------------------------------------------
Server listening on 5201 (test #9)
-----------------------------------------------------------
Accepted connection from 192.168.11.132, port 56854
[  5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56855
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   806 MBytes  6.76 Gbits/sec
[  5]   1.00-2.00   sec   812 MBytes  6.82 Gbits/sec
[  5]   2.00-3.00   sec   816 MBytes  6.85 Gbits/sec
[  5]   3.00-4.00   sec   812 MBytes  6.81 Gbits/sec
[  5]   4.00-5.00   sec   812 MBytes  6.81 Gbits/sec
[  5]   5.00-6.00   sec   812 MBytes  6.81 Gbits/sec
[  5]   6.00-7.00   sec   815 MBytes  6.84 Gbits/sec
[  5]   7.00-8.00   sec   811 MBytes  6.80 Gbits/sec
[  5]   8.00-9.00   sec   814 MBytes  6.82 Gbits/sec
[  5]   9.00-10.00  sec   815 MBytes  6.84 Gbits/sec
[  5]  10.00-10.00  sec  1.12 MBytes  4.81 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  7.94 GBytes  6.82 Gbits/sec                  receiver


iperf Done.

iperf3 debian to windows 9.5Gbits/sec

.\iperf3.exe -c 192.168.11.57  --get-server-output -R
Connecting to host 192.168.11.57, port 5201
Reverse mode, remote host 192.168.11.57 is sending
[  5] local 192.168.11.132 port 56845 connected to 192.168.11.57 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  1.11 GBytes  9.40 Gbits/sec
[  5]   1.01-2.00   sec  1.09 GBytes  9.47 Gbits/sec
[  5]   2.00-3.01   sec  1.11 GBytes  9.47 Gbits/sec
[  5]   3.01-4.00   sec  1.09 GBytes  9.47 Gbits/sec
[  5]   4.00-5.01   sec  1.11 GBytes  9.47 Gbits/sec
[  5]   5.01-6.00   sec  1.09 GBytes  9.47 Gbits/sec
[  5]   6.00-7.01   sec  1.11 GBytes  9.47 Gbits/sec
[  5]   7.01-8.00   sec  1.09 GBytes  9.47 Gbits/sec
[  5]   8.00-9.01   sec  1.11 GBytes  9.47 Gbits/sec
[  5]   9.01-10.00  sec  1.09 GBytes  9.47 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.46 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.0 GBytes  9.46 Gbits/sec                  receiver

Server output:
-----------------------------------------------------------
Server listening on 5201 (test #7)
-----------------------------------------------------------
Accepted connection from 192.168.11.132, port 56844
[  5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56845
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.01 MBytes
[  5]   1.00-2.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   3.00-4.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   4.00-5.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   5.00-6.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   6.00-7.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   7.00-8.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   8.00-9.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   9.00-10.00  sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]  10.00-10.00  sec  2.50 MBytes  7.72 Gbits/sec    0   2.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.46 Gbits/sec    0             sender


iperf Done.

I find that rather curious, something in the windows 10 tcp settings that limit the outgoing throughput, window size maybe?

Debian MTU

ip link show ens3 | grep -i "mtu"
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

::: spoiler Windows MTU

netsh interface ipv4 show subinterfaces

   MTU  MediaSenseState   Bytes In  Bytes Out  Interface
------  ---------------  ---------  ---------  -------------
  1500                1  273580016987  64376522487  Ethernet 4
you are viewing a single comment's thread
view the rest of the comments
[–] InEnduringGrowStrong@sh.itjust.works 2 points 1 day ago (1 children)

Use the -P flag to set the number of parallel streams. Probably, TCP windows size is limiting you on a single TCP connection but multiple connections would work fine.

[–] jet@hackertalks.com 1 points 1 day ago (2 children)

That is a great idea to test!

iperf windows to debian -P4 6Gbit/sec

.\iperf3.exe -c 192.168.11.57  --get-server-output  --dont-fragmen -P 4
Connecting to host 192.168.11.57, port 5201
[  5] local 192.168.11.132 port 56910 connected to 192.168.11.57 port 5201
[  7] local 192.168.11.132 port 56911 connected to 192.168.11.57 port 5201
[  9] local 192.168.11.132 port 56912 connected to 192.168.11.57 port 5201
[ 11] local 192.168.11.132 port 56913 connected to 192.168.11.57 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   190 MBytes  1.59 Gbits/sec
[  7]   0.00-1.00   sec   192 MBytes  1.60 Gbits/sec
[  9]   0.00-1.00   sec   192 MBytes  1.60 Gbits/sec
[ 11]   0.00-1.00   sec   189 MBytes  1.58 Gbits/sec
[SUM]   0.00-1.00   sec   764 MBytes  6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.01   sec   190 MBytes  1.58 Gbits/sec
[  7]   1.00-2.01   sec   190 MBytes  1.58 Gbits/sec
[  9]   1.00-2.01   sec   190 MBytes  1.58 Gbits/sec
[ 11]   1.00-2.01   sec   190 MBytes  1.58 Gbits/sec
[SUM]   1.00-2.01   sec   760 MBytes  6.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.01-3.00   sec   186 MBytes  1.58 Gbits/sec
[  7]   2.01-3.00   sec   189 MBytes  1.60 Gbits/sec
[  9]   2.01-3.00   sec   189 MBytes  1.60 Gbits/sec
[ 11]   2.01-3.00   sec   189 MBytes  1.60 Gbits/sec
[SUM]   2.01-3.00   sec   754 MBytes  6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.01   sec   190 MBytes  1.58 Gbits/sec
[  7]   3.00-4.01   sec   190 MBytes  1.58 Gbits/sec
[  9]   3.00-4.01   sec   190 MBytes  1.58 Gbits/sec
[ 11]   3.00-4.01   sec   190 MBytes  1.58 Gbits/sec
[SUM]   3.00-4.01   sec   761 MBytes  6.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.01-5.00   sec   188 MBytes  1.59 Gbits/sec
[  7]   4.01-5.00   sec   188 MBytes  1.59 Gbits/sec
[  9]   4.01-5.00   sec   188 MBytes  1.59 Gbits/sec
[ 11]   4.01-5.00   sec   188 MBytes  1.59 Gbits/sec
[SUM]   4.01-5.00   sec   754 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.01   sec   191 MBytes  1.59 Gbits/sec
[  7]   5.00-6.01   sec   191 MBytes  1.59 Gbits/sec
[  9]   5.00-6.01   sec   191 MBytes  1.59 Gbits/sec
[ 11]   5.00-6.01   sec   189 MBytes  1.58 Gbits/sec
[SUM]   5.00-6.01   sec   762 MBytes  6.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.01-7.00   sec   188 MBytes  1.59 Gbits/sec
[  7]   6.01-7.00   sec   189 MBytes  1.59 Gbits/sec
[  9]   6.01-7.00   sec   188 MBytes  1.59 Gbits/sec
[ 11]   6.01-7.00   sec   189 MBytes  1.60 Gbits/sec
[SUM]   6.01-7.00   sec   754 MBytes  6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.01   sec   192 MBytes  1.59 Gbits/sec
[  7]   7.00-8.01   sec   191 MBytes  1.59 Gbits/sec
[  9]   7.00-8.01   sec   192 MBytes  1.59 Gbits/sec
[ 11]   7.00-8.01   sec   191 MBytes  1.59 Gbits/sec
[SUM]   7.00-8.01   sec   766 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.01-9.00   sec   188 MBytes  1.59 Gbits/sec
[  7]   8.01-9.00   sec   188 MBytes  1.59 Gbits/sec
[  9]   8.01-9.00   sec   188 MBytes  1.59 Gbits/sec
[ 11]   8.01-9.00   sec   188 MBytes  1.59 Gbits/sec
[SUM]   8.01-9.00   sec   752 MBytes  6.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.01  sec   191 MBytes  1.59 Gbits/sec
[  7]   9.00-10.01  sec   191 MBytes  1.59 Gbits/sec
[  9]   9.00-10.01  sec   191 MBytes  1.59 Gbits/sec
[ 11]   9.00-10.01  sec   191 MBytes  1.59 Gbits/sec
[SUM]   9.00-10.01  sec   764 MBytes  6.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  sender
[  5]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[  7]   0.00-10.01  sec  1.86 GBytes  1.59 Gbits/sec                  sender
[  7]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[  9]   0.00-10.01  sec  1.86 GBytes  1.59 Gbits/sec                  sender
[  9]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[ 11]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  sender
[ 11]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[SUM]   0.00-10.01  sec  7.41 GBytes  6.36 Gbits/sec                  sender
[SUM]   0.00-10.01  sec  7.41 GBytes  6.35 Gbits/sec                  receiver

Server output:
-----------------------------------------------------------
Server listening on 5201 (test #15)
-----------------------------------------------------------
Accepted connection from 192.168.11.132, port 56909
[  5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56910
[  8] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56911
[ 10] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56912
[ 12] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56913
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   188 MBytes  1.57 Gbits/sec
[  8]   0.00-1.00   sec   190 MBytes  1.59 Gbits/sec
[ 10]   0.00-1.00   sec   190 MBytes  1.59 Gbits/sec
[ 12]   0.00-1.00   sec   187 MBytes  1.57 Gbits/sec
[SUM]   0.00-1.00   sec   754 MBytes  6.32 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   189 MBytes  1.58 Gbits/sec
[  8]   1.00-2.00   sec   189 MBytes  1.58 Gbits/sec
[ 10]   1.00-2.00   sec   189 MBytes  1.58 Gbits/sec
[ 12]   1.00-2.00   sec   189 MBytes  1.58 Gbits/sec
[SUM]   1.00-2.00   sec   754 MBytes  6.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   188 MBytes  1.58 Gbits/sec
[  8]   2.00-3.00   sec   191 MBytes  1.60 Gbits/sec
[ 10]   2.00-3.00   sec   191 MBytes  1.60 Gbits/sec
[ 12]   2.00-3.00   sec   191 MBytes  1.60 Gbits/sec
[SUM]   2.00-3.00   sec   760 MBytes  6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   189 MBytes  1.58 Gbits/sec
[  8]   3.00-4.00   sec   189 MBytes  1.58 Gbits/sec
[ 10]   3.00-4.00   sec   189 MBytes  1.58 Gbits/sec
[ 12]   3.00-4.00   sec   189 MBytes  1.58 Gbits/sec
[SUM]   3.00-4.00   sec   755 MBytes  6.34 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   190 MBytes  1.59 Gbits/sec
[  8]   4.00-5.00   sec   190 MBytes  1.59 Gbits/sec
[ 10]   4.00-5.00   sec   190 MBytes  1.59 Gbits/sec
[ 12]   4.00-5.00   sec   190 MBytes  1.59 Gbits/sec
[SUM]   4.00-5.00   sec   759 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   190 MBytes  1.59 Gbits/sec
[  8]   5.00-6.00   sec   190 MBytes  1.59 Gbits/sec
[ 10]   5.00-6.00   sec   190 MBytes  1.59 Gbits/sec
[ 12]   5.00-6.00   sec   188 MBytes  1.58 Gbits/sec
[SUM]   5.00-6.00   sec   758 MBytes  6.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   190 MBytes  1.59 Gbits/sec
[  8]   6.00-7.00   sec   190 MBytes  1.59 Gbits/sec
[ 10]   6.00-7.00   sec   190 MBytes  1.59 Gbits/sec
[ 12]   6.00-7.00   sec   190 MBytes  1.59 Gbits/sec
[SUM]   6.00-7.00   sec   759 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   190 MBytes  1.59 Gbits/sec
[  8]   7.00-8.00   sec   190 MBytes  1.59 Gbits/sec
[ 10]   7.00-8.00   sec   190 MBytes  1.59 Gbits/sec
[ 12]   7.00-8.00   sec   190 MBytes  1.59 Gbits/sec
[SUM]   7.00-8.00   sec   760 MBytes  6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   189 MBytes  1.59 Gbits/sec
[  8]   8.00-9.00   sec   189 MBytes  1.59 Gbits/sec
[ 10]   8.00-9.00   sec   189 MBytes  1.59 Gbits/sec
[ 12]   8.00-9.00   sec   189 MBytes  1.59 Gbits/sec
[SUM]   8.00-9.00   sec   758 MBytes  6.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   190 MBytes  1.59 Gbits/sec
[  8]   9.00-10.00  sec   190 MBytes  1.59 Gbits/sec
[ 10]   9.00-10.00  sec   190 MBytes  1.59 Gbits/sec
[ 12]   9.00-10.00  sec   190 MBytes  1.59 Gbits/sec
[SUM]   9.00-10.00  sec   759 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-10.01  sec  2.50 MBytes  1.51 Gbits/sec
[  8]  10.00-10.01  sec  2.50 MBytes  1.51 Gbits/sec
[ 10]  10.00-10.01  sec  2.50 MBytes  1.51 Gbits/sec
[ 12]  10.00-10.01  sec  2.50 MBytes  1.51 Gbits/sec
[SUM]  10.00-10.01  sec  10.0 MBytes  6.03 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[  8]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[ 10]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[ 12]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[SUM]   0.00-10.01  sec  7.41 GBytes  6.35 Gbits/sec                  receiver


iperf Done.

Though curiously i see the same behavior, 9.5Gbit linux->windows, and 5Gbit windows->linux... i'm perplexed

[–] InEnduringGrowStrong@sh.itjust.works 2 points 1 day ago (1 children)

Same results with -P but without the "don't fragment"?

[–] jet@hackertalks.com 1 points 1 day ago (1 children)

Yes, exactly the same results.

I'm thinking it might be on the linux side? i'm double checking my debian network stack now

[–] InEnduringGrowStrong@sh.itjust.works 3 points 1 day ago (1 children)

5 Gbps is the theoretical PCIe 1x limit, but I haven't seen those in a while, so probably not that.

I'm... half tempted to ask you if you can reverse the cables in the path lol.
Maybe a pair somewhere isn't as good as the others and affects data rate in that direction.
Far fetched, but..

[–] jet@hackertalks.com 1 points 1 day ago

I like the way you think! This is good engineering

I replaced both direct connect cables with new ones, I used new switch ports.

The asymmetry persists. Linux to windows 9.5GiB/s, Windows to Linux 6.5GiB/s

now we know it probably isn't the cables!