homelab

6646 readers

31 users here now

founded 4 years ago

MODERATORS

CMonster@lemmy.ml

10GbE networking asymmetry in testing setup (hackertalks.com)

submitted 1 day ago* (last edited 1 day ago) by jet@hackertalks.com to c/homelab@lemmy.ml

9 comments fedilink hide all child comments

My testing setup is, all on the same subnet, ipv4

Windows Machine with Intel x520
- Direct Connect 10Gbps cable
USW Aggregation switch (10Gbps)
- Direct Connect 10Gbps cable
Synology NAS with Intel x520
- SRVIO Connection
Debian VM

iperf3 Windows To Debian: 6Gbits/sec

.\iperf3.exe -c 192.168.11.57  --get-server-output
Connecting to host 192.168.11.57, port 5201
[  5] local 192.168.11.132 port 56855 connected to 192.168.11.57 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   817 MBytes  6.79 Gbits/sec
[  5]   1.01-2.00   sec   806 MBytes  6.82 Gbits/sec
[  5]   2.00-3.01   sec   822 MBytes  6.85 Gbits/sec
[  5]   3.01-4.00   sec   805 MBytes  6.81 Gbits/sec
[  5]   4.00-5.01   sec   818 MBytes  6.81 Gbits/sec
[  5]   5.01-6.00   sec   806 MBytes  6.82 Gbits/sec
[  5]   6.00-7.01   sec   821 MBytes  6.83 Gbits/sec
[  5]   7.01-8.00   sec   805 MBytes  6.80 Gbits/sec
[  5]   8.00-9.01   sec   820 MBytes  6.82 Gbits/sec
[  5]   9.01-10.00  sec   809 MBytes  6.84 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  7.94 GBytes  6.82 Gbits/sec                  sender
[  5]   0.00-10.00  sec  7.94 GBytes  6.82 Gbits/sec                  receiver

Server output:
-----------------------------------------------------------
Server listening on 5201 (test #9)
-----------------------------------------------------------
Accepted connection from 192.168.11.132, port 56854
[  5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56855
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   806 MBytes  6.76 Gbits/sec
[  5]   1.00-2.00   sec   812 MBytes  6.82 Gbits/sec
[  5]   2.00-3.00   sec   816 MBytes  6.85 Gbits/sec
[  5]   3.00-4.00   sec   812 MBytes  6.81 Gbits/sec
[  5]   4.00-5.00   sec   812 MBytes  6.81 Gbits/sec
[  5]   5.00-6.00   sec   812 MBytes  6.81 Gbits/sec
[  5]   6.00-7.00   sec   815 MBytes  6.84 Gbits/sec
[  5]   7.00-8.00   sec   811 MBytes  6.80 Gbits/sec
[  5]   8.00-9.00   sec   814 MBytes  6.82 Gbits/sec
[  5]   9.00-10.00  sec   815 MBytes  6.84 Gbits/sec
[  5]  10.00-10.00  sec  1.12 MBytes  4.81 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  7.94 GBytes  6.82 Gbits/sec                  receiver


iperf Done.

iperf3 debian to windows 9.5Gbits/sec

.\iperf3.exe -c 192.168.11.57  --get-server-output -R
Connecting to host 192.168.11.57, port 5201
Reverse mode, remote host 192.168.11.57 is sending
[  5] local 192.168.11.132 port 56845 connected to 192.168.11.57 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  1.11 GBytes  9.40 Gbits/sec
[  5]   1.01-2.00   sec  1.09 GBytes  9.47 Gbits/sec
[  5]   2.00-3.01   sec  1.11 GBytes  9.47 Gbits/sec
[  5]   3.01-4.00   sec  1.09 GBytes  9.47 Gbits/sec
[  5]   4.00-5.01   sec  1.11 GBytes  9.47 Gbits/sec
[  5]   5.01-6.00   sec  1.09 GBytes  9.47 Gbits/sec
[  5]   6.00-7.01   sec  1.11 GBytes  9.47 Gbits/sec
[  5]   7.01-8.00   sec  1.09 GBytes  9.47 Gbits/sec
[  5]   8.00-9.01   sec  1.11 GBytes  9.47 Gbits/sec
[  5]   9.01-10.00  sec  1.09 GBytes  9.47 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.46 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  11.0 GBytes  9.46 Gbits/sec                  receiver

Server output:
-----------------------------------------------------------
Server listening on 5201 (test #7)
-----------------------------------------------------------
Accepted connection from 192.168.11.132, port 56844
[  5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56845
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.10 GBytes  9.42 Gbits/sec    0   2.01 MBytes
[  5]   1.00-2.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   2.00-3.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   3.00-4.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   4.00-5.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   5.00-6.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   6.00-7.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   7.00-8.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   8.00-9.00   sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]   9.00-10.00  sec  1.10 GBytes  9.47 Gbits/sec    0   2.01 MBytes
[  5]  10.00-10.00  sec  2.50 MBytes  7.72 Gbits/sec    0   2.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  11.0 GBytes  9.46 Gbits/sec    0             sender


iperf Done.

I find that rather curious, something in the windows 10 tcp settings that limit the outgoing throughput, window size maybe?

Debian MTU

ip link show ens3 | grep -i "mtu"
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

::: spoiler Windows MTU

netsh interface ipv4 show subinterfaces

   MTU  MediaSenseState   Bytes In  Bytes Out  Interface
------  ---------------  ---------  ---------  -------------
  1500                1  273580016987  64376522487  Ethernet 4

all 10 comments

sorted by: hot top controversial new old

[–] InEnduringGrowStrong@sh.itjust.works 2 points 1 day ago (1 children)

Use the -P flag to set the number of parallel streams. Probably, TCP windows size is limiting you on a single TCP connection but multiple connections would work fine.

[–] jet@hackertalks.com 1 points 1 day ago (2 children)

That is a great idea to test!

iperf windows to debian -P4 6Gbit/sec

.\iperf3.exe -c 192.168.11.57  --get-server-output  --dont-fragmen -P 4
Connecting to host 192.168.11.57, port 5201
[  5] local 192.168.11.132 port 56910 connected to 192.168.11.57 port 5201
[  7] local 192.168.11.132 port 56911 connected to 192.168.11.57 port 5201
[  9] local 192.168.11.132 port 56912 connected to 192.168.11.57 port 5201
[ 11] local 192.168.11.132 port 56913 connected to 192.168.11.57 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   190 MBytes  1.59 Gbits/sec
[  7]   0.00-1.00   sec   192 MBytes  1.60 Gbits/sec
[  9]   0.00-1.00   sec   192 MBytes  1.60 Gbits/sec
[ 11]   0.00-1.00   sec   189 MBytes  1.58 Gbits/sec
[SUM]   0.00-1.00   sec   764 MBytes  6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.01   sec   190 MBytes  1.58 Gbits/sec
[  7]   1.00-2.01   sec   190 MBytes  1.58 Gbits/sec
[  9]   1.00-2.01   sec   190 MBytes  1.58 Gbits/sec
[ 11]   1.00-2.01   sec   190 MBytes  1.58 Gbits/sec
[SUM]   1.00-2.01   sec   760 MBytes  6.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.01-3.00   sec   186 MBytes  1.58 Gbits/sec
[  7]   2.01-3.00   sec   189 MBytes  1.60 Gbits/sec
[  9]   2.01-3.00   sec   189 MBytes  1.60 Gbits/sec
[ 11]   2.01-3.00   sec   189 MBytes  1.60 Gbits/sec
[SUM]   2.01-3.00   sec   754 MBytes  6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.01   sec   190 MBytes  1.58 Gbits/sec
[  7]   3.00-4.01   sec   190 MBytes  1.58 Gbits/sec
[  9]   3.00-4.01   sec   190 MBytes  1.58 Gbits/sec
[ 11]   3.00-4.01   sec   190 MBytes  1.58 Gbits/sec
[SUM]   3.00-4.01   sec   761 MBytes  6.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.01-5.00   sec   188 MBytes  1.59 Gbits/sec
[  7]   4.01-5.00   sec   188 MBytes  1.59 Gbits/sec
[  9]   4.01-5.00   sec   188 MBytes  1.59 Gbits/sec
[ 11]   4.01-5.00   sec   188 MBytes  1.59 Gbits/sec
[SUM]   4.01-5.00   sec   754 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.01   sec   191 MBytes  1.59 Gbits/sec
[  7]   5.00-6.01   sec   191 MBytes  1.59 Gbits/sec
[  9]   5.00-6.01   sec   191 MBytes  1.59 Gbits/sec
[ 11]   5.00-6.01   sec   189 MBytes  1.58 Gbits/sec
[SUM]   5.00-6.01   sec   762 MBytes  6.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.01-7.00   sec   188 MBytes  1.59 Gbits/sec
[  7]   6.01-7.00   sec   189 MBytes  1.59 Gbits/sec
[  9]   6.01-7.00   sec   188 MBytes  1.59 Gbits/sec
[ 11]   6.01-7.00   sec   189 MBytes  1.60 Gbits/sec
[SUM]   6.01-7.00   sec   754 MBytes  6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.01   sec   192 MBytes  1.59 Gbits/sec
[  7]   7.00-8.01   sec   191 MBytes  1.59 Gbits/sec
[  9]   7.00-8.01   sec   192 MBytes  1.59 Gbits/sec
[ 11]   7.00-8.01   sec   191 MBytes  1.59 Gbits/sec
[SUM]   7.00-8.01   sec   766 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.01-9.00   sec   188 MBytes  1.59 Gbits/sec
[  7]   8.01-9.00   sec   188 MBytes  1.59 Gbits/sec
[  9]   8.01-9.00   sec   188 MBytes  1.59 Gbits/sec
[ 11]   8.01-9.00   sec   188 MBytes  1.59 Gbits/sec
[SUM]   8.01-9.00   sec   752 MBytes  6.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.01  sec   191 MBytes  1.59 Gbits/sec
[  7]   9.00-10.01  sec   191 MBytes  1.59 Gbits/sec
[  9]   9.00-10.01  sec   191 MBytes  1.59 Gbits/sec
[ 11]   9.00-10.01  sec   191 MBytes  1.59 Gbits/sec
[SUM]   9.00-10.01  sec   764 MBytes  6.36 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  sender
[  5]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[  7]   0.00-10.01  sec  1.86 GBytes  1.59 Gbits/sec                  sender
[  7]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[  9]   0.00-10.01  sec  1.86 GBytes  1.59 Gbits/sec                  sender
[  9]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[ 11]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  sender
[ 11]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[SUM]   0.00-10.01  sec  7.41 GBytes  6.36 Gbits/sec                  sender
[SUM]   0.00-10.01  sec  7.41 GBytes  6.35 Gbits/sec                  receiver

Server output:
-----------------------------------------------------------
Server listening on 5201 (test #15)
-----------------------------------------------------------
Accepted connection from 192.168.11.132, port 56909
[  5] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56910
[  8] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56911
[ 10] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56912
[ 12] local 192.168.11.57 port 5201 connected to 192.168.11.132 port 56913
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   188 MBytes  1.57 Gbits/sec
[  8]   0.00-1.00   sec   190 MBytes  1.59 Gbits/sec
[ 10]   0.00-1.00   sec   190 MBytes  1.59 Gbits/sec
[ 12]   0.00-1.00   sec   187 MBytes  1.57 Gbits/sec
[SUM]   0.00-1.00   sec   754 MBytes  6.32 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   189 MBytes  1.58 Gbits/sec
[  8]   1.00-2.00   sec   189 MBytes  1.58 Gbits/sec
[ 10]   1.00-2.00   sec   189 MBytes  1.58 Gbits/sec
[ 12]   1.00-2.00   sec   189 MBytes  1.58 Gbits/sec
[SUM]   1.00-2.00   sec   754 MBytes  6.33 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   188 MBytes  1.58 Gbits/sec
[  8]   2.00-3.00   sec   191 MBytes  1.60 Gbits/sec
[ 10]   2.00-3.00   sec   191 MBytes  1.60 Gbits/sec
[ 12]   2.00-3.00   sec   191 MBytes  1.60 Gbits/sec
[SUM]   2.00-3.00   sec   760 MBytes  6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   189 MBytes  1.58 Gbits/sec
[  8]   3.00-4.00   sec   189 MBytes  1.58 Gbits/sec
[ 10]   3.00-4.00   sec   189 MBytes  1.58 Gbits/sec
[ 12]   3.00-4.00   sec   189 MBytes  1.58 Gbits/sec
[SUM]   3.00-4.00   sec   755 MBytes  6.34 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   190 MBytes  1.59 Gbits/sec
[  8]   4.00-5.00   sec   190 MBytes  1.59 Gbits/sec
[ 10]   4.00-5.00   sec   190 MBytes  1.59 Gbits/sec
[ 12]   4.00-5.00   sec   190 MBytes  1.59 Gbits/sec
[SUM]   4.00-5.00   sec   759 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   190 MBytes  1.59 Gbits/sec
[  8]   5.00-6.00   sec   190 MBytes  1.59 Gbits/sec
[ 10]   5.00-6.00   sec   190 MBytes  1.59 Gbits/sec
[ 12]   5.00-6.00   sec   188 MBytes  1.58 Gbits/sec
[SUM]   5.00-6.00   sec   758 MBytes  6.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   190 MBytes  1.59 Gbits/sec
[  8]   6.00-7.00   sec   190 MBytes  1.59 Gbits/sec
[ 10]   6.00-7.00   sec   190 MBytes  1.59 Gbits/sec
[ 12]   6.00-7.00   sec   190 MBytes  1.59 Gbits/sec
[SUM]   6.00-7.00   sec   759 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   190 MBytes  1.59 Gbits/sec
[  8]   7.00-8.00   sec   190 MBytes  1.59 Gbits/sec
[ 10]   7.00-8.00   sec   190 MBytes  1.59 Gbits/sec
[ 12]   7.00-8.00   sec   190 MBytes  1.59 Gbits/sec
[SUM]   7.00-8.00   sec   760 MBytes  6.38 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   189 MBytes  1.59 Gbits/sec
[  8]   8.00-9.00   sec   189 MBytes  1.59 Gbits/sec
[ 10]   8.00-9.00   sec   189 MBytes  1.59 Gbits/sec
[ 12]   8.00-9.00   sec   189 MBytes  1.59 Gbits/sec
[SUM]   8.00-9.00   sec   758 MBytes  6.35 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   190 MBytes  1.59 Gbits/sec
[  8]   9.00-10.00  sec   190 MBytes  1.59 Gbits/sec
[ 10]   9.00-10.00  sec   190 MBytes  1.59 Gbits/sec
[ 12]   9.00-10.00  sec   190 MBytes  1.59 Gbits/sec
[SUM]   9.00-10.00  sec   759 MBytes  6.37 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]  10.00-10.01  sec  2.50 MBytes  1.51 Gbits/sec
[  8]  10.00-10.01  sec  2.50 MBytes  1.51 Gbits/sec
[ 10]  10.00-10.01  sec  2.50 MBytes  1.51 Gbits/sec
[ 12]  10.00-10.01  sec  2.50 MBytes  1.51 Gbits/sec
[SUM]  10.00-10.01  sec  10.0 MBytes  6.03 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[  8]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[ 10]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[ 12]   0.00-10.01  sec  1.85 GBytes  1.59 Gbits/sec                  receiver
[SUM]   0.00-10.01  sec  7.41 GBytes  6.35 Gbits/sec                  receiver


iperf Done.

Though curiously i see the same behavior, 9.5Gbit linux->windows, and 5Gbit windows->linux... i'm perplexed

[–] InEnduringGrowStrong@sh.itjust.works 2 points 1 day ago (1 children)

Same results with -P but without the "don't fragment"?

[–] jet@hackertalks.com 1 points 1 day ago (1 children)

Yes, exactly the same results.

I'm thinking it might be on the linux side? i'm double checking my debian network stack now

[–] InEnduringGrowStrong@sh.itjust.works 3 points 1 day ago (1 children)

5 Gbps is the theoretical PCIe 1x limit, but I haven't seen those in a while, so probably not that.

I'm... half tempted to ask you if you can reverse the cables in the path lol.
Maybe a pair somewhere isn't as good as the others and affects data rate in that direction.
Far fetched, but..

[–] jet@hackertalks.com 1 points 1 day ago

I like the way you think! This is good engineering

I replaced both direct connect cables with new ones, I used new switch ports.

The asymmetry persists. Linux to windows 9.5GiB/s, Windows to Linux 6.5GiB/s

now we know it probably isn't the cables!

[–] N0x0n@lemmy.ml 1 points 1 day ago (1 children)

Does 10Gb needs that much cooling?

[–] jet@hackertalks.com 2 points 1 day ago (1 children)

No, the photo is of a general computer. So most of the cooling is for the CPU and GPU.

SFP modules can get quite hot, especially if they're pushing high frequencies over copper. So you'll see some people complaining about their 10 gigabit copper SFP modules overheating. Especially on passively cooled network equipment.

Fiber optics don't get nearly as hot, and don't have overheating issues. Direct connect cables also do not get hot.

I'd much rather have all of my networking equipment with fiber optics, then copper. The trade-off is it's harder to splice your own fiber optic cable and put termination on it, the benefit is the equipment is much much cheaper, More reliable, cooler, goes further distance... A fiber optic SFP module cost about $10 for one, but a copper SFP module is like $100.

[–] N0x0n@lemmy.ml 2 points 1 day ago* (last edited 1 day ago)

Thanks for the clarification ! It wasn't clear to me what I was looking at first sight. I fought that was some self-hosted/DIY 10Gbs router/switch. Sorry if my question came the wrong way or was a stupid one. I'm just a curious guy !

Thanks for the additional information !!