Cannot test QER policies with iperf3 (DL)

[xAlessandroC]

TLDR:

Cannot use iperf3 with eUPF. By default, the client isn't able to establish a connection with the server. If i disable the tcp offloading on the gw, the connection is established but I always get 0.00 bits/sec. The result is that i canno test the enforcment of QER in downlink.

Deployment architecture

graph LR
    UE["UE"] -->|192.168.60.100| ue1["ue1"]
    ue1 -->|192.168.60.50| gneth2["gneth2"]
    gneth2 --> GNB["GNB"]
    GNB -->|192.168.80.50| gnb_gneth0["gneth0"]
    gnb_gneth0 -->|192.168.80.200| eUPF_eth0["eth0"]
    eUPF_eth0 --> eUPF["eUPF"]
    eUPF -->|192.168.90.200| eUPF_eth2["eth2"]
    eUPF_eth2 -->|192.168.90.100| GW_eth0["eth0"]
    GW_eth0 --> GW["GW"]

    style UE fill:#f1c40f,stroke:#333,stroke-width:2px
    style GNB fill:#f1c40f,stroke:#333,stroke-width:2px
    style eUPF fill:#f1c40f,stroke:#333,stroke-width:2px
    style GW fill:#f1c40f,stroke:#333,stroke-width:2px

Loading

Issue report

Test

• The UE has an active PDU session with the IP 10.45.0.2
• The PDU Session has a flow with QFI (1), 5QI (9) and UL/DL limit (1Mbs).
• From the UE, an iperf3 server is set iperf3 -s.
• From the GW, the client is run iperf3 -c 10.45.0.2

UE iperf3 server log

-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.90.100, port 53198
[  5] local 10.45.0.2 port 5201 connected to 192.168.90.100 port 53202
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.06  sec  0.00 Bytes  0.00 bits/sec                  receiver

GW iperf3 client log

Connecting to host 10.45.0.2, port 5201
[  5] local 192.168.90.100 port 53202 connected to 10.45.0.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   105 KBytes   856 Kbits/sec    1   1.41 KBytes
[  5]   1.00-2.01   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes
[  5]   2.01-3.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   1.41 KBytes
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    1   1.41 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   105 KBytes  85.7 Kbits/sec    5             sender
[  5]   0.00-10.06  sec  0.00 Bytes  0.00 bits/sec                  receiver

iperf Done.

eUPF trace log

see attachment

Many thanks for your help, and let me know if you need some additional info!

eupf_trace_log.txt

[pirog-spb]

Hi Alessandro @xAlessandroC

Most of the problems we had with iperf were solved tuning drivers offloading features(because of incorrect checksum calculating) or MTU.

So using tcpdump please verify(if possible) that:

1. Iperf server generates packets in DL direction
2. These packets pass through eUPF
3. Iperf client receives DL packets
4. DL packets have valid checksum

Please see details from out iperf related cases here #555

--BR Alex

[xAlessandroC]

Hi @pirog-spb,
Many thanks for your response and help!

I have attached the pcap file, captured using tcpdump + Wireshark, related to a single execution of the iperf3 tool (I had to change the extension). The settings are the same as explained above.
Also, I forgot to mention that, on the GW, I disabled the TCP offloading with ethtool -K eth0 tx off.

In general, I noticed a lot of retransmissions. The packets seem to flow in the correct direction since I have data flowing from 192.168.90.100 to 10.45.0.2. Also, they pass through the eUPF since I have a trace log similar to the one I attached in the previous comment.

With respect to the issues (#396 and #170), I'm not able to understand if it is a problem with checksums. These are the outputs of the netstat -t -s command.

# UE (10.45.0.2)

Tcp:
    1 active connection openings
    2 passive connection openings
    0 failed connection attempts
    0 connection resets received
    1 connections established
    1856290 segments received
    42463462 segments sent out
    1055 segments retransmitted
    0 bad segments received
    0 resets sent
UdpLite:
TcpExt:
    1 TCP sockets finished time wait in fast timer
    50 packetes rejected in established connections because of timestamp
    2 delayed acks sent
    Quick ack mode was activated 232 times
    1181948 packet headers predicted
    298585 acknowledgments not containing data payload received
    129153 predicted acknowledgments
    TCPSackRecovery: 49
    Detected reordering 85 times using SACK
    Detected reordering 5 times using time stamp
    7 congestion windows fully recovered without slow start
    5 congestion windows partially recovered using Hoe heuristic
    TCPDSACKUndo: 6
    1055 fast retransmits
    TCPLossProbes: 2
    TCPBacklogCoalesce: 10199
    TCPDSACKOldSent: 232
    TCPDSACKRecv: 43
    TCPDSACKIgnoredOld: 2
    TCPDSACKIgnoredNoUndo: 1
    TCPSackMerged: 117
    TCPSackShiftFallback: 79
    TCPRcvCoalesce: 367940
    TCPOFOQueue: 2898
    TCPSpuriousRtxHostQueues: 164
    TCPAutoCorking: 4816
    TCPOrigDataSent: 41955124
    TCPHystartTrainDetect: 1
    TCPHystartTrainCwnd: 42
    TCPACKSkippedPAWS: 12
    TCPACKSkippedSeq: 8
    TCPDelivered: 41956138
    TCPAckCompressed: 1202
    TCPDSACKRecvSegs: 1055
IpExt:
    InOctets: 44945640067
    OutOctets: 60382288144
    InNoECTPkts: 31795937

# GW (192.168.90.100)

Tcp:
    7 active connection openings
    0 passive connection openings
    0 failed connection attempts
    0 connection resets received
    0 connections established
    10931 segments received
    6622 segments sent out
    8 segments retransmitted
    0 bad segments received
    0 resets sent
UdpLite:
TcpExt:
    6 TCP sockets finished time wait in fast timer
    4 delayed acks sent
    8802 packet headers predicted
    8 acknowledgments not containing data payload received
    18 predicted acknowledgments
    TCPLostRetransmit: 7
    TCPTimeouts: 9
    TCPLossProbes: 1
    TCPBacklogCoalesce: 10
    1 connections aborted due to timeout
    TCPRcvCoalesce: 3253
    TCPOFOQueue: 1762
    TCPOrigDataSent: 53
    TCPDelivered: 49
    TCPAckCompressed: 963
    TcpTimeoutRehash: 8
IpExt:
    InOctets: 44905467
    OutOctets: 400742
    InNoECTPkts: 30701

# UPF (192.168.90.200)

IcmpMsg:
    InType3: 19
    InType5: 3
    OutType3: 834
Tcp:
    2 active connection openings
    0 passive connection openings
    0 failed connection attempts
    2 connection resets received
    0 connections established
    820 segments received
    452 segments sent out
    0 segments retransmitted
    0 bad segments received
    2 resets sent
UdpLite:
TcpExt:
    1 ICMP packets dropped because socket was locked
    1 delayed acks sent
    714 packet headers predicted
    16 acknowledgments not containing data payload received
    4 predicted acknowledgments
    TCPBacklogCoalesce: 1
    2 connections reset due to early user close
    TCPRcvCoalesce: 399
    TCPOrigDataSent: 20
    TCPDelivered: 22
IpExt:
    InOctets: 4534954
    OutOctets: 215624
    InNoECTPkts: 5656

Finally, concerning issue #555, the guy had a fragmented IP and solved the issue by setting a proper MTU value. I think this is not the case.

Many thanks again!

iperf3_eupf_test.pcap.txt

[xAlessandroC]

for the filtering of the pcap I used "gtp or iperf3" on wireshark

[s5uishida]

@xAlessandroC

In my environment, I have confirmed that the following configuration works by setting the MTU of the DN/GW N6 interface to 1456.

[OK] iperf3 client (UE) <---> iperf3 server (DN/GW)
[OK] iperf3 server (UE) <---> iperf3 client (DN/GW)

Please refer to the information below [3] for your reference.

https://github.com/s5uishida/simple_confirmed_info_for_mobile_network#footnotes

--Shigeru

[xAlessandroC]

Hi @s5uishida,
Many thanks for the answer. By setting the MTU to 1456 and disabling TX offloading, I'm able to run iperf3 without getting a 0 bitrate.