Open
Description
Description:
I am trying to run the OMEC UPF with the OAI SMF on an EC2 instance. The setup involves running the UPF on Kubernetes (MicroK8s) as per the tutorial. The OAI SMF is running in Docker using Docker Compose. However, the UPF crashes with the error error: unexpected EOF when it receives a PFCP Association Request from the OAI SMF. Below are the details of the setup, logs, and steps to reproduce the issue.
Steps to Reproduce:
- Set up the OMEC UPF on Kubernetes (MicroK8s) as per the tutorial.
- Run the OAI SMF using Docker Compose with the provided
docker-compose-oai-nonrf-omec-upf.yamlfile, see attached zip folder. - Update the IP address in the Docker Compose file to match the UPF's ClusterIP:
extra_hosts: - "oai-upf:10.152.183.35"
- Start the OAI 5GC:
docker compose -f docker-compose-oai-nonrf-omec-upf.yaml up -d
- Observe the UPF crashing with the error
error: unexpected EOFwhen it receives a PFCP Association Request from the OAI SMF. (microk8s kubectl get events --all-namespaces --sort-by=.metadata.creationTimestamp.)
Expected Behavior:
The UPF should successfully establish a PFCP association with the OAI SMF and remain in a running state.
Actual Behavior:
The UPF crashes with the error error: unexpected EOF after receiving the PFCP Association Request. The Kubernetes events indicate that the UPF restarts due to InvalidDiskCapacity.
Logs and Error Messages:
PFCP Association Request (from OAI SMF):
Frame 108231: 78 bytes on wire (624 bits), 78 bytes captured (624 bits) on interface any, id 0
Interface id: 0 (any)
Interface name: any
Encapsulation type: Linux cooked-mode capture v1 (25)
Arrival Time: Mar 17, 2025 13:16:40.177487757 CET
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1742213800.177487757 seconds
[Time delta from previous captured frame: 0.000035102 seconds]
[Time delta from previous displayed frame: 0.000035102 seconds]
[Time since reference or first frame: 756.897843794 seconds]
Frame Number: 108231
Frame Length: 78 bytes (624 bits)
Capture Length: 78 bytes (624 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: sll:ethertype:ip:udp:pfcp]
[Coloring Rule Name: UDP]
[Coloring Rule String: udp]
Linux cooked capture v1
Packet type: Sent by us (4)
Link-layer address type: Ethernet (1)
Link-layer address length: 6
Source: ee:ee:ee:ee:ee:ee (ee:ee:ee:ee:ee:ee)
Unused: 62b3
Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: 172.16.0.10, Dst: 10.1.46.43
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 62
Identification: 0x97de (38878)
Flags: 0x40, Don't fragment
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
...0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 63
Protocol: UDP (17)
Header Checksum: 0xbf8a [validation disabled]
[Header checksum status: Unverified]
Source Address: 172.16.0.10
Destination Address: 10.1.46.43
User Datagram Protocol, Src Port: 20602, Dst Port: 8805
Source Port: 20602
Destination Port: 8805
Length: 42
Checksum: 0xe481 [unverified]
[Checksum Status: Unverified]
[Stream index: 881]
[Timestamps]
[Time since first frame: 0.000000000 seconds]
[Time since previous frame: 0.000000000 seconds]
UDP payload (34 bytes)
Packet Forwarding Control Protocol
Flags: 0x20
001. .... = Version: 1
...0 .... = Spare: 0
.... 0... = Spare: 0
.... .0.. = Follow On (FO): False
.... ..0. = Message Priority (MP): False
.... ...0 = SEID (S): False
Message Type: PFCP Association Setup Request (5)
Length: 30
Sequence Number: 4056313
Spare: 0
Node ID : FQDN: oai-smf
IE Type: Node ID (60)
IE Length: 9
0000 .... = Spare: 0
.... 0010 = Address Type: FQDN (2)
FQDN: oai-smf
Recovery Time Stamp : Mar 17, 2025 12:16:40.000000000 UTC
IE Type: Recovery Time Stamp (96)
IE Length: 4
Recovery Time Stamp: Mar 17, 2025 12:16:40.000000000 UTC
CP Function Features :
IE Type: CP Function Features (89)
IE Length: 1
0... .... = UIAUR: Not supported
.0.. .... = ARDR: Not supported
..0. .... = MPAS: Not supported
...0 .... = BUNDL: Not supported
.... 0... = SSET: Not supported
.... .0.. = EPFAR: Not supported
.... ..1. = OVRL: Supported
.... ...1 = LOAD: Supported
[Response In: 108232]
PFCP Association Response (from OMEC UPF):
Frame 108232: 101 bytes on wire (808 bits), 101 bytes captured (808 bits) on interface any, id 0
Interface id: 0 (any)
Interface name: any
Encapsulation type: Linux cooked-mode capture v1 (25)
Arrival Time: Mar 17, 2025 13:16:40.177824238 CET
[Time shift for this packet: 0.000000000 seconds]
Epoch Time: 1742213800.177824238 seconds
[Time delta from previous captured frame: 0.000336481 seconds]
[Time delta from previous displayed frame: 0.000336481 seconds]
[Time since reference or first frame: 756.898180275 seconds]
Frame Number: 108232
Frame Length: 101 bytes (808 bits)
Capture Length: 101 bytes (808 bits)
[Frame is marked: False]
[Frame is ignored: False]
[Protocols in frame: sll:ethertype:ip:udp:pfcp]
[Coloring Rule Name: UDP]
[Coloring Rule String: udp]
Linux cooked capture v1
Packet type: Unicast to us (0)
Link-layer address type: Ethernet (1)
Link-layer address length: 6
Source: 86:11:d4:6b:70:27 (86:11:d4:6b:70:27)
Unused: 97a6
Protocol: IPv4 (0x0800)
Internet Protocol Version 4, Src: 10.1.46.43, Dst: 172.16.0.10
0100 .... = Version: 4
.... 0101 = Header Length: 20 bytes (5)
Differentiated Services Field: 0x00 (DSCP: CS0, ECN: Not-ECT)
0000 00.. = Differentiated Services Codepoint: Default (0)
.... ..00 = Explicit Congestion Notification: Not ECN-Capable Transport (0)
Total Length: 85
Identification: 0x3a23 (14883)
Flags: 0x40, Don't fragment
0... .... = Reserved bit: Not set
.1.. .... = Don't fragment: Set
..0. .... = More fragments: Not set
...0 0000 0000 0000 = Fragment Offset: 0
Time to Live: 64
Protocol: UDP (17)
Header Checksum: 0x1c2f [validation disabled]
[Header checksum status: Unverified]
Source Address: 10.1.46.43
Destination Address: 172.16.0.10
User Datagram Protocol, Src Port: 8805, Dst Port: 20602
Source Port: 8805
Destination Port: 20602
Length: 65
Checksum: 0xe498 [unverified]
[Checksum Status: Unverified]
[Stream index: 881]
[Timestamps]
[Time since first frame: 0.000336481 seconds]
[Time since previous frame: 0.000336481 seconds]
UDP payload (57 bytes)
Packet Forwarding Control Protocol
Flags: 0x20
001. .... = Version: 1
...0 .... = Spare: 0
.... 0... = Spare: 0
.... .0.. = Follow On (FO): False
.... ..0. = Message Priority (MP): False
.... ...0 = SEID (S): False
Message Type: PFCP Association Setup Response (6)
Length: 53
Sequence Number: 4056313
Spare: 0
Node ID : IPv4 address: 10.152.183.35
IE Type: Node ID (60)
IE Length: 5
0000 .... = Spare: 0
.... 0000 = Address Type: IPv4 address (0)
IPv4: 10.152.183.35
Cause : Request accepted(success)
IE Type: Cause (19)
IE Length: 1
Cause: Request accepted(success) (1)
Recovery Time Stamp : Mar 17, 2025 12:16:40.000000000 UTC
IE Type: Recovery Time Stamp (96)
IE Length: 4
Recovery Time Stamp: Mar 17, 2025 12:16:40.000000000 UTC
UP Function Features :
IE Type: UP Function Features (43)
IE Length: 4
0... .... = TREU: Not supported
.0.. .... = HEEU: Not supported
..0. .... = PFDM: Not supported
...1 .... = FTUP: Supported
.... 0... = TRST: Not supported
.... .0.. = DLBD: Not supported
.... ..0. = DDND: Not supported
.... ...0 = BUCP: Not supported
0... .... = EPFAR: Not supported
.0.. .... = PFDE: Not supported
..0. .... = FRRT: Not supported
...0 .... = TRACE: Not supported
.... 0... = QUOAC: Not supported
.... .0.. = UDBC: Not supported
.... ..0. = PDIU: Not supported
.... ...0 = EMPU: Not supported
0... .... = GCOM: Not supported
.0.. .... = BUNDL: Not supported
..0. .... = MTE N4: Not supported
...0 .... = MNOP: Not supported
.... 0... = SSET: Not supported
.... .0.. = UEIP: Not supported
.... ..0. = ADPDP: Not supported
.... ...0 = DPDRA: Not supported
0... .... = MPTCP: Not supported
.0.. .... = TSCU: Not supported
..0. .... = IP6PL: Not supported
...0 .... = IPTV: Not supported
.... 0... = NORP: Not supported
.... .0.. = VTIME: Not supported
.... ..0. = RTTL: Not supported
.... ...0 = MPAS: Not supported
User Plane IP Resource Information (removed in Rel 16.3) : internetAccess
IE Type: User Plane IP Resource Information (removed in Rel 16.3) (116)
IE Length: 15
Flags: 0x61, ASSOSI (Associated Source Instance), ASSONI (Associated Network Instance), V4 (IPv4)
01.. .... = Spare: 1
.1.. .... = ASSOSI (Associated Source Instance): Present
..1. .... = ASSONI (Associated Network Instance): Present
...0 00.. = TEIDRI (TEID Range Indication): 0x0
.... ..0. = V6 (IPv6): Not Present
.... ...1 = V4 (IPv4): Present
...0 00.. = TEID Range Indication: 0
IPv4 address: 172.16.1.10
Network Instance: internet
0000 .... = Spare: 0
.... 0000 = Source Interface: Access (0)
[Response To: 108231]
[Response Time: 0.000336481 seconds]
Kubernetes Events:
bess-upf 5m3s Normal Pulled pod/upf-0 Container image "omecproject/upf-epc-pfcpiface:rel-2.0.1" already present on machine
default 9s Normal Starting node/ip-172-16-0-10 Starting kubelet.
default 9s Normal NodeHasSufficientMemory node/ip-172-16-0-10 Node ip-172-16-0-10 status is now: NodeHasSufficientMemory
default 8s Normal Starting node/ip-172-16-0-10
default 9s Normal NodeAllocatableEnforced node/ip-172-16-0-10 Updated Node Allocatable limit across pods
default 9s Warning InvalidDiskCapacity node/ip-172-16-0-10 invalid capacity 0 on image filesystem
default 9s Normal NodeHasNoDiskPressure node/ip-172-16-0-10 Node ip-172-16-0-10 status is now: NodeHasNoDiskPressure
default 9s Normal NodeHasSufficientPID node/ip-172-16-0-10 Node ip-172-16-0-10 status is now: NodeHasSufficientPID
bess-upf 8s Warning UnexpectedAdmissionError pod/upf-0 Allocate failed due to no healthy devices present; cannot allocate unhealthy devices intel.com/intel_sriov_vfio_access, which is unexpected
bess-upf 7s Warning FailedMount pod/upf-0 MountVolume.SetUp failed for volume "configs" : object "bess-upf"/"upf" not registered
kube-system 7s Normal Pulled pod/calico-node-h8hsk Container image "docker.io/calico/cni:v3.28.1" already present on machine
bess-upf 7s Normal Started pod/init-net-kqztj Started container init-sriov-plugin
bess-upf 7s Normal Created pod/kube-sriov-device-plugin-amd64-7bqw6 Created container: kube-sriovdp
kube-system 7s Normal Created pod/calico-node-h8hsk Created container: install-cni
bess-upf 7s Normal Started pod/kube-sriov-device-plugin-amd64-7bqw6 Started container init-sriov-plugin
bess-upf 7s Normal Created pod/kube-sriov-device-plugin-amd64-7bqw6 Created container: init-sriov-plugin
bess-upf 7s Normal Pulled pod/kube-sriov-device-plugin-amd64-7bqw6 Container image "docker.io/omecproject/omec-cni:1.1.7" already present on machine
kube-system 7s Normal Started pod/calico-node-h8hsk Started container install-cni
kube-system 7s Normal AddedInterface pod/calico-kube-controllers-5947598c79-xg9jn Add eth0 [10.1.46.1/32]
kube-system 7s Normal Pulled pod/calico-kube-controllers-5947598c79-xg9jn Container image "docker.io/calico/kube-controllers:v3.28.1" already present on machine
kube-system 7s Normal Created pod/calico-kube-controllers-5947598c79-xg9jn Created container: calico-kube-controllers
kube-system 7s Normal Started pod/calico-kube-controllers-5947598c79-xg9jn Started container calico-kube-controllers
kube-system 7s Normal Pulled pod/calico-node-h8hsk Container image "docker.io/calico/cni:v3.28.1" already present on machine
kube-system 7s Normal Started pod/kube-sriov-device-plugin-amd64-f5lpj Started container kube-sriovdp
kube-system 7s Normal Started pod/calico-node-h8hsk Started container upgrade-ipam
bess-upf 7s Normal AddedInterface pod/init-net-kqztj Add eth0 [10.1.46.9/32]
bess-upf 7s Normal Pulled pod/kube-sriov-device-plugin-amd64-7bqw6 Container image "docker.io/nfvpe/sriov-device-plugin:v3.3.2" already present on machine
bess-upf 7s Normal Started pod/kube-sriov-device-plugin-amd64-7bqw6 Started container kube-sriovdp
kube-system 7s Normal Created pod/calico-node-h8hsk Created container: upgrade-ipam
kube-system 7s Normal Created pod/kube-sriov-device-plugin-amd64-f5lpj Created container: kube-sriovdp
kube-system 7s Normal Pulled pod/kube-sriov-device-plugin-amd64-f5lpj Container image "nfvpe/sriov-device-plugin:v3.3" already present on machine
kube-system 7s Normal AddedInterface pod/coredns-79b94494c7-j8zd9 Add eth0 [10.1.46.21/32]
kube-system 7s Normal Pulled pod/coredns-79b94494c7-j8zd9 Container image "coredns/coredns:1.10.1" already present on machine
kube-system 7s Normal Created pod/coredns-79b94494c7-j8zd9 Created container: coredns
kube-system 7s Normal Started pod/coredns-79b94494c7-j8zd9 Started container coredns
bess-upf 7s Normal Created pod/init-net-kqztj Created container: init-sriov-plugin
kube-system 7s Normal Pulled pod/kube-multus-ds-amd64-nn8gg Container image "docker.io/nfvpe/multus:stable" already present on machine
kube-system 7s Normal Created pod/kube-multus-ds-amd64-nn8gg Created container: kube-multus
kube-system 7s Normal Started pod/kube-multus-ds-amd64-nn8gg Started container kube-multus
bess-upf 7s Normal Pulled pod/init-net-kqztj Container image "docker.io/omecproject/omec-cni:1.1.4" already present on machine
kube-system 6s Normal Started pod/calico-node-h8hsk Started container calico-node
kube-system 6s Normal Created pod/calico-node-h8hsk Created container: calico-node
kube-system 6s Normal Pulled pod/calico-node-h8hsk Container image "docker.io/calico/node:v3.28.1" already present on machine
Additional Information:
- The UPF remains in a running state if no requests are sent from the OAI SMF.
- The issue occurs specifically when the UPF receives a PFCP Association Request.
- Kubernetes events show
InvalidDiskCapacityand other related errors, which might be contributing to the crash.
Metadata
Metadata
Assignees
Labels
No labels