PCPB-25616: Smart Internal Batching to Improve Signaling Latency

A study guide for understanding the data-plane and pfcp-endpoint repositories in the context of the smart batching feature.

Chapter 1: What Are These Services?

You're looking at two C microservices that together form the User Plane (UP) of a 3GPP Packet Core (PCG/PCC):

Repo sizes at a glance

Chapter 2: Where They Fit in PCG/PCC

Key interfaces

• N4 (PFCP): Between SMF and PEP — session establishment/modification/deletion
• Punter interface: DP → PEP (incoming PFCP from network, forwarded via UDP)
• Relay interface: PEP → DP (outgoing PFCP responses, sent back via UDP)
• Session provisioning: PEP → DP (FlatBuffers over internal protocol to install PDRs/FARs)
• N3/N9: GTP-U tunnels (user traffic in/out of DP)
• N6: Towards data network (internet)

Chapter 3: The PFCP Protocol

PFCP (Packet Forwarding Control Protocol, 3GPP TS 29.244) is the protocol between the Control Plane and User Plane. As a former system tester, you've likely seen PFCP messages in traces. Here's the developer perspective:

Message types PEP handles

Session = collection of rules

• PDR (Packet Detection Rule): Which packets match this rule?
• FAR (Forwarding Action Rule): What to do with matched packets?
• QER (QoS Enforcement Rule): Rate limiting, gating
• URR (Usage Reporting Rule): Counting bytes/packets

Chapter 4: The Feature — Smart Internal Batching (PCPB-25616)

The feature study PowerPoint is at:
/lab/epg_st_sandbox/etahris/PCPB/PCPB-25616/FS/

Problem statement

When the PEP receives a burst of PFCP session messages, it processes them and sends provisioning requests to the DP. Currently, each message may trigger individual internal operations (DB writes, mbox messages, etc.) that could be batched together to reduce overhead and latency.

Why it matters

• Lower signaling latency → faster session setup for subscribers
• Better CPU utilization under load
• Improved throughput for session operations

Where to look in code

Chapter 5: Data-Plane Architecture

The data-plane runs on multiple vCPUs, each assigned specific roles:

Role descriptions

Key concepts

• Fast-path: Once a flow's PDR is determined, it's cached. Subsequent packets skip DPI and go directly to the assigned egress CPU.
• Flow spraying: Distributing packets across egress CPUs based on 5-tuple hash.
• RCU: Read-Copy-Update for lock-free data structure access across CPUs.
• Eventdev: DPDK event device library for passing events between foreground roles.

Source layout (data-plane/)

data-plane/
├── main/           # main.c, dp.c — the application entry point & orchestration
├── upf/            # UPF module: sessions, PDRs, FARs, QERs, URRs, DPI
├── pktio/          # Packet I/O: NIC abstraction, backends (DPDK, Linux, TAP)
├── mbox/           # Mailbox: inter-CPU message passing
├── core-loop/      # Core loop: the run-loop for each CPU role
├── protocol/       # Protocol handlers: GTP, IP/UDP, ARP, BFD, etc.
├── vrf/            # VRF (Virtual Routing & Forwarding)
├── cgnat/          # Carrier-Grade NAT
├── firewall/       # Firewall/ACL
├── itc/            # Internal Traffic Capture
├── up-common/      # Shared libraries (evl, logging, net, tls, etc.)
├── CMakeLists.txt  # Top-level build
├── Makefile        # Developer convenience targets
└── ARCHITECTURE.md # The architecture doc you should read first!

Chapter 6: PFCP-Endpoint Architecture

PEP is a single-threaded event-loop application (with a few helper threads). It's much simpler than the DP in terms of threading.

Key source files (pfcp-endpoint/src/)

Chapter 7: How DP and PEP Communicate

Message flow: CP → UP session establishment

The dp-ctrl header

Messages between DP and PEP are wrapped in a private "dp-control" header containing:

• Source/destination IP and port
• Network instance ID
• Timestamp (when DP received the message from network)

PCG vs EPG steering

• EPG: DP uses partition table to steer to specific PEP instance (same board)
• PCG: DP forwards to PEP's service/cluster IP (Kubernetes). Node messages via TCP, session messages via UDP.

Chapter 8: Threading & Event Loops

PEP threading model

• Main thread (evl): Runs the event loop, processes all business logic
• ext_adapter thread: Receives UDP from DP, puts work into work_manager queues
• Monitor thread: Health checks, metrics

DP threading model

• Each vCPU runs a core loop with its assigned role(s)
• The controller CPU runs an EVL for background tasks
• Foreground CPUs are poll-mode (no sleeping, no EVL)
• Communication between CPUs: mailbox (lock-free queues)

Chapter 9: UPF Module (data-plane)

The upf/ directory is the heart of session handling in the data-plane. It's where PFCP sessions live after PEP provisions them.

Key files

Session message types (from upf_session_queue.h)

UPF_SESSION_MSG_TYPES:
  PFCP                    // Generic PFCP operation
  PFCP_ESTABLISHMENT      // New session
  REPORT                  // Usage report
  PAYLOAD                 // Packet triggered re-evaluation
  INVALIDATE              // Session invalidation
  TERMINATE              // Session deletion
  GEO_ESTABLISHMENT      // Geo-redundancy
  INTERNAL_MODIFICATION  // Internal config change
  ...

Chapter 10: Mailbox (mbox)

The mailbox is the primary mechanism for passing messages between CPUs in the data-plane.

How it works

// mbox/include/mbox/mbox.h
typedef enum mbox_priority {
    MBOX_PRIORITY_CRITICAL,  // Highest
    MBOX_PRIORITY_HIGH,
    MBOX_PRIORITY_MID,
    MBOX_PRIORITY_LOW,       // Lowest
} mbox_priority_t;

typedef struct mbox_msg {
    struct {
        uint32_t u32_1;
        uint32_t u32_2;
        void*    ptr;
        uint64_t u64;
    } data;
} mbox_msg_t;

The mbox uses lock-free MPSC queues (Multiple Producer, Single Consumer) — multiple CPUs can send to one CPU without locks.

Relevance to batching

When the controller provisions a session, it sends mbox messages to egress CPUs. If many sessions are being provisioned simultaneously, batching these mbox messages could reduce overhead (fewer cache-line bounces, fewer wakeups).

Chapter 11: PKTIO & Roles

PKTIO (Packet I/O) is the bottom layer abstracting NIC access. It has a frontend and multiple backends:

The frontend provides hooks for:

• ACL matching
• FIB lookup
• Traffic capture (ITC)
• Traffic steering (partition lookup)
• MTU enforcement

Chapter 12: Session Engine (PEP)

The session engine (pep_session_engine.c, 662K) is the largest file in PEP. It handles:

• PFCP Session Establishment Request → allocate SEID, build session, provision to DP
• PFCP Session Modification Request → update PDRs/FARs/QERs/URRs
• PFCP Session Deletion Request → clean up session
• Session Report Request → send usage reports to CP

Key flow: Session Establishment

// Simplified flow in pep_session_engine.c:
1. Receive PFCP Session Establishment Request (from work_manager)
2. Decode PFCP IEs (PDRs, FARs, QERs, URRs)
3. Allocate UP SEID (Session Endpoint Identifier)
4. Build internal session representation
5. Encode session as FlatBuffer
6. Send to DP via session_client (provisioning)
7. Wait for DP acknowledgment
8. Build PFCP Session Establishment Response
9. Send response via relay interface back through DP to CP

Related files

• pep_internal_session_engine.c (220K) — internal session operations
• pep_session_info.c (59K) — session information management
• session_commands.c (81K) — command handling
• pfcp_resource_encoder.c (65K) — encoding PFCP resources for DP

Chapter 13: Work Manager (PEP)

The work manager is PEP's internal scheduler. It's critical for understanding where batching can be applied.

Priority queues (highest to lowest)

Overload protection

When queues grow (ext_adapter adds faster than main thread processes), the work manager checks max queue time. If exceeded, lower-priority work is dropped. Establishments drop first, then modifications, then deletions.

Chapter 14: Module Lifecycle

Both repos follow a strict module lifecycle pattern (documented in data-plane/MODULE_GUIDELINES.md):

Rules

• Start must not fail — if resources are missing, retry until available
• Start is called once — no support for multiple start calls
• Stop must complete all ongoing work before calling on_stopped_cb
• Stop must be callable before start completes (for interrupted startups)
• Dependencies are injected in create() or start(), never exposed via getters

API pattern

// Every module exposes:
module_t* module_create(dependencies...);
void      module_start(module_t*, on_started_cb, cb_arg);
void      module_stop(module_t*, on_stopped_cb, cb_arg);
void      module_destroy(module_t**);

Chapter 15: Control & Engine Pattern

Components that span both background (controller) and foreground (egress) CPUs follow the Control & Engine pattern:

Communication: Control → Engine

// Via mailbox:
mbox_send(mbox, egress_cpuid, MBOX_PRIORITY_HIGH, &msg);

// The engine polls its mbox in the core loop and processes messages

Examples in the codebase

• upf_session_ctrl.c (control) ↔ upf_session_engine.c (engine)
• upf_ctrl.c (control) ↔ upf_engine.c (engine)
• li_session_ctrl.c (control) ↔ li_session_engine.c (engine)

Chapter 16: RCU in Data-Plane

Read-Copy-Update is used extensively to allow lock-free reads on shared data structures:

// Pattern:
// 1. Reader (egress, hot path):
rcu_read_lock();
element_t* elem = hash_table_lookup(table, key);
// use elem... (pointer valid only within rcu_read_lock/unlock)
rcu_read_unlock();

// 2. Writer (controller):
element_t* old = hash_table_lookup(table, key);
element_t* new = copy_and_modify(old);
hash_table_replace(table, key, new);
call_rcu(old, free_element);  // deferred free after all readers done

Chapter 17: Async Config (EVL Batch Iterator)

From CONFIG_GUIDELINES.md — the DP is moving from synchronous to asynchronous configuration application:

Rules for large-number objects

• Never iterate over large-number objects in a single blocking loop
• Each object handled by its own low-priority deferred job
• Use EVL Batch Iterator with defer_priority = EVL_DEFER_PRIORITY_LOW and batch_size = 1

Two-step config model

// Step 1: Parent builds module-specific config
module_config_t* cfg = build_module_config(raw_config);

// Step 2: Child applies it
module_set_config(module, cfg, on_done, on_done_ctx, on_done_arg);
// All deferred work started inside set_config
// on_done called exactly once when all work complete

Chapter 18: Build System

Tools

• CMake: Primary build system for both repos
• Bob: CI/CD build orchestrator (wraps Docker + CMake)
• Makefile: Developer convenience (data-plane only)

Data-plane Makefile targets

make test        # Build and run tests (fast, clang + sanitizers)
make testsan     # Build with slow sanitizers (ASAN+UBSAN)
make testcov     # Get UT/SFT coverage
make lsp         # Generate compile_commands.json for clangd
make image       # Build Docker image for system test
make lint        # clang-tidy on head commit only
make builds/san  # Sanitizer build
make builds/debug # Debug build (no optimization, good for GDB)

Building pfcp-endpoint standalone

# Using bob:
./bob/bob init-dev
./bob/bob generate:3pp
./bob/bob generate:cmake
./bob/bob build

# Or manually with CMake:
mkdir build && cd build
cmake .. -DPLATFORM=Linux_elc
make -j$(nproc)

Key CMake options

Chapter 19: Testing Layers

PEP SFT architecture

// tests/fixture/pep_sft_fix.c — the test fixture
// Simulates:
//   - Control Plane (sends PFCP messages)
//   - Data Plane (receives provisioning, sends responses)
//   - Config (NWCMA simulator)
//   - LEP (Local Endpoint)
//   - UEIP Allocator

// tests/sft/pep_sft_sessions.c — session test cases
// tests/sft/pep_sft_associations.c — association test cases

Chapter 20: CI/CD Pipeline

Pipeline stages (both repos)

Gerrit workflow

# 1. Create branch
git checkout -b my-feature

# 2. Make changes, commit
git add -A
git commit  # Include Change-Id from commit-msg hook

# 3. Push for review
git push origin HEAD:refs/for/master

# 4. Wait for PreCodeReview (+1/-1)
# 5. Address review comments, amend
git commit --amend
git push origin HEAD:refs/for/master

# 6. Get Code-Review +2, Submit

Chapter 21: The Signaling Latency Problem

Let's trace the full signaling path and identify where latency accumulates:

Where batching helps

When many PFCP messages arrive in a burst (e.g., during mass attach), the current model processes them one-by-one sequentially. Smart batching can:

1. Batch DB operations: Multiple sessions writing to Redis can be pipelined
2. Batch mbox messages: Send one batch notification to egress instead of N individual messages
3. Batch provisioning: Send multiple sessions to DP in one request
4. Reduce context switches: Process a batch before yielding to the event loop

Chapter 22: Batching Concept

Smart batching strategy

Key design questions for the feature

1. Batch trigger: When to flush a batch? (queue depth? timer? both?)
2. Batch size: Maximum messages per batch?
3. Scope: Which operations can be batched together?
4. Error handling: If one message in a batch fails, what happens to others?
5. Ordering: Must messages for the same session be ordered?

Existing batching in the codebase

// pfcp-endpoint/src/pep_options.h:
pep_options_kvdb_request_batch_interval(const pep_options_t* opt);
// Already has a concept of batching DB requests!

// data-plane/upf/src/upf_session_queue.h:
// Per-session queue already serializes operations
// Multiple sessions can be processed in parallel

Chapter 23: Code Reading Plan

Here's your recommended reading order to understand the feature context:

Phase 1: Understand the architecture (Week 1)

Phase 2: Understand the signaling path (Week 2)

Phase 3: Understand batching points (Week 3)

Chapter 24: Day-to-Day Workflow

Setting up your environment

# 1. Generate compile_commands.json for IDE
cd /workspace/git/etahris/data-plane
make lsp

# 2. For pfcp-endpoint:
cd /workspace/git/etahris/pfcp-endpoint
./bob/bob init-dev
./bob/bob generate:3pp
./bob/bob generate:cmake
# compile_commands.json will be in the build dir

Running tests locally

# Data-plane: fast test cycle
cd /workspace/git/etahris/data-plane
make test                    # All tests
make test t=upf_session_SUITE  # Specific suite

# PEP: using bob
cd /workspace/git/etahris/pfcp-endpoint
./bob/bob build
./bob/bob test

Debugging tips

• GDB: Use make builds/debug for unoptimized builds
• ASAN: Use make builds/san to catch memory bugs
• Logs: Both services use structured JSON logging. Use upi_tools/frog_rs or format_json_logs to read them
• Core dumps: Already have some in your pfcp-endpoint dir — use gdb pfcp-endpoint core.X

Commit message format

Short summary (max 50 chars)

Longer description of what and why (not how).
Wrap at 72 characters.

Change-Id: I1234567890abcdef  (auto-generated by hook)

Key contacts

• Team Infinity: Owns pfcp-endpoint (PDLGCTEAMI@pdl.internal.ericsson.com)
• Team Fiji: Owns async config work in data-plane