Mastering Linux eBPF Network Observability

## Introduction

Extended Berkeley Packet Filter (**eBPF**) is a revolutionary technology within the Linux kernel that allows developers to run sandboxed programs directly inside kernel space without modifying kernel source code or loading untrusted kernel modules.

In this deep dive, you will understand the **eBPF verifier**, write real-time syscall tracing scripts using `bpftrace`, and explore how modern networking solutions like **Cilium** replace legacy `iptables` for line-rate packet routing.

---

## Step 1 — Understanding the eBPF Architecture

When an eBPF program is compiled into bytecode via LLVM/Clang, it is loaded into the kernel via the `bpf()` syscall. Before execution, the **eBPF Verifier** performs mathematical proofs to ensure:
- The program contains no unbounded infinite loops.
- All memory pointers are dereferenced safely.
- Kernel memory cannot be corrupted.

---

## Step 2 — Real-Time Syscall Tracing with bpftrace

Let's monitor all `execve` syscalls system-wide to inspect which processes are being launched by users in real-time:

```bash
# Trace all new process executions with PID and command arguments
sudo bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("PID: %d | Comm: %s | Arg: %s\n", pid, comm, str(args->filename)); }'
```

Output sample:
```text
Attaching 1 probe...
PID: 4321 | Comm: bash | Arg: /usr/bin/curl
PID: 4322 | Comm: curl | Arg: https://api.bitcodematrix.com/v1/ping
PID: 4323 | Comm: sshd | Arg: /usr/sbin/sshd
```

---

## Step 3 — Measuring Socket Connection Latency

Let's measure TCP connect latency across your network stack:

```c
// tcp_latency.bt
kprobe:tcp_v4_connect
{
    @start[tid] = nsecs;
}

kretprobe:tcp_v4_connect
/@start[tid]/
{
    $lat_us = (nsecs - @start[tid]) / 1000;
    @connect_latency_us = hist($lat_us);
    delete(@start[tid]);
}
```

Run the probe:

```bash
sudo bpftrace tcp_latency.bt
```

---

## Summary

eBPF transforms the Linux kernel into a programmable runtime. Whether for security audit logging, distributed tracing, or sub-millisecond network routing, eBPF is the bedrock of modern cloud native infrastructure.