← Interview Prep

Linux Kernel — System Calls

The user→kernel boundary: the syscall instruction & mode switch, the syscall number/table & handler, user-pointer validation (copy_from/to_user), negative-errno returns, why syscalls cost more than function calls (KPTI, batching, io_uring), the vDSO, per-arch ABI, and strace/seccomp.

The controlled doorway from user space into the kernel: how a syscall is invoked, why it's costlier than a function call, and how the kernel returns a result safely. Original, interview-focused notes.

A system call is the only way userspace asks the kernel to do a privileged operation (I/O, memory, processes). It's a mode switch (user → kernel ring), not a jump — the CPU raises privilege, the kernel runs the requested service, then returns to user mode.

The mechanism

1. The program (usually via a libc wrapper) puts the syscall number in a register (rax on x86-64) and arguments in registers (rdi, rsi, rdx, r10, r8, r9), then executes the syscall instruction (legacy: int 0x80).
2. The CPU switches to kernel mode, jumps to a fixed entry point, and the kernel uses the number to index the system-call table (sys_call_table) → the handler (sys_read, sys_openat, …).
3. The handler runs on the task's kernel stack, validating every user pointer (copy_from_user/copy_to_user — never trust a userspace address).
4. The return value goes back in rax; on error it's a negative errno, which the libc wrapper turns into -1 + errno.

Why it's expensive (vs a function call)

• A plain call is a jump within your address space. A syscall is a privilege transition: save user state, switch stacks, enter the kernel, validate inputs, and return — orders of magnitude more work.
• Modern CPU mitigations (KPTI, retpolines) added cost by (un)mapping kernel pages and taming speculation on the boundary — which is why syscall-heavy workloads got slower after Meltdown/Spectre.
• Hence batching: readv/writev, sendmmsg, epoll, and especially io_uring (submit many ops, few syscalls) — the modern way to cut boundary crossings.

vDSO — the syscall that isn't

Some "syscalls" are read-only and hot (gettimeofday, clock_gettime). The kernel maps a tiny shared page — the vDSO — into every process so those run entirely in user space, no mode switch. (The old fixed-address vsyscall is its deprecated predecessor.)

Numbers, ABI & tools

• Syscall numbers are an ABI — stable per architecture, and different across arches, so the table is per-arch. New syscalls are added, never renumbered.
• Userspace almost never calls syscall directly — glibc/musl wrappers handle the register setup, errno, and cancellation. syscall(2) exists for the raw path.
• Observe them: strace traces syscalls + args + return; seccomp filters which syscalls a process may make (sandboxing); perf trace for performance.

$ strace -f -e trace=openat,read,write -c ./app
% time     seconds  usecs/call     calls    errors syscall
 61.2    0.004131          4        980           read
 27.4    0.001849          3        612        12 openat
 ...