Processes versus threads, main run loop, event sources, queues, QoS, synchronization, races, deadlocks, reentrancy, priority inversion.
Operating-systems reference
The machinery beneath AppKit.
Study OS concepts through the framework behaviors they explain. Your goal is to trace, diagnose, and design—not reproduce a kernel textbook.
Depth guide
Know four areas deeply.
For this role, depth means you can connect the concept to AppKit behavior, predict a failure mode, and choose an observation tool. You do not need to implement a scheduler or virtual-memory manager.
Virtual memory pages, resident and dirty footprint, allocations, backing stores, invalidation, compositing, display deadlines.
XPC, launchd, sandbox and entitlements, Mach-O, dyld, symbol binding, ABI, framework loading, process failure.
Awareness is enough
Kernel scheduling algorithms, page-replacement math, device drivers, filesystem internals, signals, sockets, and Metal command submission. Go deeper only if the recruiter says the loop includes systems coding or low-level graphics.
Two critical paths
Trace input in and pixels out.
These diagrams are conceptual debugging maps, drawn narrow so they stay readable on a phone. They identify responsibility and process boundaries without claiming every private implementation detail is a public contract.
When input or animation stalls, locate the stage, the thread state, and the resource wait before changing code. The two paths meet on one thread: the main run loop services the input path and issues the commit that ends the render path, so anything that blocks it stops both.
Core concepts
Explain each in one minute.
Use the “because” test: if you cannot connect the concept to a visible behavior or framework design choice, your explanation is still too abstract.
Process and thread
A process owns an address space and OS resources. Threads share that address space but have independent execution and stacks. A crash can terminate the process; a race can corrupt shared state.
Run loop
An event-processing loop attached to a thread. It waits on input/timer sources, dispatches handlers, notifies observers, and sleeps when idle. Blocking the main thread prevents the main run loop from servicing UI work.
GCD, QoS, scheduler
You submit work and express intent; the system maps it to available threads and cores. QoS communicates urgency and energy tradeoffs. Queue order is not a universal thread guarantee.
Race, deadlock, inversion
A race is unsynchronized conflicting access. Deadlock is circular waiting with no progress. Priority inversion is lower-priority work blocking higher-priority work. Each needs different evidence and repair.
Virtual memory
A process sees virtual addresses divided into pages. Resident pages occupy RAM; clean mapped pages can often be discarded and reloaded; dirty pages hold modified data and contribute to footprint, compression, or swap pressure.
Allocation and lifetime
ARC manages object ownership, not total memory efficiency. Temporary objects, caches, images, layer backing, retain cycles, and autorelease lifetime all affect peak and sustained footprint.
Compositing
Your process prepares view/layer content; system services compose windows for the display. Redundant drawing, oversized backing content, excessive transactions, or main-thread stalls can all harm visual responsiveness.
XPC and launchd
XPC sends messages across processes. launchd manages service lifecycle. Isolation contains crashes and privileges, but adds serialization, latency, cancellation, versioning, and connection-failure work.
Sandbox and entitlements
The sandbox restricts access to files, devices, and services. Entitlements grant specific capabilities. Framework design must not assume every client process has identical authority.
Mach-O, dyld, ABI
Mach-O packages executable images and dynamic libraries; dyld loads dependencies and binds symbols. ABI stability protects already-compiled clients from changes in binary-visible behavior and layout.
Visual OS atlas
Open the picture, then teach it.
These are deliberately simplified models. Click each one to reveal the interview explanation and a practical way to prove it on macOS.
01One process, many execution pathsprocess · threads · run loop · QoS
Interview explanation: The process owns the address space; threads share it but execute independently. The main thread is not “the UI” by magic—it is the thread whose run loop services UI work. A synchronous wait from it on lower-priority work creates a responsiveness and QoS problem.
Practical proof: Put a blocking read in a button action, capture a hang sample, then move the read to a worker and dispatch only the state change back to the main thread.
02Virtual memory is not just allocationspages · resident · dirty · pressure
Interview explanation: A large virtual region does not prove equal RAM use. Ask which pages are resident, dirty, shared, compressed, or reclaimable. ARC controls object ownership, while caches, image backing, and dirty pages determine sustained footprint.
Practical proof: Render the same 500-thumbnail trace twice. Compare Allocations, VM Tracker, resident size, dirty memory, and peak versus steady state before bounding the cache.
03In-process UI versus XPC isolationmessages · failure · authority
Interview explanation: An AppKit or SwiftUI bridge normally stays in one process. XPC adds a protocol and a failure boundary: serialization, latency, cancellation, invalidation, versioning, sandbox authority, and helper recovery become part of the API.
Practical proof: Sketch a request/response protocol, terminate the helper mid-request, and define the client’s timeout, cancellation, reconnection, and stale-result behavior.
Chapter · memory
Heap allocation.
Where your objects live, what it costs to put them there, and why “I freed it” and “the memory came back” are two different claims. This chapter works upward: what the stack and the heap actually are, what happens inside malloc, how the heap relates to the virtual-memory system the kernel actually accounts for, what ARC does and does not promise, and which instrument answers which question. It ends with two runnable broken programs and twelve interview questions.
Which allocation is growing, whether it is useful, abandoned or leaked, and what the user pays for it.
Name the instrument, the lifespan filter, and the measurement that would falsify your hypothesis.
Allocator internals below the size-class ladder, page-replacement policy, compressor implementation, GPU and IOSurface accounting.
The sentence to have ready
“free() is a promise about reuse, not about residency.” Almost every confusing memory measurement on macOS — a leak fix that does not lower the footprint, an evicted cache that returns nothing, a process that stays large after the work is done — resolves once you separate what the allocator owns from what the kernel accounts for.
1 · Stack and heap
Two regions, two lifetimes, two failure modes.
Both are just memory in the same address space. The difference is who decides when the memory stops being yours, and that single difference produces every bug in this chapter.
Simplified diagram, not a screenshot and not a memory map of any real process. It exists to make one point: “using memory” splits into reserving address space and touching pages, and only the second one is charged to you.
The stack, in one sentence
A per-thread region whose lifetime discipline is the call itself: entering a function reserves space, returning releases it, and nothing has to be remembered. It is fast because “when does this die?” has a compile-time answer. It is bounded — the default for a secondary thread is far smaller than the main thread’s — and overflowing it is a crash, not an ENOMEM.
The heap, in one sentence
Apple’s framing: “Malloc lets your app dynamically allocate long-lived memory. Allocations stay alive until they’re explicitly freed meaning they can live past the scope of the code that created them.”
documented WWDC24 10173 · Analyze heap memory, 3:07–3:13. https://developer.apple.com/videos/play/wwdc2024/10173/
Which one holds your Swift value
A class instance is a heap allocation, always. A struct or enum is stored inline wherever it lives — on the stack as a local, inside its parent’s allocation as a property, inside the array’s buffer as an element. But a struct containing a String, Array, Dictionary, or any class reference points at heap storage, so “I used a struct” is not by itself an allocation argument.
The three kinds of heap memory
Apple names them precisely, at 20:26–20:51: useful — reachable and will be used again; abandoned — reachable, “could be used but won’t actually ever be used again … counts against your app’s footprint and is just wasted”; and leaked — “unreachable memory that can’t ever be used again”. Different diagnoses, different tools, different repairs.
documented WWDC24 10173. https://developer.apple.com/videos/play/wwdc2024/10173/
The vocabulary interviews actually test
Allocation — one block handed back by
malloc. Has a requested size and an actual size, and they are usually different.Region — a chunk of address space the allocator obtained from the kernel and subdivides. A region is returned only when every allocation in it is free.
Page — the kernel’s unit of residency. 16 KiB on Apple silicon; check with
sysctl -n hw.pagesize.Resident — pages currently in RAM. Dirty — written to, so not discardable. Clean — discardable and re-faultable.
Footprint —
phys_footprint, the number the OS uses for memory limits. Not virtual size, not resident size.Leak vs abandoned — unreachable vs reachable-but-dead.
leaks(1) finds the first; only you can find the second.2 · Inside malloc
The path from malloc(1) to a page.
You ask for a number of bytes. What you get back is a block from a size class, carved out of a region the allocator already owns, or — if it owns nothing suitable — out of a fresh mapping it asks the kernel for. Knowing the shape of that path is what lets you predict the overhead of a data structure before you measure it.
Rounding is documented, and it starts at 16
Apple, on the allocator’s rules: “their minimum allocation size and alignment is 16 bytes, which means that if you ask for 4 bytes, your request gets rounded up to 16.”
documented WWDC24 10173, 3:19–3:23. https://developer.apple.com/videos/play/wwdc2024/10173/
Freed memory is zeroed, since macOS 13
malloc(3): “Starting in macOS 13, iOS 16.1 and aligned releases, free(3) fully zeroes many blocks immediately. This may expose some previously-silent bugs … read-after-free bugs may now observe zeroes instead of the previous content.” Apple describes it in session as a security feature. Version-sensitive: code that “worked” on an older OS by reading freed memory changes behaviour here.
“Many” is load-bearing, and the boundary is measurable. Filling a block with 0x77, freeing it and reading byte 64 back — one size per fresh process, so allocator reuse cannot confuse the answer — gives a clean split at malloc_size 8,192: every block at or below it reads back zero, every block from 10,240 upward still reads back 0x77. A read-after-free of a decoded image buffer therefore still silently “works”, which is exactly where the bug does the most damage.
documented man 3 malloc on macOS 26.3. measured here The 8,192-byte boundary. negative result MallocScribble is only observable above that boundary — below it the zeroing has already happened, so “I turned on MallocScribble and saw nothing” is not evidence of a clean program.
Zones are a real, public API
malloc_create_zone, malloc_default_zone, malloc_zone_from_ptr, malloc_destroy_zone. A zone is an independent allocator: destroying one “deallocates all memory associated with objects in zone as well as zone itself”, which makes whole-generation teardown a single call.
documented man 3 malloc_zone_malloc. measured here A zone created and named through this API is enumerated correctly by malloc_get_all_zones and disappears on destroy.
malloc_size is the honest number
It reports what the allocation actually occupies, not what you asked for. Use it whenever you are reasoning about the overhead of many small objects; the difference is often larger than the payload.
documented man 3 malloc_size. https://developer.apple.com/documentation/xcode/gathering-information-about-memory-use
malloc_good_size answers without allocating
malloc_size needs a pointer, so asking it costs an allocation and perturbs the heap you are measuring. malloc_good_size(n) answers the same question from the size alone. Measured here it reproduces the ladder exactly — 1 → 16, 100 → 112, 129 → 160, 200 → 224, 1,000 → 1,024, 20,000 → 20,480, 40,000 → 49,152 — which makes it the right tool for sizing a struct against the next class boundary at design time, before any of it exists.
documented man 3 malloc_size. measured here
2AThe size-class ladder, measured on this machineevery boundary from 1 byte to 64 KiB
A small program that calls malloc(n) for every n from 1 to 70,000 and prints each point where malloc_size() changes. The result is a clean banded ladder: eight classes per band, and the quantum doubles at every band boundary.
band quantum classes worst-case waste
1 – 128 16 B 16 32 48 64 80 96 112 128 15 B on a 1-byte request
129 – 256 32 B 160 192 224 256 31 B (129 -> 160 = 24%)
257 – 512 64 B 320 384 448 512 63 B
513 – 1024 128 B 640 768 896 1024 127 B
1025 – 2048 256 B 1280 1536 1792 2048 255 B
2049 – 4096 512 B 2560 3072 3584 4096 511 B
4097 – 8192 1 KiB 5120 6144 7168 8192
8193 – 16384 2 KiB 10240 12288 14336 16384
16385 – 32768 4 KiB 20480 24576 28672 32768
32769 – 65536 16 KiB 49152 65536 page-granular from hereWhat to do with this. The relative overhead is worst just above a band boundary. A 129-byte object occupies 160 bytes — 24% overhead before a single byte of fragmentation. If you are allocating millions of one thing, measuring malloc_size once and nudging the layout under the next class boundary is a real and cheap win. Below 16 bytes there is nothing to win: everything rounds to 16.
Above 32 KiB the quantum becomes the page size, which is the practical threshold at which an allocation starts behaving like its own mapping. That is the same threshold exercise 02 turns on.
measured here Apple M4 Pro, macOS 26.3 (25D125), arm64, 16 KiB pages. inference The banded shape is a general property of size-class allocators; the exact boundaries are an implementation detail of this OS build and must be re-measured, not quoted, on any other.
NEGATIVE RESULT · THE ZONE API CANNOT SEE THE NANO PATH — BUT SOMETHING ELSE CAN. A great deal of writing about macOS allocation describes small allocations being served by a separate MallocNanoZone visible through the zone API. Through that API, on macOS 26.3 on arm64, it is not. malloc_get_all_zones() returns exactly one zone, named DefaultMallocZone; malloc_zone_from_ptr() reports that same zone for allocations of 1 byte, 4 KiB and 1 MiB alike; and setting MallocNanoZone=0 changes none of the reported sizes or zone names. That is not a broken measurement — a zone created with malloc_create_zone appears in the very same enumeration and disappears on destroy.
The conclusion “the variable does nothing” is the part that is wrong, and it is wrong because the probe was pointed at the wrong instrument. vmmap reports the division directly: the one zone is internally carved into region families — MALLOC_NANO, MALLOC_TINY, MALLOC_SMALL, MALLOC_LARGE, plus metadata and guard pages — and MallocNanoZone=0 removes the MALLOC_NANO metadata region from the map while leaving every size and zone name identical. Say “one zone, several region families; the zone API cannot see the division and vmmap names it”. measured here version-sensitive
Simplified diagram, not a memory map of a real process. Boundaries are an implementation detail of macOS 26.3 on arm64 — reproduce them, do not quote them.
| What you ask the allocator | Default | MallocNanoZone=0 | MallocNanoZone=1 |
|---|---|---|---|
malloc_get_all_zones() | 1 — DefaultMallocZone | 1 — DefaultMallocZone | 1 — DefaultMallocZone |
malloc_size for 1 B / 4 KiB / 1 MiB | 16 / 4096 / 1048576 | 16 / 4096 / 1048576 | 16 / 4096 / 1048576 |
vmmap → MALLOC_NANO metadata region | present | absent | present |
measured here Three runs of one probe under three environments, each reading the zone API in-process and vmmap from outside. The bottom row is the whole point: a probe written only against malloc_size and malloc_get_all_zones concludes, wrongly, that the variable does nothing. inference The zone API is an allocator-level view; region families are a virtual-memory-level view, and nothing requires the two to agree.
vmmap --summary also prints the fragmentation figure this chapter otherwise has to infer
MALLOC ZONE SIZE ALLOCATED FRAG SIZE % FRAG
DefaultMallocZone_0x100484000 20.5M 9327K 161K 2%Section 5 defines fragmentation as free bytes the allocator cannot hand back. This line is that number, per zone, for a process you did not write, without adding any code to it. measured here Absolute values are one process at one instant — reproduce; do not quote.
MEASUREMENT HAZARD · THE INSTRUMENT CHANGES THE SIZES. With MallocStackLogging enabled, malloc_size() reports different numbers, because the instrumented allocator uses a different layout. measured here a malloc(4096) reports 4096 normally and 5104 under stack logging; malloc(100) reports 112 normally and 103 under stack logging — smaller, not larger. leaks(1) correspondingly reports a 4 KiB leak as 5.00K. Never quote a size-class or overhead figure that was gathered with stack logging on, and when you report a leak’s size say which mode produced it. negative result
And here is why it happens, which the number alone does not tell you. heap -zones reports one zone in an ordinary process and three under MallocStackLogging=lite — DefaultMallocZone, MallocStackLoggingLiteZone and MallocStackLoggingLiteZone_Wrapper. The allocations move to a different allocator with a different block layout. That also bounds the claim above it: “exactly one zone” is true only without stack logging. measured here
Probe it yourself · C
#include <stdio.h>
#include <stdlib.h>
#include <malloc/malloc.h>
int main(void) {
size_t prev = 0;
for (size_t s = 1; s <= 70000; s++) {
void *p = malloc(s);
size_t got = malloc_size(p);
if (got != prev) {
printf("first request needing %8zu bytes: %8zu\n", got, s);
prev = got;
}
free(p);
}
return 0;
}Probe it yourself · Swift
import Darwin
// What a Swift class instance actually occupies.
final class Node { var a = 0; var b = 0; var next: Node? }
let n = Node()
let raw = Unmanaged.passUnretained(n).toOpaque()
print("class_getInstanceSize-ish:", malloc_size(raw))
// And what an Array's buffer costs for N elements.
var xs = [Int](); xs.reserveCapacity(1000)
xs.withUnsafeBufferPointer { buf in
print("1000 Ints requested:", 1000 * MemoryLayout<Int>.stride)
print("buffer base:", buf.baseAddress != nil)
}The Swift form is deliberately less precise: a class instance’s header and the exact buffer layout are runtime details, and malloc_size on the object pointer is the honest way to ask rather than computing it from field sizes. inference
3 · Heap and virtual memory
Allocating is free. Touching is what costs.
The allocator hands out bytes. The kernel accounts for pages. Almost every confusing memory number on macOS comes from measuring one and reasoning about the other.
The rule, in Apple’s words
“Memory that the app allocates at runtime doesn’t initially contribute to this metric. Such memory is ‘clean,’ and iOS doesn’t need to dedicate physical RAM to store it. When the app writes to the allocated memory, it becomes ‘dirty,’ and iOS dedicates RAM to storing its content … Dirty memory contributes to the memory-use metric.”
documented https://developer.apple.com/documentation/xcode/reducing-your-app-s-memory-use inference That page is written for iOS; the clean/dirty mechanism and the 16 KiB page are the same on macOS on Apple silicon, and the measurements below confirm the behaviour directly on macOS 26.3.
One byte can cost a page
Apple again: “iOS measures memory use as the number of memory pages in use multiplied by page size, which is typically 16 KB. Writing a single byte to allocated memory can increase memory use by 16 KB if iOS must allocate a new page to store that byte.” This is the real cost model for sparse data structures.
documented same page.
Three page states, two that count
“Clean pages are memory that hasn’t been written to … pretty cheap as the system can discard and fault these pages again at any time. Dirty pages are memory that’s been written to recently … If there’s memory pressure, the system can swap them, either compressing or writing them to disk. Of these three, only dirty and swapped count towards an application’s memory footprint.”
documented WWDC24 10173, 2:10–2:42. https://developer.apple.com/videos/play/wwdc2024/10173/
Which number to quote
phys_footprint from task_info(TASK_VM_INFO) is what the system uses for limits. virtual_size is meaningless on its own — measured here a trivial C program that has allocated nothing reports 425 GB of virtual size at startup. Never open an answer with virtual size.
3AFive operations, and what each one did to the footprintmeasured · 512 MB · macOS 26.3
virtual resident footprint
baseline 425097 MB 1.3 MB 0.9 MB
after mmap 512MB (untouched) 425609 MB 1.3 MB 0.9 MB <- +512MB virtual, +0 charged
after touching all pages 425609 MB 513.3 MB 513.2 MB <- the write is the cost
after MADV_FREE 425609 MB 513.3 MB 513.2 MB <- no immediate drop
after re-dirtying 425609 MB 513.3 MB 513.2 MB
after munmap 425097 MB 1.3 MB 0.9 MB <- unmapping DOES return it
after malloc(512MB) 425609 MB 1.3 MB 1.0 MB
after touching it 425609 MB 513.3 MB 513.2 MB
after free() 425609 MB 513.3 MB 513.2 MB <- free() returned NOTHINGRead the last line again. A 512 MB block was freed and the process footprint did not move. That is not a bug and not a leak: free() returned the bytes to the allocator’s free lists, which is exactly what it promises. Nothing in the C standard, and nothing in Apple’s documentation, says free returns pages to the kernel.
And the MADV_FREE line. madvise(MADV_FREE) marks pages reclaimable, and the footprint still did not move. It is a hint to the kernel that the contents are expendable, redeemable under pressure — not an instruction to charge you less right now. negative result This experiment cannot show the reclaim actually happening, because inducing genuine system-wide memory pressure is not something a fixture should do.
What did work: munmap. Unmapping returned every page immediately and unconditionally. That is the mechanism, and the practical rule that follows is in the next section. The I/O chapter looks at the same call from the other side — I/O · 3 · read vs mmap is about what a file-backed mapping buys and charges, and its clean pages are exactly the ones this chapter says are not charged to you.
And so did a second pair, which is the correction this section needed. The conclusion “you cannot make a footprint graph go down on demand” is true of MADV_FREE and false of Darwin. Re-running the same probe with two more advice values:
after touching all pages footprint = 513.2 MB
madvise(MADV_FREE) rc=0 footprint = 513.2 MB <- unchanged, as above
madvise(MADV_FREE_REUSABLE) rc=0 footprint = 1.2 MB <- returned, immediately
madvise(MADV_FREE_REUSE) rc=0 footprint = 513.2 MB <- re-charged, without re-touching
after munmap footprint = 1.0 MBMADV_FREE_REUSABLE tells the kernel to stop charging you for these pages; MADV_FREE_REUSE takes the charge back when you start using them again. That pair is the mechanism underneath purgeable memory and evictable caches, and it is the answer to “my cache is evicted and the graph has not moved” for anything you obtained with mmap. It also sharpens section 5’s list of repairs that do not work: malloc_zone_pressure_relief returning 0 is a real dead end for heap blocks, but for a mapping this pair works completely.
measured here Values 7 and 8. documented Declared in $(xcrun --show-sdk-path)/usr/include/sys/mman.h as “pages can be reused (by anyone)” and “caller wants to reuse those pages”. negative result Neither appears in man 2 madvise, which documents only MADV_NORMAL, MADV_RANDOM, MADV_SEQUENTIAL, MADV_WILLNEED, MADV_DONTNEED, MADV_FREE, MADV_ZERO_WIRED_PAGES and MADV_ZERO. version-sensitive Treat an SDK-header-only facility as coupled to the OS version, not as a portable contract, and re-measure on any other build.
What this still does not show. Both results are accounting changes observed immediately. Nothing here observes the kernel actually reclaiming a page under genuine system-wide memory pressure, because inducing that is not something a fixture should do.
measured here Apple M4 Pro, macOS 26.3 (25D125), 48 GB, 16 KiB pages. Reproduce with task_info(mach_task_self(), TASK_VM_INFO, …); the full probe is in the exercise bundle’s exercise 02.
When to reach past malloc
An allocation that is a whole number of pages, independently lifetimed, and few enough that two system calls each is affordable belongs in its own mapping: mmap(MAP_PRIVATE|MAP_ANON) and munmap, or NSData/Data backed by a mapped file. Decoded images, tile caches, audio buffers and scratch render targets usually qualify. Everything smaller or more numerous belongs on the heap, where the allocator’s whole value is amortising those system calls away.
The mirror image for files: Data(contentsOf:options:.mappedIfSafe) maps instead of copying, so the pages stay clean and are reclaimable under pressure — the single highest-leverage change for an app that loads large read-only assets. https://developer.apple.com/documentation/foundation/data/readingoptions
4 · ARC and lifetimes
Ownership the compiler inserts, and the graph it cannot see.
ARC is not a garbage collector, and the distinction is the whole of this section. ARC maintains a count; it never traces reachability, so it can never discover that two objects are keeping each other alive while nothing else can reach either of them.
What ARC actually does
The compiler inserts retain and release calls at the points where ownership begins and ends. When a strong count reaches zero the object is deinitialised and its allocation freed. There is no scan, no pause, and no collector thread — and correspondingly no way to notice an unreachable cycle.
Closures are heap allocations too
Apple: “When Swift closures need to capture values, they allocate memory on the heap to store the captures. The Memory Graph Debugger labels these allocations as closure contexts. Each closure context in your app’s heap corresponds 1:1 with a live closure. Closures capture references strongly by default, making it possible to create reference cycles.”
documented WWDC24 10173, 21:54–22:06. https://developer.apple.com/videos/play/wwdc2024/10173/
weak versus unowned
Both break a cycle. weak becomes nil and costs a side-table entry; unowned is a promise the referent outlives the reference and traps if it does not. Choose by asking whether the referent can legitimately die first. If it can, weak. If you are not sure, weak — a returned nil is a branch, a broken unowned is a crash.
Autorelease pools cause transient growth
Apple: “Swift can produce autoreleased objects when it calls into frameworks that use or expose Objective-C APIs … Threads usually have a top-level autorelease pool, but it’s not cleaned very often. This can matter a lot when code fills up the pool with objects, which easily happens in loops.” The fix is “a nested, local autorelease pool scope”.
documented WWDC24 10173.
The shape that leaks
final class Thumbnailer {
var pixels: [UInt8] = []
var onComplete: (() -> Void)? // stored ON self
func begin() {
onComplete = { // captures self STRONGLY
use(self.pixels)
}
}
}
// Thumbnailer -> onComplete -> Thumbnailer
// Nothing else references either one. deinit never runs.The repair, and the reasoning
func begin() {
onComplete = { [weak self] in
guard let self else { return } // gone? nothing to complete.
use(self.pixels)
}
}The rule is ownership, not convenience: the owner holds strongly, the owned refers back weakly. A completion handler does not own the object it completes for, so that is the edge to weaken. Saying which edge and why is the difference between a memorised answer and an understood one.
The measurement trap this chapter exists to prevent
measured here The two programs above, run once over 4,000 objects, report the same process footprint — 322,848 KB broken against 323,200 KB fixed. One of them leaked every single object and the footprint could not tell you. Run the same browse eight times and the difference is total: the broken build climbs to 2,574,594 KB while the fixed build flattens at 423,488 KB.
So: measure a repeated operation, not a single one, and assert that growth is flat, not that it is zero — the fixed build still grew about 100 MB over eight rounds as the allocator settled at its high-water mark. An assertion of zero would fail on correct code, which is worse than no assertion.
Abandoned memory is the harder bug
A cache with no eviction policy, a singleton that accumulates, a dictionary keyed by something that is never equal twice. leaks(1) will report nothing, because every byte is reachable. Only generations — or knowing what your program is supposed to hold — find these.
The bug Apple demonstrates is an abandoned-memory bug
In WWDC24 10173 the persistent growth turns out to be a thumbnail cache keyed on “the current time” instead of the file’s creation date, so “we’ll never find anything in the cache, and we’ll always cache a new PhotoThumbnail each time”. No cycle, no leak, unbounded growth.
documented
Use a cache that can be evicted
NSCache “incorporates various auto-eviction policies, which ensure that a cache doesn’t use too much of the system’s memory”, and is safe to use from several threads “without having to lock the cache yourself”. A bare Dictionary has neither property.
documented https://developer.apple.com/documentation/foundation/nscache
Say the bound out loud
Every cache needs an answer to “what stops this growing?” — a count limit, a cost limit, an eviction policy, or a scope that ends. “It is only a cache” is not an answer; abandoned memory is, in Apple’s words, memory that “counts against your app’s footprint and is just wasted”.
5 · Fragmentation & churn
Free bytes that are not a free page.
Two distinct costs get confused with each other. Fragmentation is a property of the layout of live objects: bytes are free but no region is. Churn is a property of the rate: allocation and free traffic that costs CPU and produces transient spikes. They have different symptoms and different repairs.
5AEvicting half a cache and getting nothing backmeasured · 20,000 × 32 KiB
from malloc from mmap
20,000 tiles allocated 628.9 MB 626.4 MB
evict every OTHER tile 628.9 MB 313.9 MB <- 312 MB freed, 0 returned
malloc_zone_pressure_relief 628.9 MB 313.9 MB <- returned 0 bytes
free the remaining tiles 246.7 MB 1.1 MB
recovered fraction at half 0.000 0.500The mechanism, drawn. The allocator serves 32 KiB requests from larger regions. A region returns to the kernel only when every allocation inside it is free. Evicting alternate tiles leaves every region with live tenants:
region [ T0 | T1 | T2 | T3 | T4 | T5 | T6 | T7 ] before eviction
region [ .. | T1 | .. | T3 | .. | T5 | .. | T7 ] after evicting the evens
^ 50% free, 0% returnablemalloc_zone_pressure_relief returned 0 bytes and moved nothing. It is the documented way to ask the allocator to give back what it can, and here the honest answer is “nothing” — which is the measurement that rules out “the allocator is just being lazy”. measured here
Even after every tile was freed, the heap version still held 246.7 MB. The mapped version dropped to 1.1 MB.
Apple names this cost too. On transient spikes: “The long-term effect of memory spikes is also bad, as it causes fragmentation or holes in heap memory regions.” documented WWDC24 10173.
Fragmentation: the four repairs
1 Give page-multiple, independently-lifetimed allocations their own mapping. 2 Group objects with the same lifetime into an arena or a zone you destroy whole. 3 Make the sizes uniform so freed blocks are reusable by the next request. 4 Reduce the peak, so there is less to fragment. Not on the list: calling something that asks the allocator nicely.
Churn: the symptom is CPU, not footprint
Per-element allocation in a hot loop shows up as time in malloc/free and as a sawtooth in Allocations, not as a rising baseline. The repairs are different too: reserveCapacity, reuse pools, value types that stay inline, and hoisting allocations out of the loop.
Transient spikes have three costs
Apple lists them: they “cause memory pressure, and the system reacts” with swapping, compression and background-task termination; in the worst case your own app is terminated; and they fragment the heap. A spike that “comes back down” is not free.
documented WWDC24 10173.
The autorelease-pool loop
The classic transient spike in Swift code that touches Objective-C APIs. Wrap the loop body in autoreleasepool { } so objects drain per iteration instead of accumulating until the thread’s top-level pool is cleaned. Visible in Allocations as @autoreleasepool content nodes.
documented https://developer.apple.com/documentation/foundation/nsautoreleasepool
6 · Diagnosis
Choose the instrument from the question.
There are four distinct memory questions and they do not share an answer. Naming which one you are asking, before naming a tool, is most of what separates a senior answer here.
| The question | First move | What it shows | What it cannot tell you |
|---|---|---|---|
| Is memory growing at all? | Xcode memory report, or footprint/task_info over a repeated scenario | The shape: flat, sawtooth, staircase, or ramp | Apple: “it can’t tell you why memory use is growing” |
| What is spiking and then going away? | Allocations, lifespan filter Created & Destroyed over the spike range, call-tree view | The code allocating the most transient memory | Anything about what persists |
| What is growing and staying? | Allocations, Mark Generation before and after the operation — or, with no Xcode at all, two leaks --outputGraph snapshots and heap --diffFrom (section 6A) | Allocations that survived that interval, by type | Why they survived — that needs the graph |
| Why is this specific object still alive? | Memory Graph Debugger (or an exported .memgraph), filter by address — or leaks <graph> --trace=<addr> (section 6A) | The chain of references holding it, with allocation backtraces | Whether keeping it was intended |
| Is anything unreachable? | Leaks instrument, or leaks <pid> | Cycles and lost pointers, with the allocating stack | Abandoned memory — every byte of which is reachable |
| Where did the address space go? | vmmap <pid>, or VM Tracker in the Allocations template | Regions, and their dirty/clean/swapped split | Individual objects |
Turn on stack logging first, or you get addresses instead of answers
MallocStackLogging records a call stack for every allocation. Without it, leaks and the Memory Graph Debugger can show you that something is alive but not where it came from. In Xcode it is the Malloc Stack checkbox in the scheme’s Diagnostics tab; from a terminal it is an environment variable. man 3 malloc documents the modes: lite records “stack traces for current allocations only, without history … recorded to in-memory data structures” and full records “allocation and deallocation events to an on-disk log”. Prefer lite unless you need history.
And remember it perturbs the measurement — see the hazard box in section 2. Sizes reported under stack logging are not production sizes.
Command line · no GUI, no elevation
# unreachable memory, with allocating stacks
MallocStackLogging=lite ./myapp &
leaks $!
# what is on the heap, by class and size
heap $!
# regions, and the dirty/clean split
vmmap $! | head -40
vmmap --summary $!
# every malloc/free event for one address
MallocStackLogging=full ./myapp
malloc_history <pid> <address>measured here A leaks run against a fixture that drops 64 blocks reports them grouped as STACK OF 64 INSTANCES OF 'ROOT LEAK: <malloc in leak_one>' with the full allocating stack and 64 leaks for 327680 total leaked bytes. Apple confirms the same set: “Leaks, heap, vmmap, and malloc_history can analyze macOS and Simulator processes directly, or investigate issues using already-captured memory graphs.”
In-process · when you want an assertion
import Darwin
func footprintBytes() -> UInt64 {
var info = task_vm_info_data_t()
var count = mach_msg_type_number_t(
MemoryLayout<task_vm_info_data_t>.size
/ MemoryLayout<natural_t>.size)
let kr = withUnsafeMutablePointer(to: &info) {
$0.withMemoryRebound(to: integer_t.self,
capacity: Int(count)) {
task_info(mach_task_self_,
task_flavor_t(TASK_VM_INFO), $0, &count)
}
}
return kr == KERN_SUCCESS ? UInt64(info.phys_footprint) : 0
}Put this behind a debug flag and assert that a repeated operation returns to its baseline. A regression test that runs the operation ten times and bounds the growth catches abandoned memory that no instrument will ever flag, because nothing about it is unreachable.
Where you can profile heap memory
Apple’s guidance is unusually permissive here and worth knowing, because it contradicts the general rule: “With most profiling, it’s important to run a release build on a real device for accurate timing. For heap analysis, though, the Simulator environment is a lot closer in behavior, and it’s fine to use for memory profiling.” documented WWDC24 10173. For a Mac app the question does not arise — you are already on the target architecture.
What this chapter did not do. No Instruments GUI session was opened and no screenshot appears anywhere on this page. What is still quoted rather than observed is the user interface: the Allocations lifespan filter, the Generations view, VM Tracker, and the Memory Graph Debugger’s canvas. The underlying questions those views answer were exercised on the command line instead — section 6A shows the whole chain running against a real growing process — so “no Xcode” is no longer a reason to have no answer. Still true: no memory reclaim under genuine system-wide pressure was ever observed, and every absolute figure is one machine and one OS build.
6A · The workflow without a window
Generations and the reference chain, from a terminal.
The two hardest rows of the table above — what grew and stayed, and why is this one object still alive — are usually answered by pointing at Xcode. Both have complete command-line answers that ship with macOS, need no elevation and no GUI, and work against a process you did not build. This is that chain, run end to end against a program that abandons eight thousand objects.
Simplified diagram, not a screenshot. Steps 3 and 4 are what the Generations view does; step 6 is what the Memory Graph Debugger draws.
# 0. stacks, or you get addresses instead of answers
MallocStackLogging=lite ./growth &
PID=$!
# 1. is any of it unreachable? -> rules out an entire category
leaks $PID
# Process 23488: 0 leaks for 0 total leaked bytes.
# 2. two snapshots bracketing the operation == Instruments "Mark Generation"
leaks $PID --outputGraph=A.memgraph # before
leaks $PID --outputGraph=B.memgraph # after
# 3. what survived, by type
heap B.memgraph --diffFrom=A.memgraph -s -H
# COUNT BYTES AVG CLASS_NAME BINARY
# 8000 156.2M 20.0K Swift._ContiguousArrayStorage<Swift.UInt8> libswiftCore.dylib
# 1 400K 400.0K Swift._DictionaryStorage<Swift.String, Tile> libswiftCore.dylib
# 8000 375K 48.0 Tile growth
# 4. pick one surviving instance
heap B.memgraph --addresses=Tile --noContent
# 0xc210580c0: Tile (48 bytes)
# 5. WHY is it alive? == the Memory Graph Debugger's reference chain
leaks B.memgraph --trace=0xc210580c0
# Found 1 root referencing: <Tile 0xc210580c0> [48]
# VM: Stack 0x16ee40000-0x16f63c000 rw-/rwx thread 0
# specialized static Main.main() + 24 --> <Swift._DictionaryStorage<Swift.String, Tile>> [212992]
# +191680: --> <Tile 0xc210580c0> [48]measured here Every line above is real output from one run. Step 5 is the one that matters: it names the chain from a root — here the main frame holding the dictionary — down to the object, which is exactly the picture the Memory Graph Debugger draws, obtained headlessly. inference The equivalence to Generations is behavioural, not Apple’s claim: both compare two points in time and report what survived.
THE MISTAKE THAT WASTES THE FIRST ATTEMPT. Both snapshots must actually bracket the operation. Taking A after the growth has already happened gives No new objects detected between memgraphs. — which reads exactly like a clean result and is not one. measured here This happened on the first run of the chain above; the fix was to start the program idle, snapshot, then trigger the work. If a generation diff comes back empty, suspect your timing before you believe the program.
--forkCorpse snapshots without stopping
leaks, heap and footprint all accept it. The kernel forks a corpse of the target and the tool reads that, so a long analysis does not hold the live process still. Useful when the process is serving something while you look at it.
documented each tool’s own usage output.
leaks --atExit is the CI shape
leaks --atExit -- ./yourtool args runs the command and checks at exit, with a non-zero status when it finds something. That turns “did this change leak” into a build step rather than a ritual someone has to remember.
documented leaks --help.
footprint(1) is the zero-code answer
“How big is this process?” without writing any task_info code: footprint -p <name|pid>, with --json <file>, --all, --minFootprint <MiB> and --forkCorpse. It reports the same phys_footprint the limits are enforced against.
measured here from its own usage output.
What still needs the GUI
The lifespan filter (Created & Destroyed over a dragged range), the Generations view, VM Tracker’s region timeline, and the Memory Graph canvas. The command line answers the same questions; it does not give you the same exploration.
inference
6BThe malloc development aids, and what each one missesenvironment variables and Guard Malloc
man 3 malloc documents seventeen environment variables. Three earn their place in an interview answer, and each has a limitation worth stating in the same breath — an answer that names the tool without its blind spot is the weaker answer.
| Variable | Documented effect | The limitation to say out loud |
|---|---|---|
MallocStackLogging=lite|full | lite records stacks for current allocations in memory; full logs allocation and deallocation events to disk | It moves allocations into MallocStackLoggingLiteZone, so every size figure gathered under it is wrong measured here |
MallocScribble | Fill allocated memory with 0xaa and freed memory with 0x55, so stale reads are obvious | Only observable above the zero-on-free boundary — below malloc_size 8,192 the zeroing has already happened measured here |
MallocGuardEdges | “Add a guard page before and after each large block” | Large blocks only. It will not catch a 32-byte overrun documented |
And the answer to “heap corruption I cannot localise”: Guard Malloc, which “uses the virtual memory system to identify memory access bugs … Each malloc allocation is placed on its own virtual memory page”. A 32-byte block written at index 40 survives and exits 0 under the ordinary allocator; under DYLD_INSERT_LIBRARIES=/usr/lib/libgmalloc.dylib the process takes SIGSEGV at the offending write instead. documented man 3 libgmalloc.
Its own banner states the limit, which is the part people leave out: “Allocations will be placed on 16 byte boundaries … Some buffer overruns may not be noticed.” Guard Malloc traps the page boundary, not your block boundary, so a small overrun inside the same page is still invisible. It is also enormously slower and uses a page per allocation, so it is a bisection tool, not something you leave on.
version-sensitive The seventeen-variable list, the zeroing boundary and the zone names are all properties of this libmalloc build. Re-read man 3 malloc on the machine in front of you rather than quoting this table.
7 · Fixing exercises
Two broken programs. Diagnose, repair, prove.
Both ship a deliberately broken program with a deterministic symptom, an interview-style prompt, the evidence you are expected to collect, a bounded success criterion, progressive hints, and a separate solution — plus a fixed source file, a patch from one to the other, and a check that proves both halves. The broken starting points are immutable. Copy one, repair your copy, and leave the original for the next time you want the drill cold.
Get the bundle, then run one command
Download os-memory-io-ipc-exercises.tar.gz — 44 files, 55,129 bytes. SHA-256 0a04e5544c047fc5376919d91fdcf5943a1c1316aa38996d93bac105b23ba13f
Raw link: labs/os-memory-io-ipc-exercises.tar.gz · bundle source: labs/os-memory-io-ipc-exercises/
The archive covers all four chapters — two exercises each for heap, scheduling, I/O and IPC. It is built deterministically from the same sources linked below — sorted member list, pinned timestamps, zeroed ownership, gzip -n — so rebuilding it reproduces that hash byte for byte, which was verified from a clean extraction. It carries no build products and no absolute paths, and the packer refuses to write an archive in which one appears.
tar xzf os-memory-io-ipc-exercises.tar.gz
cd os-memory-io-ipc-exercises
./run-all.shFor each exercise it compiles the broken and the fixed source each alone in a fresh temporary directory and fails on any unexpected warning; applies solution.patch to a copy of the broken file and requires the result to equal the fixed file byte for byte; runs the broken build and asserts its documented symptom; runs the fixed build and asserts its documented success; asserts the two builds produce the same answer; and repeats every timing-sensitive measurement. measured here From a clean extract on the reference machine: 129 assertions, 0 failures, 21–23 seconds, re-verified from a second independent extract, and ./tools/pack.sh rebuilt the archive to the same SHA-256 byte for byte.
One assertion used to be system-state sensitive, and the repair is worth reading as a lesson in its own right. Exercise 01 bounded the repaired build’s footprint growth at one round’s worth of pixels — 256,000 KB. That bound sat inside the range the correct build actually produces: twelve runs here spanned 162,288–326,112 KB, or 0.63×–1.27× of one round, and about half of them exceeded it. A learner extracting the archive could watch a correct answer fail. The growth is a high-water mark the allocator settles on, so it is neither zero nor repeatable to the megabyte. The bound is now a loose ceiling at two rounds — above every correct run observed, and still a factor of 4.4 below the broken build — and the assertion that decides is the one that never moved: the broken/fixed growth ratio, measured at 6.9×–21.5× against a threshold of 3.0, alongside the exact structural counts (deinits == created, live == 0, identical checksums). The general rule it demonstrates: assert the thing your fix actually changes. The repair changes object lifetime, which shows up exactly in the ratio and the counts; the absolute footprint is downstream of an allocator you do not control. measured here version-sensitive
./run-all.sh 01 02 runs only this chapter’s two.
SAFETY · READ BEFORE RUNNING. Exercise 01 allocates up to about 2.5 GB and exercise 02 up to about 1.1 GB, briefly. On a machine with little free memory, run them on their own. Every fixture carries an in-process watchdog thread that calls _exit(75) after a fixed budget — reporting through write(2) and exiting through _exit(2) rather than printf/exit — and every check.sh adds an external hard timeout. Nothing writes outside $TMPDIR, nothing needs elevation, and nothing touches the network.
01 · The browser that never gives memory back
A thumbnail browser decodes 4,000 thumbnails per folder. Each Thumbnailer stores a completion handler on itself so a failed decode can be retried. When the user navigates away, the cache is emptied. QA reports memory climbing and never coming back. The engineer who wrote it points out, correctly, that the file compiles with zero warnings under Swift 6 strict concurrency, and contains no unowned, no manual retain, no C and no unsafe pointer. Prove the defect exists, fix it, then explain why the fix does not reduce the footprint on the first folder — and what measurement does show it.
swiftc -swift-version 6 -O broken/thumbcache.swift -o /tmp/thumb_broken
/tmp/thumb_brokenExpected signal. deinits=0, live=32000, and a footprint that rises by exactly one folder’s worth every round — 322,848 KB after round 1 to 2,574,594 KB after round 8. The fixed build reports deinits=32000, live=0, an identical checksum, and flattens at 423,488 KB. measured here
Success criterion. deinits == created, live == 0, checksum and records unchanged, and growthKB at most one round’s worth of pixels — flat, not zero, and you can say why zero would be the wrong assertion. You can name which edge of the cycle you broke and why that one is correct rather than merely effective.
Progressive hints
- Draw the arrows. Three objects: the array, the
Thumbnailer, the closure. Draw every strong reference. The array’s disappears atremoveAll. Two remain, pointing at each other. - ARC is not a collector. It releases when a strong count reaches zero. It does not trace, and it will never notice that these two objects are unreachable from your program. That is not a bug in ARC; tracing is exactly the cost ARC exists to avoid.
- Which edge, and why that one. The owner holds strongly; the owned refers back weakly. Ask which of the two can meaningfully outlive the other. A completion handler for an object that no longer exists has nothing to complete.
Solution
Weaken the closure’s capture so the reference graph is no longer a cycle:
onComplete = { [weak self] in
guard let self else { return }
self.counters.checksum &+= UInt64(self.pixels[0]) &+ UInt64(self.index)
self.counters.records += 1
}Why weak and not unowned. Both break the cycle. unowned is a promise that the referent outlives the reference, and traps when broken. Here the referent is precisely the object whose death we are arranging, and the handler can in principle be invoked from a retry path after release. guard let self else { return } turns “the object is gone” into a no-op, which is the correct semantics for a retry.
Why the footprint did not drop on round one. Freeing an object returns its bytes to the allocator’s free lists. The allocator hands them out again on the next request; it does not hand pages back to the kernel. Round one costs the same either way and only the slope differs. This is not a Swift property and not an ARC property — section 3 measures the same thing directly in C.
What it costs. One optional unwrap per invocation and a weak reference, which means a side-table entry the first time the object is weakly referenced. Both negligible here. The real cost is that somebody has to keep noticing: every closure stored on the object it captures is a review trigger. inference
02 · The cache we emptied and the memory we did not get back
A map tile cache holds 20,000 decoded tiles of 32 KiB each. When the viewport moves, the tiles that left it are evicted — interleaved, scattered through the allocation order. The field reports that evicting half the cache does not move the process footprint at all. There is no leak; every byte is freed. Explain the measurement, fix it, and tell me what your fix gives up.
clang -O2 -g -Wall -Wextra broken/tilecache.c -o /tmp/tiles_broken
/tmp/tiles_brokenExpected signal. recoveredFractionAtHalf=0.000, reliefBytes=0, and a final footprint of 246.7 MB after every tile has been freed. The fixed build recovers 0.500 at half and finishes at 1.1 MB, with an identical checksum. measured here
Success criterion. recoveredFractionAtHalf at least 0.40, final footprint at most 10% of peak, checksum unchanged, and you can state the size threshold above which your repair is worth applying and what it costs below that threshold.
Progressive hints
- The allocator is not the virtual-memory system.
malloc/freemanage bytes inside regions;mmap/munmapmanage regions. Only the second pair talks to the kernel about residency. - Look at the size.
sysctl -n hw.pagesizeis 16384. A tile is 32768 bytes: exactly two pages with nothing left over. An allocation that is a whole number of pages and dies on its own has no reason to share a region. - And know when not to. Whatever you reach for has a per-call kernel cost and a one-page minimum granularity. Applying it to 64-byte objects would be a catastrophe. Work out roughly where the crossover is before you write the code.
Solution
Give each tile its own anonymous mapping:
tiles[i] = mmap(NULL, TILE_BYTES, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANON, -1, 0);
...
munmap(tiles[i], TILE_BYTES);Why it works and free() cannot. free() is a promise about reuse: the bytes may be handed out again by this allocator. It deliberately does not promise to return anything to the OS, because doing so on every free would mean a system call per free and a fresh page fault on the next allocation. Amortising those away is the allocator’s entire value. munmap is a promise about residency, and you pay for it in system calls.
What it costs. Two kernel transitions per object instead of roughly zero; page granularity, so a 20 KiB allocation would round to 32 KiB and waste 12; and a fresh zero-fill fault per page on first touch, because a new anonymous mapping starts unbacked. Measured here that is invisible against 32 KiB of memset; against a 64-byte object it would dominate completely.
The repairs that do not work. malloc_zone_pressure_relief is already in the fixture and returns 0 — it can only release regions that are entirely free, which is what fragmentation prevents. Evicting in allocation order helps, and means the eviction policy is now chosen by the allocator’s convenience rather than by what the user is looking at. A custom zone destroyed whole genuinely works for whole-generation eviction, and does nothing for the interleaved case in the prompt. inference
8 · Interview questions
Sixteen questions, with the follow-up that comes next.
Answer out loud before opening each one. The graded part is rarely the first sentence — it is whether you can name the evidence that would prove you wrong.
ExplainWhat is the difference between a memory leak and high memory use?
Three categories, not two. Useful memory is reachable and will be used again. Abandoned memory is reachable and will never be used again — a cache with no eviction, a growing singleton, a dictionary keyed on something never equal twice. Leaked memory is unreachable and can never be used again, typically a lost pointer or a reference cycle.
The distinction is operational, not academic: leaks(1) and the Leaks instrument find only the third kind, because they work by reachability. Abandoned memory is invisible to them — every byte has a live reference. Finding it needs generations, or knowing what your program is supposed to be holding.
High memory use is usually the second category, and it is usually the larger number.
Follow-up: “Your leak tool is clean and memory still grows. What now?” — Mark Generation before and after the operation, sort the surviving allocations by growth, and ask of the top type: should this still be alive?
ExplainWalk me through what happens when I call malloc(100).
The request is rounded up to a size class — 112 bytes on macOS 26.3, because the band from 97 to 128 has a 16-byte quantum. The allocator looks for a free block of that class in a region it already owns. If it finds one, that is the whole cost: no system call, no page fault, and the block may already be resident.
If it does not, it obtains a fresh region from the kernel and carves the block from that. The pages of a new region are clean and unbacked until written; the first write to each page takes a zero-fill fault and is the point at which the memory starts counting toward the process footprint.
The minimum is 16 bytes with 16-byte alignment, so malloc(4) also occupies 16.
Follow-up: “How would you check the 112?” — malloc_size() on the returned pointer. And do not do it with MallocStackLogging on, because the instrumented allocator rounds differently — 100 bytes reports 103 there.
DiagnoseA colleague fixed a retain cycle and the memory graph looks identical. Did the fix work?
Probably yes, and the measurement is wrong. Freeing an object returns bytes to the allocator, not pages to the kernel, so a single pass through the operation costs the same footprint either way. measured here the leaking and the repaired build of the same program both report about 322 MB after one round.
The discriminating measurement is repetition. Run the operation eight times: the leaking build climbs linearly to 2.5 GB, the repaired one flattens at 423 MB after the third round.
The direct evidence is cheaper still and needs no profiler: count deinit calls. A deinit that never runs is a fact.
Follow-up: “Why did the repaired build still grow 100 MB?” — the allocator settling at its high-water mark over the first few rounds. Assert that growth is flat, not that it is zero; an assertion of zero fails on correct code.
ExplainWhy did our footprint not drop when we evicted half the cache?
Because free bytes are not a free page. The allocator serves your 32 KiB requests from larger regions it owns, and a region returns to the kernel only when every allocation inside it is free. Evicting interleaved entries leaves every region with live tenants, so not one region qualifies. measured here evicting 10,000 of 20,000 tiles released 312 MB of bytes and returned 0.0 MB of footprint, and malloc_zone_pressure_relief returned 0 as well.
The repair is to stop putting those objects on the shared heap. A tile is a whole number of pages with an independent lifetime, so it belongs in its own mapping, where munmap returns it immediately regardless of order.
Follow-up: “When is that the wrong repair?” — below roughly a page, and for very numerous short-lived objects, where two system calls each dominates and page rounding wastes more than fragmentation did.
ExplainWhat is the difference between virtual size, resident size and footprint?
Virtual size is address space reserved. It is nearly meaningless: a trivial C program that has allocated nothing reports 425 GB of it on macOS 26.3, because the runtime reserves enormous ranges it never touches.
Resident size is pages currently in RAM, including clean pages that are backed by a file and could be discarded at any moment.
Footprint (phys_footprint) is what the system charges you: dirty pages plus swapped/compressed pages. It is the number that memory limits are enforced against, and the only one worth putting in a bug report.
Follow-up: “Show me the difference.” — mmap 512 MB and measure: virtual goes up 512 MB, resident and footprint do not move at all. Then memset it and all three move.
ChooseWhere would you put a 40 MB decoded image?
Not on the general-purpose heap if its lifetime is independent of everything around it. Options, in order of how often they are right:
Map the file if it is read-only — Data(contentsOf:options:.mappedIfSafe). Those pages stay clean, so they do not count toward the footprint and the system can discard and re-fault them under pressure. This is the largest single win available for read-only assets.
Its own anonymous mapping if it must be written — returned in full by munmap whenever you are done, regardless of what its neighbours are doing.
An evictable cache if it is a recomputable convenience — NSCache, which “incorporates various auto-eviction policies”, rather than a Dictionary, which has none.
Follow-up: “What stops your cache growing?” — if you cannot answer with a count limit, a cost limit, an eviction policy or a scope that ends, you have designed abandoned memory.
DiagnoseMemory spikes to a gigabyte during an import and comes back down. Is that a problem?
Yes, for three separate reasons, and they are worth naming separately. The spike causes system memory pressure, and the system reacts by compressing and swapping dirty pages, discarding clean ones and terminating background tasks. In the worst case your own process is terminated at the peak, and “it came back down” is no defence against that. And the spike fragments the heap — Apple states this directly: memory spikes cause “fragmentation or holes in heap memory regions”.
To find it: Allocations, select the spike interval, set the lifespan filter to Created & Destroyed, and switch the detail view to a call tree. That names the code allocating the transient memory.
Follow-up: “What is the most common cause in Swift?” — an autorelease pool that is not drained inside a loop, because calling into Objective-C-backed frameworks produces autoreleased objects that live until the thread’s top-level pool is cleaned. Wrap the loop body in autoreleasepool { }. It shows up in Allocations as @autoreleasepool content nodes.
ExplainWhy is unowned not just a faster weak?
Because it is a different claim. weak says “this may be gone, and I will check”. unowned says “this will outlive me”, and if the claim is false the access traps — a crash, in production, at an address that is no longer what it was.
The cost difference is real but small: weak needs a side-table entry so the reference can be zeroed, and every read is an optional. Choosing unowned to save that is trading a crash risk for an unwrap.
Use unowned when you can state the lifetime argument out loud — a child that cannot outlive its parent, a back-reference in a structure you own entirely. Otherwise weak.
Follow-up: “Where does the compiler help?” — nowhere. Both compile clean. The Memory Graph Debugger distinguishes strong, weak/unowned, unmanaged and conservative references, and that is the tool that shows you which edge you actually created.
ChooseYou have 30 seconds and a growing process. What do you run?
MallocStackLogging=lite on the process, then leaks <pid>. It is one command, it needs no Xcode, no GUI and no elevation, and it either names a cycle with its allocating stack or tells you the growth is reachable — which is itself the answer, because it rules out an entire category and points you at generations instead.
If it reports nothing, vmmap --summary <pid> next, to find out whether the growth is even in the heap. It may be IOSurface, or a mapped file, or thread stacks.
Follow-up: “And if it is a shipped app you cannot relaunch?” — vmmap and heap still work against a running process, but without stack logging you get sizes and types and no backtraces. Which is why the Malloc Stack checkbox belongs in your debug scheme permanently.
ExplainDoes a Swift struct avoid heap allocation?
The struct’s own storage is inline — on the stack as a local, inside the parent allocation as a property, inside the buffer as an array element. But a struct is not a promise about the heap: a String, Array, Dictionary, Set or any class reference inside it points at heap storage that is allocated exactly as it would be anywhere else.
A struct of four Ints costs nothing on the heap. A struct with one String in it costs a heap allocation per non-small string.
Closures are the case people forget: capturing values means allocating a closure context on the heap, one per live closure.
Follow-up: “How would you confirm it for a specific type?” — profile with Allocations and look at what actually appears, rather than reasoning from the declaration. Copy-on-write buffers in particular allocate at a moment that is not where the code is written.
DiagnoseAn engineer says they cannot reproduce a memory problem because Instruments changes the numbers. Are they right?
Partly, and the honest answer is more useful than either extreme. Stack logging genuinely perturbs the allocator. measured here a malloc(4096) reports 4096 normally and 5104 under MallocStackLogging; malloc(100) reports 112 normally and 103 under it. leaks correspondingly describes a 4 KiB leak as 5.00K. So any claim about size-class overhead or exact footprint must not be gathered with it on.
What it does not perturb is the thing you are usually looking for: whether an allocation survives, what holds it, and where it was created. Those are the questions stack logging exists to answer.
So: use stack logging to find what and why, and measure sizes and footprints without it.
Follow-up: “Can you profile heap on the Simulator?” — for heap analysis specifically, yes; Apple says the Simulator “is a lot closer in behavior, and it’s fine to use for memory profiling”, which is a deliberate exception to the usual run-on-device rule.
DesignYou own a framework other teams embed. What do you promise about memory?
Four things, and all four are contract, not implementation.
Ownership. Which objects the client owns, which you own, and which references are weak. A delegate that is accidentally strong is a leak in their app that they cannot fix.
Bounds. Every cache you keep has a stated limit and a way to clear it. “It is only a cache” is how abandoned memory gets shipped.
Peak, not just steady state. If a call transiently allocates ten times its result, say so, because their process may be near a limit even though yours never is.
A teardown that actually tears down. An object whose deinit never runs because your own closure retains it is your bug, and it will be diagnosed as theirs.
Follow-up: “How would you test that?” — a regression test that runs the public entry point ten times and asserts phys_footprint growth is flat. It catches abandoned memory, which no leak tool will ever flag.
Diagnoseleaks is clean and the process grows on every navigation. No Xcode available. Walk me through it.
Name the three categories first, and say that reachability tools only find the unreachable one — so a clean leaks run has ruled out a category, not the problem. What is left is almost certainly abandoned memory, and finding it needs two points in time rather than one.
The chain, with no GUI: run under MallocStackLogging=lite so allocations carry backtraces; leaks $PID --outputGraph=A.memgraph before the operation and --outputGraph=B.memgraph after; heap B --diffFrom=A -s -H to see what survived grouped by type; heap B --addresses=<Type> for one address; and leaks B --trace=<addr> for the chain of references holding it from a named root.
The strong answer also names the failure mode of that procedure: if A was taken after the growth already happened, the diff reports no new objects, which reads exactly like a clean result. Suspect your timing before you believe the program.
Follow-up: “What does turning on stack logging cost you?” — it moves allocations into MallocStackLoggingLiteZone, a separate allocator with a different layout, so every size figure gathered under it is wrong. Gather sizes without it and stacks with it.
Explainfree() returned and my footprint did not move. Is that a bug?
No. free is a promise about reuse, not about residency: the bytes went back to the allocator’s free lists, which is all it ever promised. Nothing in the C standard or Apple’s documentation says free returns pages to the kernel, and measured here a freed 512 MB block moved the footprint by zero.
What does return pages: munmap, immediately and unconditionally. And on Darwin specifically, madvise(ptr, n, MADV_FREE_REUSABLE) drops the accounting for a mapping without unmapping it — measured, 513.2 MB to 1.2 MB — while MADV_FREE_REUSE takes the charge back on reuse. Plain MADV_FREE does nothing to the footprint.
That pair is declared in the SDK header and absent from man 2 madvise, so treat it as coupled to the OS version rather than as a portable contract.
Follow-up: “So how would you actually shrink a cache’s footprint on demand?” — page-multiple, independently-lifetimed allocations in their own mappings, then munmap or MADV_FREE_REUSABLE. On the heap it is not reliably possible: measured, evicting alternate entries of a 32 KiB tile cache returned zero, because every region still had a live tenant.
ExplainFreed memory is zeroed on modern macOS. What does that actually buy you?
Quote the manual page and notice the hedge: “Starting in macOS 13 … free(3) fully zeroes many blocks immediately.” Many is load-bearing. Measured one size per fresh process, the split is clean at malloc_size 8,192: at or below it a freed block reads back zero, from 10,240 upward it still reads back whatever you wrote.
So the mitigation has a size boundary, and it is on the wrong side of the interesting bugs. A read-after-free of a small node now observes zeroes and probably crashes early, which is the point. A read-after-free of a decoded image buffer still silently “works”.
Follow-up: “Does MallocScribble help you see it?” — only above the boundary. Below it the zeroing has already happened, so “I enabled MallocScribble and saw nothing” is not evidence of a clean program. And MallocGuardEdges is large-block only, so it will not catch a 32-byte overrun either — that is what libgmalloc is for, and its own banner admits it misses overruns inside a page.
ChooseYou want to shrink a cache to save memory. What could go wrong?
A bound chosen without reference to the working set. Measured on a 600-file working set scanned sequentially eight times behind an LRU cache: at countLimit 128, 256 and 512 the hit rate was exactly zero — not degraded, zero — and at 1,024 it was 87.5%. Below the working set of a cyclic scan, every entry is evicted before it is reused.
The consequence is worse than no cache: you pay the full memory, the full allocation traffic and the full lookup cost, and you get nothing back. A cache bound is a claim about the working set, and it should be measured rather than rounded.
Follow-up: “How would you have noticed?” — report hits and misses, not just size. A cache with no hit-rate counter is a cache you cannot reason about, and this failure is invisible in every memory graph because the memory usage looks better.
Drill
Answer first, then read the explanation.
One defensible first move each. Say it out loud before clicking, then say why the other three are premature rather than merely wrong.
Scenario 01 · The fix that looks like nothing
You broke a retain cycle. The process footprint after one pass through the feature is identical to before. What do you do?
Scenario 02 · Clean leak report, growing process
leaks reports nothing and memory still climbs with every navigation. First move?
Scenario 03 · The allocation that costs nothing
You map 512 MB and the footprint does not change. Why?
Scenario 04 · A read-only asset bundle
Your app loads 200 MB of read-only reference data at launch and never writes to it. What change helps most?
Record your answer · 1
Bound one cache you own.
Pick a real cache in your code. Write: what it holds → what stops it growing (count, cost, policy, or scope) → what the user pays if the bound is wrong in each direction → the measurement that would tell you.
Record your answer · 2
Rehearse one memory diagnosis.
For a growth problem you have actually seen, write the full chain: symptom → leaked or abandoned or transient → first discriminating tool → the evidence that would rule each out → the smallest fix → the regression guard.
Primary sources for this chapter
Apple documentation, WWDC sessions, and macOS manual pages
Every quoted sentence in this chapter comes from one of these. Raw URLs are printed beside each title so they can be copied without following a link. Behaviour and availability drift between releases — re-check against your deployment target before repeating a claim in an interview.
| Source | URL (copyable) | Used in this chapter for |
|---|---|---|
| WWDC24 10173 · Analyze heap memory | https://developer.apple.com/videos/play/wwdc2024/10173/ | Timestamps verified against the published transcript: clean/dirty/swapped and what counts 2:10–2:42; malloc’s contract 3:07–3:13; 16-byte minimum and alignment 3:19–3:23; autorelease pools in loops 10:50–11:19; useful vs abandoned vs leaked 20:26–20:51; closure contexts 21:54–22:06. Cited without a verified timestamp: zero-on-free, generations, fragmentation from spikes, the four reference kinds, Simulator for heap profiling |
| Reducing your app’s memory use | https://developer.apple.com/documentation/xcode/reducing-your-app-s-memory-use | Pages × page size as the metric; “writing a single byte … can increase memory use by 16 KB”; clean-to-dirty transition |
| Gathering information about memory use | https://developer.apple.com/documentation/xcode/gathering-information-about-memory-use | Memory report limits; Memory Graph Debugger; Malloc Stack checkbox; Allocations and the Generations view; vmmap and leaks |
| NSCache | https://developer.apple.com/documentation/foundation/nscache | Auto-eviction policies; thread safety without external locking |
| Data.ReadingOptions | https://developer.apple.com/documentation/foundation/data/readingoptions | Mapping a file instead of copying it |
| NSAutoreleasePool | https://developer.apple.com/documentation/foundation/nsautoreleasepool | Pool scoping inside loops |
| WWDC18 416 · iOS Memory Deep Dive | https://developer.apple.com/videos/play/wwdc2018/416/ | Background reading on footprint and page state; cited by Apple’s own memory documentation |
| WWDC21 10180 · Detect and diagnose memory issues | https://developer.apple.com/videos/play/wwdc2021/10180/ | Background reading on the memory-graph workflow |
macOS 26.3 manual page malloc(3) | man 3 malloc | MallocStackLogging lite/full semantics; MallocScribble; MallocZeroOnFree and its macOS 13 provenance; MallocGuardEdges |
macOS 26.3 manual page malloc_zone_malloc(3) | man 3 malloc_zone_malloc | Zone creation, default zone, malloc_zone_from_ptr, destroy semantics |
macOS 26.3 manual pages malloc_size(3), mmap(2), madvise(2) | man 3 malloc_size · man 2 mmap · man 2 madvise | Actual allocation size; anonymous private mappings; MADV_FREE semantics |
macOS 26.3 manual pages leaks(1), vmmap(1), heap(1), malloc_history(1) | man 1 leaks · man 1 vmmap · man 1 heap · man 1 malloc_history | The command-line workflows in section 6 |
How to read the labels on every claim in this chapter
documented — Apple documentation, an official WWDC or Tech Talk transcript, Swift Evolution, or a Darwin manual page or SDK header on macOS 26.3. Quoted verbatim with the source named beside it.
measured here — observed on one machine: Apple M4 Pro (10 performance + 4 efficiency cores), macOS 26.3 (25D125), 48 GB, 16 KiB pages, Swift 6.2.4, Apple clang 17.0.0, Xcode 26.3. Evidence of a mechanism, never a benchmark. Reproduce; do not quote.
inference — our reasoning joining the two above. Marked so you can disagree with the reasoning without doubting the evidence.
negative result — something that was tried and did not show what was expected. These bound what a chapter is allowed to claim, and they are kept deliberately visible.
correction — a correction to an earlier reading published on this page, naming what was wrong and why. This page’s whole method is labelled provenance, so its own errata carry a label too rather than disappearing into a rewrite.
version-sensitive — observed behaviour of this OS build that must be re-measured on another before it is relied on.
Provenance for this chapter. Everything labelled measured here here was run on 2026-09-23, with 129 bundle assertions passing and 0 failing from a clean extraction. Two earlier negatives in this chapter have since been narrowed rather than removed: the zone enumeration is still one zone, but vmmap sees the region families the zone API cannot; and MADV_FREE still moves nothing, but MADV_FREE_REUSABLE does. A negative result bounds a claim only as far as the instrument was the right one.
Scope of the evidence, stated plainly. Every figure labelled measured here comes from a small C or Swift program compiled with clang -Wall -Wextra or swiftc -swift-version 6 and run from a terminal; the two exercises are in the downloadable bundle and their assertions run in CI-style from a clean extract. What this chapter does not have: no Instruments window was opened and no screenshot appears anywhere, so the Allocations lifespan filter, the Generations view, VM Tracker and the Memory Graph Debugger canvas are documentation rather than observation — but the questions those views answer are now exercised on the command line in section 6A, against a real growing process, with heap, leaks --outputGraph, heap --diffFrom, leaks --trace and malloc_history all run. No reclaim under genuine system-wide memory pressure was ever observed, because inducing that is not something a fixture should do: MADV_FREE_REUSABLE returning the footprint is an accounting change observed immediately, not evidence about behaviour when the machine is actually short of memory. malloc_history -highWaterMark and heap --layouts are named from their tools’ usage output and were not run. The exact boundary between the MALLOC_TINY and MALLOC_SMALL region families was not bisected, and MALLOC_NANO was never observed serving an allocation — only its metadata region appeared, so whether the nano path is inactive on this build or simply not attributed by vmmap is unresolved, and section 2 is worded to state only what was observed.
Chapter 5 · threading and scheduling
Threading and scheduling.
What runs, where it runs, and who decided. This chapter holds both halves of that question in one place: the concurrency you write — threads, queues, tasks, actors, quality of service — and the scheduling the kernel does underneath it. Locks are chapter 6; the ways concurrent code stops making progress are chapter 7; the diagnostics, experiments, exercises and assessment all three share are in the execution lab that follows them.
The problem, in plain words
A machine has a handful of cores and far more work than cores. A thread is one line of that work — somewhere to keep how far it has got. The system decides, moment by moment, which threads sit on a core and which ones wait. It cannot stop and ask you, so you tell it indirectly: you say how urgent each piece of work is, and it does the rest.
Two words are worth separating on day one. Concurrency is how much work you have in flight. Parallelism is how much is actually executing at this instant. You choose the first. The hardware chooses the second, and on Apple silicon the number it chooses can change under you.
What breaks without it. One thread draws your window. If it ends up waiting behind something slow, the app stops answering — no clicks, no keys, no drawing. Almost every freeze, stutter and “slow only sometimes” report in the rest of this reference starts as a scheduling story.
How to recognise it
The symptom arrives as a complaint about time, never about threads: “it judders”, “it hangs for a second”, “the fans spin up”, “it got slower when we added workers”. What makes it this chapter rather than the next two is that nothing is broken and nothing is shared — there is simply more runnable work than there are cores, or the work is running at the wrong urgency.
The discriminator is one question, and it is cheap: is the slow thread busy, or is it waiting? A thread burning a core is a profiling problem. A thread parked on a lock is chapter 6. A thread that is neither — ready to run with no core free — is this chapter, and it is the case ordinary profilers are worst at showing.
The tell: the average looks fine and the user does not. When a mean, a total or a “time spent in function” exonerates the code while the complaint survives, you are looking at a tail, and a tail is made of threads that were ready and not running. Measure percentiles of a fixed unit of work against an idle baseline before you change a line.
The idea
A thread is always in exactly one of three states: running on a core, ready to run with no core free, or waiting for something that has not happened yet. Every scheduling question in this chapter is really a question about which of the three a thread is in, and for how long.
The one people forget is the middle one. Waiting is visible — a blocking frame sits in the backtrace and every tool shows it. Ready leaves no trace at all: no lock is held, no stack is interesting, nothing is hot, and yet the work is not happening. That is why adding threads so often makes an app slower rather than faster: past the core count, an extra thread cannot add throughput, only competition and switches.
Quality of service is how you influence the choice. It is not a priority number you set; it is a statement of intent the system maps onto priority bands and core types. Declaring everything urgent means nothing is.
Watch it run
Three threads, two cores
Nothing is broken here. No lock is taken, no thread is blocked, nothing crashes — and one of the three still spends most of its life making no progress.
running · holds a core
ready · no core free
waiting · blocked on something
Thread Arunningdecoding an image on core 1
Thread Brunningdecoding an image on core 2
Thread Creadythe one the user is waiting for — and there is no core
Step 1. Two cores, three threads. Everything is healthy by every ordinary measure: no lock, no wait, both cores busy.
Thread Areadyits turn ended — the kernel took the core away
Thread Brunningstill decoding
Thread Crunningfinally on a core
Step 2. A preemption. The kernel rotates the core to C. That switch is not free: the incoming thread finds the caches full of someone else’s data.
Thread Arunningback on a core
Thread Bwaitingreading a file — its core is free
Thread Crunningstill going
Step 3. B blocks on I/O and gives its core back. This is the case where an extra thread helps: a blocked thread is covered by another one.
Thread Arunningholds core 1
Thread Breadyits read finished — and now it queues for a core
Thread Crunningholds core 2
Step 4. B’s wait is satisfied, so it is ready — not running. The wait that the tools showed you is over; the delay that the user feels is not.
Thread Cfinishedlate, by the sum of every interval it spent ready
Profilersaw nothingno hot frame, no blocking frame, no lock
Mean latencyfinethe tail is where the whole defect lives
Step 5. C finishes late without ever being blocked and without ever being hot. Ready is a state, it costs real time, and almost nothing reports it by default.
Section 2 gives this its vocabulary, section 9 measures what the switches cost, and section 11 shows the percentile measurement that makes it visible.
The code
Every claim in this chapter has a program behind it, and they are runnable from a terminal with no Xcode window. The ones that belong to this chapter are:
How this chapter is built. Every rewritten chapter in this reference runs the same eight steps in the same order: overview in plain words (above) → mental model and vocabulary (1–2) → Darwin mechanism and where the public contract stops (3–5) → deeper internals (6–10) → Mac lab (11–11A, plus C in the execution lab) → failure story and diagnosis workflow (12, plus A, B, D and E) → interview questions (13, plus F and G) → mistakes, follow-ups and recap (14).
Which thread runs this code, which core it can land on, and what the user experiences when the answer is “none of them, right now”.
Name the instrument before the fix, and say what result would falsify your hypothesis.
Kernel scheduler internals, the cooperative pool’s implementation, Mach thread-policy APIs, real-time thread admission math.
The sentence to have ready
“Concurrency is my design; parallelism is the hardware’s.” Almost every question in this chapter — how many threads, why more of them did not help, why the average looked fine, why a background job made the UI wait — resolves faster once you separate the work you put in flight from the work a core is actually executing.
1 · Execution model
Process, thread, run loop.
A process is an isolation boundary: its own virtual address space, file descriptors, ports, and entitlements. A thread is a schedulable execution context inside that boundary: its own stack, register state, priority, and scheduling state — but the same heap as every sibling. That single asymmetry is the source of every problem in this chapter.
Simplified diagram, not a screenshot and not a model of any specific kernel scheduler. It is drawn to make one interview point: “the app is stuck” splits into busy on a core and waiting for something, and those two have completely different repairs.
What a thread owns
A stack, register state, thread-local storage, a scheduling priority and QoS, and a name. What it does not own is the heap, globals, file descriptors, or any object graph — all of that is shared with every other thread in the process.
Why the main thread is special
Not by magic. It is the thread that runs the main run loop, and AppKit gives it exclusive ownership of the view hierarchy. Apple states the rule plainly: “there’s only one thread that can make UI updates: the main thread.”
Source: Understanding hangs in your app.
The run loop is a wait, not a spin
The run loop sleeps in the kernel on its input sources — Mach ports, timers, custom sources — wakes when one is signalled, dispatches the handler, notifies observers, then sleeps again. A run loop pegged at 100% CPU is not looping; your handler is.
Budget, stated by Apple
Under 100 ms of synchronous main-thread work for a discrete interaction; under one display refresh interval (8 or 17 ms) for continuous interaction, and “if the work … is less than 5 ms, the update is usually ready in time.”
Source: Improving app responsiveness.
1AOne turn of the main run loopevent → handler → CA commit
Interview explanation: Apple describes the main thread’s job for an event as three stages: deliver the event to the right handler, make state changes, then perform a Core Animation commit that submits the view-hierarchy changes to the render server. Your code dominates stage two, but influences how expensive stage three is.
Why it matters for locks: every stage runs on the one thread that must never wait. A lock taken on the main thread is a lock whose worst-case hold time is now a user-visible latency budget. That is the whole argument for fine granularity on UI paths.
The kernel’s half of this chapter
What the scheduler is actually deciding.
The sections whose subject is Dispatch, Swift concurrency or an actor are the concurrency you write. The sections whose subject is thread states, priority bands, preemption and donation are what the kernel does underneath it — the states a thread can be in, what a priority actually is, what a preemption costs, how urgency travels across a wait, and what System Trace shows. This reference teaches the two together because interviews test them together, and because you cannot explain a stall from either side alone. Where they touch, they link rather than repeat.
Which state a thread is in, why the scheduler chose what it chose, and what the user experiences as a result.
Separate running from runnable from blocked with evidence, and know which percentile to look at.
Run-queue data structures, load-balancing heuristics, the cluster-migration policy, real-time thread admission math.
The sentence to have ready
“Runnable is a third state, and it is where latency hides.” Almost every scheduling question — why the mean looks fine, why nothing is blocked and the app still stutters, why adding threads did not help — resolves once you stop collapsing “running” and “ready to run but not on a core” into one idea.
2 · Three states
Running, runnable, blocked.
A thread is doing exactly one of three things at any instant: executing on a core, waiting for a core, or waiting for something that is not a core. They have three different costs and three different repairs, and the single most common diagnostic mistake is not distinguishing the first two.
Simplified diagram, not a model of any specific kernel scheduler. It exists to make one point: “the app is slow” is three different problems, and the first job is to say which.
What the kernel actually reports — and the limitation that matters
measured here thread_info(..., THREAD_BASIC_INFO) on a process with one spinning thread, one sleeping thread, one thread waiting on a mutex and one thread waiting on a read:
# run_state cpu_usage policy
0 RUNNING 0 timeshare <- main, mostly idle
1 RUNNING 969 timeshare <- the spinner (96.9%)
2 WAITING(interruptible) 0 timeshare <- sleeping on a timer
3 WAITING(interruptible) 0 timeshare <- blocked on a mutex
4 WAITING(interruptible) 0 timeshare <- blocked on a readRead rows 2, 3 and 4. A thread sleeping on a timer, a thread parked on a lock, and a thread waiting on I/O all report the same TH_STATE_WAITING. The kernel’s state field tells you a thread is not runnable; it does not tell you what it is waiting for.
So the state is never the whole answer — you always need the stack too. That is exactly why sample, spindump and Instruments’ Thread State Trace pair a state with a backtrace, and why “it says WAITING” is an incomplete diagnosis. inference
Simplified diagram from a headless System Trace export, thread-state table, aggregated over the traced process. Bar lengths are proportional; five of the seven observed states are shown. The share moves with load — a second run of the same shape reported 45.9% Preempted. Reproduce; do not quote.
Running costs a core
And therefore energy, and therefore a share every other runnable thread does not get. A thread that is running but computing nothing — a spin loop — is the purest form of waste, because it is indistinguishable from useful work to the scheduler.
Runnable costs latency
The thread is ready and there is no core for it. Nothing is blocked, no lock is held, no profiler samples it, and the user sees a hitch. This is the state that ordinary CPU profiling is worst at showing, and it is what section 9 measures.
Blocked is cheap
Off every run queue, costing a stack and a scheduler slot. Cheap, but not free — a thousand blocked threads is a thousand stacks. And the wait itself can be a correctness problem if it is holding something.
The cost of not distinguishing them
measured here A worker pool that spins while idle held 12.64 cores continuously to process 200 small frames, against 0.003 CPU-seconds for the same pool blocking on a condition variable — a factor of about 5,700, with identical output and slightly better wall time.
Three states is the kernel’s model. Your profiler reports eight — and the difference will send you to the wrong lane.
A System Trace recording exported from the command line carries a thread-state table whose values are Blocked, Idle, Interrupted, Preempted, Runnable, Running, Terminated and Unknown. The three-state model above still holds — those eight collapse onto it — but the middle state is split in two, and the names are not the ones this section just taught you:
| This section’s state | What the trace calls it | When |
|---|---|---|
| Runnable | Preempted | The thread was running and lost its core. This is the one you will actually see. |
Runnable | The thread was just woken and has not been dispatched yet. Almost always near zero. | |
| Running | Running, Interrupted | Interrupted is an interrupt stealing the core for microseconds at a time. |
| Blocked | Blocked, Idle, Terminated | Waiting for something that is not a core, or gone. |
Why this matters more than a naming quibble. Told to “look for runnable”, a reader opens the Thread State Trace, finds the Runnable lane essentially empty, and concludes the machine is not oversubscribed. Measured here on 28 CPU-bound threads at USER_INTERACTIVE on 14 cores:
state intervals total ms share
Preempted 39933 21647.2 0.526 <- ready to run, no core
Running 106762 16686.2 0.406
Blocked 30 1419.5 0.035
Terminated 29 1258.0 0.031
Interrupted 66799 122.6 0.003
Runnable 1 0.0 0.000 <- the lane people are told to watchMore than half of all thread-time was spent ready to run with nowhere to run it, and the lane named Runnable recorded a single interval. That is this section’s central claim — “runnable costs latency” — with a number on it, and it is also the reason to say Preempted out loud when you give the answer. measured here version-sensitive The share moves with load: a second run of the same shape reported 0.459. Quote the ordering, not the fraction.
The narrative rows are quotable and teach the mechanism directly — “Running at priority 31 on CPU 9 (P Core)” and “made runnable by saturate 0x1aecbd7 running on CPU 12 (P Core)”. The second one names who performed the wakeup, which is what turns “my thread was late” into “this other thread woke it late”. measured here
How to get this table without opening Instruments is section 11A. The threading half of this chapter reaches the same eight-state taxonomy from the other direction, through its own System Trace work — see 3 · Scheduling & cores, which measures the same oversubscription result with the same instrument and is worth reading beside this.
3 · Scheduling & cores
Concurrency is your design. Parallelism is the hardware’s.
Concurrency is how many things are in flight; parallelism is how many are executing at this instant. You choose the first. The second is activeProcessorCount, and on Apple silicon it can shrink under you. Every oversubscription argument in this chapter follows from keeping those two apart.
Two core types, a similar microarchitecture
Apple is explicit that you should not write different code for each: “the P and E cores use a similar microarchitecture … They’ve been designed so that developers don’t need to care whether a thread runs on a P or E core.” The scheduler decides placement, and QoS is how you influence it.
documented Source: Tech Talk 110147 · Tune CPU job scheduling for Apple silicon games (2022).
P-core availability is not promised
From the same session: “note the availability of P cores is not guaranteed. The system reserves the right to make them unavailable under critical thermal scenarios.” So the width of every pool you depend on is a runtime value, not a build-time constant.
documented
The number that actually sizes your pool
Apple: “Whereas the processorCount property reports the number of advertised processing cores, the activeProcessorCount property reflects the actual number of active processing cores … including boot arguments, thermal throttling, or a manufacturing defect.” Size pools from the second one.
documented Source: ProcessInfo.activeProcessorCount.
Do not scale threads with workload
Apple’s guideline, verbatim: “scale the thread count to match the CPU core count. Avoid recreating new thread pools in each framework or middleware you are using. Do not scale your thread count based on your workload either.” The corollary is that a per-subsystem pool is a bug even when each one is individually reasonable.
documented Tech Talk 110147 (2022).
Simplified diagram, not a screenshot, and not a model of the XNU scheduler. Two boxes are collapsed on purpose: Instruments distinguishes Runnable (ready, has not run this quantum) from Preempted (was running, lost the core), and both mean “ready, waiting for a core”. documented The eight state names are the distinct values present in an exported thread-state table from a whole-system trace; Interrupted means a core taken by an interrupt handler, not by the scheduler.
The diagnostic payload most people never reach for
Thread State Trace’s Narrative column tells you who woke this thread — either “made runnable by a timer expiration” or “made runnable by <thread> (<process>, pid: N)”. That single string separates “waiting on a timer” from “waiting on a peer”, and it is how you find the other end of a lock wait without guessing. inference Note the trace attributes the wakeup, not ownership; for a mutex those coincide, for a condition variable or semaphore they need not.
| QoS set at thread creation | Dominant scheduling priority | Running time on P cores | Preempted in a 1.5 s window |
|---|---|---|---|
USER_INTERACTIVE | 31 | 99.9% | 6.0 ms |
USER_INITIATED | 31 | 99.9% | 7.1 ms |
DEFAULT | 31 | 99.8% | 7.4 ms |
UTILITY | 20 | 99.1% | 55.8 ms |
BACKGROUND | 4 | 0.0% — 100% on E cores | 221.4 ms |
measured here Five threads, each with its QoS set at creation through pthread_attr_set_qos_class_np, each spinning 1.5 s, recorded headlessly with xcrun xctrace record --template "System Trace" on an Apple M4 Pro (10 performance + 4 efficiency cores), macOS 26.3 (25D125). Reproduce these; do not quote them.
QoS is a placement decision, not a speed dial
BACKGROUND did not run “a bit slower”. It ran on a different kind of core, exclusively, and was preempted roughly 37× more than user-interactive work in the same window. That ratio is the mechanism behind “the system may defer it”.
measured here inference — the reading of the ratio is ours.
The background band is capped
A holder plus N background spinners never exceeded ~3.6 of 14 cores, and quadrupling the thread count from 14 to 56 to 112 bought nothing. QoS is a resource budget: more threads inside a capped band just divide the same budget more ways.
measured here
Declaring everything urgent means nothing is
The top three classes collapsed to the same band (31) in this process. It is a command-line tool, not a foreground UI app — other processes’ threads in the same whole-system trace sat at 46, 47, 37, 20 and 4. The higher bands exist; a non-UI process simply does not get them.
measured here Always say which kind of process a QoS measurement came from.
Oversubscription costs latency, not throughput
At 1×, 2×, 3× and 4× the core count on a CPU-bound workload, total throughput stayed flat. What changed was per-thread progress (divided by the factor) and the spread between luckiest and unluckiest thread, which widened from ~1.06× to ~1.4×.
measured here correction An earlier single-pair reading suggested a 9% throughput drop; three runs at each of four levels showed that was noise. Promise latency, not throughput.
3AWhere the extra threads actually goPreempted 19.7% → 71.0%
The measurement. The same CPU-bound program recorded under System Trace at 14 threads (1×) and 42 threads (3×) on 14 cores, aggregating every thread-state interval for the process:
| Interval kind | 14 threads (1×) | 42 threads (3×) |
|---|---|---|
| Running | 16,757.5 ms (79.7%) | 17,319.5 ms (27.1%) |
| Preempted | 4,146.1 ms (19.7%) | 45,398.7 ms (71.0%) |
| Interrupted | 93.7 ms (0.4%) | 103.6 ms (0.2%) |
| (Preempted + Runnable) ÷ Running | 24.7% | 262.1% |
Interview explanation: the Running row barely moves — 16.76 s to 17.32 s of core time in the same wall window — because 14 cores were already saturated at 1×. Everything the extra 28 threads bought was waiting-to-run time, which rose more than tenfold. That is the empirical content of Apple’s sentence: “As the number of threads running on the device increases, the operating system schedules each thread less often on a CPU core.” documented (Improving app responsiveness)
The number that corrects a common belief: 200 parked threads cost 16.2 KB of physical footprint each, not the 512 KB of their default secondary stack — that stack is reserved address space, faulted in a page at a time. Thread creation measured 13–17 µs per create-and-join. measured here So the argument against many threads is not “you will run out of RAM”. It is scheduling.
Where to go next. The kernel’s half of this chapter sits one layer below this section and does not repeat it: 2 · Three states reconciles this section’s eight-state Instruments taxonomy with the kernel’s three; 4 · Priority and QoS measures the QoS → band mapping and P/E placement per class; and 11A shows how to record these traces without opening Instruments.
4 · Priority and QoS
What you declare, and what the kernel does with it.
You do not set a priority number. You declare an intent, as a quality-of-service class, and the system maps that to a scheduling band. Knowing the classes, their order, and what each one means in Apple’s own words is the concrete part; knowing that a declaration is only meaningful relative to what else is queued is the part that actually gets used.
| QoS class | Raw value | Apple’s description | Typical use |
|---|---|---|---|
QOS_CLASS_USER_INTERACTIVE | 0x21 | “work performed by this thread is interactive with the user … a request to run with nearly all available system CPU and I/O bandwidth even under contention. This is not an energy-efficient QOS class to use for large tasks.” | Main thread, animation, audio render |
QOS_CLASS_USER_INITIATED | 0x19 | Work the user started and is waiting on | Opening a document the user just picked |
QOS_CLASS_DEFAULT | 0x15 | “Threads created by pthread_create() without an attribute specifying a QOS class will default to QOS_CLASS_DEFAULT. This QOS class value is not intended to be used as a work classification” | Nothing, deliberately — it is the absence of a decision |
QOS_CLASS_UTILITY | 0x11 | Work that “may or may not be initiated by the user and that the user is unlikely to be waiting for” | Indexing with visible progress, sync |
QOS_CLASS_BACKGROUND | 0x09 | “work … not initiated by the user and that the user may be unaware of the results … should be run in the most energy and thermally-efficient manner” | Prefetch, maintenance, opportunistic cleanup |
QOS_CLASS_UNSPECIFIED | 0x00 | Legacy / no information | — |
documented Quotations and raw values from <sys/qos.h> in the macOS 26.3 SDK. https://developer.apple.com/documentation/dispatch/dispatchqos
QoS is per-thread, not per-process
pthread_set_qos_class_self_np affects the calling thread. A thread created by a thread that set a class does not inherit it through pthread_create. The practical consequence: a pool must declare its class on each worker, as that worker’s first act. Getting this wrong is the single most common version of “I set the QoS and nothing changed”.
Declaring the top class is not a fix
measured here A latency-sensitive thread that had already called pthread_set_qos_class_self_np(QOS_CLASS_USER_INTERACTIVE, 0) still saw its tail inflate 79× when 56 undeclared CPU-bound threads ran on 14 cores. A priority is only meaningful relative to what else is in the queue — the repair was demoting the other work.
The ladder is not a switch
measured here Same load at three settings, as p99 dispatch latency against an idle baseline: undeclared 79×, UTILITY ≈3×, BACKGROUND ≈2.7×. Utility is a real, intermediate position. “I set it to background” is a weaker answer than “utility was not enough, and here is the number”.
The scheduler in use, named
measured here sysctl kern.sched reports edge on macOS 26.3 on Apple silicon, and hw.perflevel0.logicalcpu / hw.perflevel1.logicalcpu report 10 and 4 on this M4 Pro. inference These are observable but undocumented as API — useful for understanding the machine in front of you, never something to branch on in shipping code.
4AWhat each class actually buys, measured three waysone thread per class, one trace, plus untraced wall time
The table above is the declaration. This is the consequence. Five CPU-bound threads, one per class, each named after its class and each setting its own class as its first act, recorded once with System Trace and read back per thread.
thread sched priorities seen P samples E samples P share running ms
qos-ui 31 675 10 0.985 345.1
qos-uinit 31 697 19 0.973 342.0
qos-def 31 670 8 0.988 343.6
qos-util 20 (and 31 before it declared) 1438 136 0.914 343.1
qos-bg 4 (and 31 before it declared) 489 4153 0.105 512.61 · The top three classes are one band. USER_INTERACTIVE, USER_INITIATED and DEFAULT all resolve to priority 31 and all ran on performance cores. This is the measured form of the rule section 3 states in words: declaring everything urgent means nothing is, because they were already the same thing.
2 · The ladder has two real steps. UTILITY lowers the priority from 31 to 20 and keeps performance cores. BACKGROUND lowers it to 4 and moves the work to efficiency cores. Those are the only two declarations that changed anything here.
3 · Placement is a preference, not a partition. The BACKGROUND thread still took 489 samples on performance cores. There is no core the class cannot reach; there is a strong preference the scheduler applies.
4 · The demotion has a price, and you should quote it. Competing with the other four, BACKGROUND needed 512.6 ms of processor time for the same work the others finished in ~343 ms. Measured separately on an idle machine with nothing to compete against, four BACKGROUND threads took 1.47 s against 0.60 s for USER_INTERACTIVE — about 2.4×. If the batch has a deadline of its own, demotion is the wrong repair; make it smaller or interruptible instead.
5 · And there is a teaching detail hiding in column two. The util and bg rows carry samples at priority 31 as well as at 20 and 4 — the thread starts in the default band and drops at the instant pthread_set_qos_class_self_np runs. That is direct visual proof of the rule in the card above: the declaration has to be the worker’s first act, because everything before it runs at the wrong priority.
measured here One headless System Trace plus three untraced repetitions for the wall-time figures. inference The mapping is read from a trace, not from an API. version-sensitive 31, 20 and 4 are observations on one OS build and must never be branched on. What transfers is the shape: three classes collapse, two do something, and only one of them moves you to the efficiency cluster.
Simplified diagram from one headless System Trace on one M4 Pro, kern.sched=edge. Priorities and core shares are observations, not documented API — reproduce; do not quote.
taskpolicy(1) demotes a process without touching its code
The question “how does my app behave when the system demotes it?” usually gets answered by editing the app. It does not have to be.
taskpolicy -c utility ./probe # clamp the QoS class
taskpolicy -b ./probe # process-level Darwin background tier
taskpolicy -b -p <pid> # retag a process that is already running
# invocation qos_class_self() getpriority(PRIO_DARWIN_PROCESS)
# ./probe 0x21 USER_INTERACTIVE 0
# taskpolicy -b ./probe 0x21 USER_INTERACTIVE 1
# taskpolicy -c utility … 0x11 UTILITY 0Read the middle row. -b sets the process-level background tier and leaves the thread’s declared class alone — so a program that checks only qos_class_self() will not notice it has been demoted. The in-process way to ask is getpriority(PRIO_DARWIN_PROCESS, 0). measured here documented man 2 getpriority; flags from taskpolicy’s own usage output.
What this chapter does not repeat
The threading half of this chapter already covers, with its own measurements: how QoS maps to observed priority bands, that the background band is capped to a small fraction of the cores, that declaring everything urgent collapses the top three classes into one band, and Apple’s guidance to size pools from activeProcessorCount and explicitly not to scale thread count with workload. Read 3 · Scheduling & cores alongside this section rather than instead of it.
5 · Threads vs tasks
Two schedulers, stacked.
The kernel schedules threads onto cores. The Swift runtime schedules tasks onto a fixed pool of threads. Both are real, they have different units of preemption, and confusing which one you are reasoning about produces most of the surprising behaviour in async code.
| Kernel thread | Swift task on the cooperative pool | |
|---|---|---|
| Scheduled by | The kernel, onto a core | The Swift runtime, onto a pool thread |
| Preemption | Pre-emptive — the core can be taken at any instruction | Cooperative — only at a suspension point (await) |
| Cost to create | A stack (megabytes of address space) and a kernel object | A heap allocation; cheap enough to make millions |
| Blocking it | Costs a stack and a scheduler slot | Costs a whole pool thread, which the pool will not replace |
| Pool width | You choose, and should choose from the core count | Fixed by the runtime. Measured here it is exactly activeProcessorCount — see 5A |
| What starves it | Higher-priority runnable threads | A task that never suspends, holding a pool thread |
The forward-progress contract
The cooperative pool is sized near the core count precisely because tasks are expected to make progress or suspend. Blocking a task on a lock, a semaphore, or a synchronous read breaks that contract: the pool thread is gone and the pool will not grow to compensate. The failure mode is not slowness, it is a stall.
Why await is a scheduling word
An await is the point at which the runtime may take the thread back and run something else. Between two awaits your task owns its thread outright — which is why a long synchronous computation inside an async function is exactly as blocking as it would be anywhere else, and no amount of async spelling changes that.
The two questions to keep separate
“Which thread is this running on?” and “which executor is this isolated to?” have different answers and different failure modes. The first governs whether you can block; the second governs whether you can touch state.
Where a thread is still the right unit
Real-time audio, anything with a hard deadline, and anything that must call a blocking C API for a long time. Those want a dedicated thread with a declared class — not a task on a shared pool whose other occupants you do not control.
5AWhere the cliff is, and how to make it reproduce on any machinemeasured · blocking tasks on the cooperative pool
The card above states the contract. Here is the number. Task.detached jobs, each blocking on a semaphore, three runs per point, on a machine whose activeProcessorCount is 14:
blocking jobs 10 11 12 13 14 15 28 64
outcome OK OK OK OK STARVED STARVED STARVED STARVED
^ exactly the core countThirteen blocking jobs always completed. Fourteen always stalled, with CPU near zero and nothing to see in a profiler. Pre-warming the pool with CPU-bound work first does not move the cliff: the pool grows to meet runnable demand, and a thread parked in the kernel is invisible to that decision. measured here inference on the mechanism.
Three things follow, and the third is the one that saves an afternoon.
1 · Make it deterministic. LIBDISPATCH_COOPERATIVE_POOL_STRICT=1 pins the pool to width one, so a single blocking task starves — on any machine, whatever its core count. measured here 64 non-suspending 200 ms tasks finish in 1,000 ms with effective parallelism 12.8 normally, and take 12,800 ms with parallelism 1.0 under the strict pool. Run async test suites under it and a forward-progress violation stops being a machine-dependent flake.
2 · “It compiles under strict concurrency” is not evidence here. Swift 6 rejects the obvious spelling: sem.wait() written directly in an async function is “unavailable from asynchronous contexts; Await a Task handle instead”. One non-async frame between the task and the blocking call hides it completely — which is exactly how it ships. measured here
3 · A second, different bug wears the same costume. Task { … } written inside @main’s static func main() async inherits MainActor isolation, so those tasks serialise on one actor rather than spreading across the pool. The symptom is identical — work stops finishing — but it stalls at one blocking job rather than fourteen, warmed or not. The discriminator is this section’s own question: which executor is this isolated to? Task.detached does not inherit isolation, so if the threshold jumps to the core count when you switch, it was isolation and not the pool.
The repair, and its own price. Keep the blocking, change where it happens: hand the synchronous work to a Dispatch queue and suspend the task with withCheckedContinuation. Measured here the repaired build completes at 14, 56 and 200 jobs — and used 41 kernel threads at 200 jobs. The fix converts a hard stall into real Dispatch thread growth, so it needs a concurrency bound of its own. A repair whose cost you cannot state is half an answer.
measured here Reproduced in this session: 11 of 11 assertions in the fixture, including the below-threshold control and the strict-pool case. version-sensitive The threshold is a property of this Swift runtime; what transfers is that a threshold exists and is near the core count.
A negative result worth keeping. Timing a Task.yield() round trip under this load showed no starvation signal at all. The defect is a forward-progress failure, not a latency one, so percentile measurements of an already-running task will not find it. Measure completion, not latency.
The threading half of this chapter covers the Dispatch and Swift-concurrency layer in depth — queues versus tasks, actor reentrancy, thread explosion, the cooperative pool’s width, and the runtime deadlock that follows from blocking it. See 6 · Queues vs tasks and, in chapter 7, 1 · Failure catalogue. This section exists only to place those mechanisms against the kernel’s.
6 · Queues vs tasks
Three ways to say “concurrent”.
Raw threads, Dispatch queues, and Swift concurrency are not three flavours of the same thing. They make different promises about how many threads exist, what happens when work blocks, and what the compiler can check. Interviewers probe the seam between them, because that is where real apps break.
| Model | Unit of work | What it promises | How it fails |
|---|---|---|---|
Thread (Thread, pthread) | A function on a dedicated stack. | You control it exactly: name, stack size, priority. | You also own the count. Threads are the resource that runs out. |
| Dispatch queue | A block submitted to a FIFO queue. | Serial queues serialise; concurrent queues overlap. Threads come from a system pool. | Blocking inside a concurrent queue causes the pool to grow. Apple: “If too many tasks block, the system may run out of threads for your app.” |
| Swift task | An async function body with suspension points. | Structured lifetime, cancellation propagation, compiler-checked isolation. | Blocking a cooperative-pool thread breaks the runtime contract the pool is sized around. |
Sources: DispatchQueue — Avoiding Excessive Thread Creation; WWDC21 · Swift concurrency: Behind the scenes.
Simplified diagram. Thread counts are illustrative, not measured. The behaviour it depicts is Apple’s stated design: the cooperative pool “will only spawn as many threads as there are CPU cores, thereby making sure not to overcommit the system. Unlike GCD’s concurrent queues, which will spawn more threads when work items block, with Swift threads can always make forward progress.” (WWDC21 · Swift concurrency: Behind the scenes)
The rule that follows from that contract
Do not block a cooperative-pool thread. Apple’s own phrasing: “do not use primitives that create unstructured tasks and then retroactively introduce a dependency across task boundaries by using a semaphore or an unsafe primitive … This violates the runtime contract of forward progress for threads.” A DispatchSemaphore.wait() inside an async function is the canonical violation.
Queue ≠ thread
“Except for the dispatch queue representing your app’s main thread, the system makes no guarantees about which thread it uses to execute a task.” A serial queue guarantees one at a time, never the same thread each time. Thread-local storage and recursive-lock ownership both break on that distinction.
sync onto your own queue — it traps, it no longer hangs
Apple’s prose still says “Attempting to synchronously execute a work item on the main queue results in deadlock.” documented On macOS 26.3 the observed behaviour is different and better: libdispatch detects the re-entrancy and traps — EXC_BREAKPOINT/SIGTRAP, exit 133, with BUG IN CLIENT OF LIBDISPATCH: dispatch_sync called on queue already owned by current thread in the crash report. measured here Name the divergence in an interview rather than picking a side: a candidate who says “it deadlocks” will not connect that crash string to their own answer.
Many serial queues are worse than many concurrent ones
Apple warns that “each dispatch queue consumes thread resources” documented and that is the obvious half. The non-obvious half: 256 blocking items on one concurrent queue peaked at 71 threads — the constrained cap — while 200 blocking items on 200 separate serial queues peaked at 201, because a serial queue that needs a thread gets an overcommit thread that bypasses that cap entirely. measured here
Interop is allowed and normal
DispatchSerialQueue conforms to both SerialExecutor and TaskExecutor, so an actor can be pinned to an existing queue during a migration. Apple documents this as the escape hatch for “specific thread” requirements when interoperating with non-Swift runtimes.
Source: SerialExecutor.
| Queue shape, all work blocking | Peak live threads | What the ceiling is |
|---|---|---|
| 16 items, one private concurrent queue | 17 | Pool grows 1:1 with blocked work |
| 64 items, one private concurrent queue | 65 | Still 1:1 |
| 256 items, one private concurrent queue | 71 | 1 main + 70 = kern.wq_max_constrained_threads |
256 items, the global .utility queue | 71 | Identical ceiling |
| 200 items, 200 separate serial queues | 201 | Each asks for an overcommit thread — not subject to the constrained cap |
| 600 items, 600 separate serial queues | 513 | 1 main + 512 = kern.wq_max_threads |
| 200 serial queues targeted at a private concurrent root | 71 | One line of code puts them back under the cap |
| 200 serial queues targeted at one private serial root | 2 | One worker does all the work |
measured here Live thread counts read with Mach task_threads(); ceilings cross-checked against sysctl kern.wq_max_threads kern.wq_max_constrained_threads, which report 512 and 70 on this machine. inference The mapping from those sysctl names to these ceilings is ours — the numbers match exactly, but no Apple text states the mapping. The repair is Apple’s own: “For serial tasks, set the target of your serial queue to one of the global concurrent queues.”
6ABarriers work on exactly one kind of queueand fail silently on the others
The documented scope: “The queue you specify should be a concurrent queue that you create yourself … If the queue you pass to this function is a serial queue or one of the global concurrent queues, this function behaves like the dispatch_async function.” documented (dispatch_barrier_async) There is no warning, no assertion, and no crash — the flag is simply ignored.
What that looks like when measured. 200 plain items and 200 .barrier items submitted to each queue kind, recording the peak number of barrier blocks running at once. A honoured barrier can never exceed 1:
private concurrent queue peak concurrent barrier blocks = 1 -> barrier HONOURED
global concurrent queue (.default) peak concurrent barrier blocks = 38 -> barrier IGNORED
global concurrent queue (.utility) peak concurrent barrier blocks = 40 -> barrier IGNORED
serial queue peak concurrent barrier blocks = 1 -> barrier HONOUREDRead the last row carefully. It is not evidence the barrier worked — the documentation says it degrades on a serial queue too. The serialisation comes from the queue being serial. That distinction is exactly what an interviewer probes when you claim the barrier is doing the work.
A second lesson for free: those 38–40 blocks each slept 200 µs, and the global queue grew to 38–40 threads to cover them. That is thread explosion visible in one number. measured here
Swift · a barrier that is honoured, and one that is silently ignored
import Foundation
// A barrier is honoured ONLY on a concurrent queue you created yourself.
final class RowCache: @unchecked Sendable {
// RIGHT: a private concurrent queue.
private let queue = DispatchQueue(label: "com.example.rows", attributes: .concurrent)
private var rows: [String: Int] = [:]
func value(for key: String) -> Int? {
queue.sync { rows[key] } // readers overlap
}
func set(_ value: Int, for key: String) {
queue.async(flags: .barrier) { self.rows[key] = value } // writer runs alone
}
}
// WRONG: `.barrier` on a global queue is documented to degrade to a plain async.
// Measured on macOS 26.3: 38-40 "barrier" blocks ran simultaneously.
func brokenBarrier(_ work: @escaping @Sendable () -> Void) {
DispatchQueue.global(qos: .default).async(flags: .barrier, execute: work)
}7 · Actors
Isolation the compiler can check.
An actor is a serialised region of mutable state with a compiler-enforced boundary. The interview value is not “actors are the new locks” — it is that you can say precisely what an actor guarantees, what it does not guarantee, and which of your invariants survive the difference.
Simplified diagram. “Mailbox” is a teaching label for the actor’s pending jobs, not a public API. The reentrancy behaviour it shows is documented: code between suspension points “runs sequentially, without the possibility of interruption from other concurrent code,” and await marks where that protection ends.
Isolation, in Apple’s words
“Swift guarantees that only code running on an actor can access that actor’s local state. This guarantee is known as actor isolation.” Reads from outside are asynchronous because they must first run on the actor.
Reentrancy is deliberate
Apple designed it in so priority can work: “Actors are designed to allow the system to prioritize work well due to the notion of reentrancy,” in contrast to a serial queue’s “strict first-in, first-out” order, where five low-priority items still have to finish before a high-priority one starts.
Actors are non-blocking
When an actor suspends, “the thread it was executing on is now freed up to do other work.” A contended lock parks a thread; a contended actor parks a task. That is the single biggest behavioural difference to state in an interview.
@MainActor is a global actor
A singleton actor whose executor is the main thread. Because there is only one instance, the type alone identifies it, so the isolation can be spelled as an attribute. Marking a class @MainActor is how you make “UI thread only” a compile error instead of a Main Thread Checker warning.
Swift · an actor that also de-duplicates in-flight work
import Foundation
actor ThumbnailCache {
private var storage: [URL: Data] = [:]
private var inFlight: [URL: Task<Data, Error>] = [:]
func thumbnail(for url: URL, make: @Sendable @escaping (URL) async throws -> Data) async throws -> Data {
if let cached = storage[url] { return cached }
if let running = inFlight[url] { return try await running.value }
let task = Task { try await make(url) }
inFlight[url] = task
defer { inFlight[url] = nil } // runs after the await below resumes
let data = try await task.value // suspension point: other calls interleave here
storage[url] = data
return data
}
}The follow-up that separates levels
“Your actor has an await in the middle of thumbnail(for:). Between the check and the store, another task can run on that actor. Why is this code still correct?” The answer is the inFlight table: the invariant is not “only one task is inside this function”, it is “at most one make runs per URL”, and that invariant is established before the suspension point and consulted by everyone who arrives during it.
The counterweight
Compiles clean. Sanitizer clean. Still wrong.
This is the single most useful artifact in the chapter for the rule do not claim actors eliminate races. Swift 6 strict concurrency passes it, Thread Sanitizer passes it, and the program is catastrophically incorrect.
Simplified diagram. The actor honoured every guarantee it makes: one body executed at a time, no torn reads, no lost updates, no data race. measured here Six concurrent withdrawals of 100 against a balance of 100: all six succeeded and the balance reached −500, reproduced 5 of 5 runs, with Thread Sanitizer reporting nothing at all. Apple’s own framing: “it eliminates low-level data races, which involve data corruption. You still need to reason about atomicity at a high level … you can end up with a high-level data race where the program is in an unexpected state, even though no data is actually corrupted.” documented (WWDC22 110351 · Eliminate data races using Swift Concurrency, 2022, 19:03–19:49)
The one sentence to quote
From Swift’s own actor proposal: “the easiest way to avoid breaking invariants across an await is to encapsulate state updates in synchronous actor functions. Effectively, synchronous code in an actor provides a critical section, whereas an await interrupts a critical section.” documented (SE-0306 · Actors, 2021) The same proposal is careful where people are not: reentrancy “all but eliminates the potential for deadlocks” — note the “all but” — and reentrant actors “are thread-safe but are not automatically protecting from the ‘high level’ kinds of races.”
Swift · the broken actor and two repairs, in preference order (deliberately-wrong code, clearly marked)
import Foundation
// DELIBERATELY WRONG. Compiles under Swift 6 strict concurrency with zero
// warnings, has no data race, and Thread Sanitizer reports nothing - and it
// still lets the account go overdrawn.
actor BrokenAccount {
private(set) var balance: Int
init(balance: Int) { self.balance = balance }
func withdraw(_ amount: Int) async -> Bool {
guard balance >= amount else { return false } // 1. CHECK - true right now
await auditLog(amount) // 2. SUSPEND - actor RELEASED here
balance -= amount // 3. ACT - on a stale decision
return true
}
}
// REPAIR 1 - re-check after every suspension. Cheapest; the suspended work still ran.
actor RecheckAccount {
private(set) var balance: Int
init(balance: Int) { self.balance = balance }
func withdraw(_ amount: Int) async -> Bool {
guard balance >= amount else { return false }
await auditLog(amount)
guard balance >= amount else { return false } // <-- the repair
balance -= amount
return true
}
}
// REPAIR 2 - commit the resource BEFORE releasing the actor, compensate on failure.
actor ReserveAccount {
private(set) var balance: Int
private(set) var refunds = 0
init(balance: Int) { self.balance = balance }
func withdraw(_ amount: Int) async -> Bool {
guard balance >= amount else { return false }
balance -= amount // committed before any await
await auditLog(amount)
if Task.isCancelled { balance += amount; refunds += 1; return false }
return true
}
}
func auditLog(_ amount: Int) async { try? await Task.sleep(for: .milliseconds(120)) }Type-checked with swiftc -typecheck -swift-version 6: 0 errors, 0 warnings. measured here Both repairs bring the six-caller run to exactly one success and a final balance of 0. A third repair exists — an explicit in-flight flag with a continuation queue — and it works (measured maxInFlight=1), but it reintroduces the queueing, ordering and starvation questions that reentrancy exists to avoid, so reach for it only when the first two do not fit.
Why reentrancy exists at all — the answer to “why not just make actors FIFO?”
Because strict FIFO creates priority inversion. Apple: “Dispatch queues execute the items received in a strict first-in, first-out order. Unfortunately, this means that after item A has executed five low-priority items need to execute before getting to the next high-priority item. This is called a priority inversion. Serial queues work around priority inversion by boosting the priority of all of the work in the queue that’s ahead of the high-priority work … However, it does not resolve the main issue … Solving this issue requires changing the semantic model away from strict first-in, first-out.” documented (WWDC21 10254 · Swift concurrency: Behind the scenes, 2021) inference So reentrancy is the price paid for making priority work across the isolation boundary — stating that trade is what separates design understanding from API recall.
Escape hatch 1 · nonisolated
Not an escape from safety — the compiler still checks what it touches. Apple: “Nonisolated code is very flexible, because you can call it from anywhere: if you call it from the main actor, it will stay on the main actor. If you call it from a background thread, it will stay on a background thread.”
documented WWDC25 268 · Embracing Swift concurrency (2025), 13:56–14:09.
Escape hatch 2 · @unchecked Sendable
“These types are conceptually Sendable, but there is no way for Swift to reason about that. Use unchecked Sendable to disable the compiler’s checking. Be careful with this, because smuggling mutable state through @unchecked Sendable undermines the data race safety guarantees Swift is providing.”
documented WWDC22 110351 (2022), 07:53–08:16.
Escape hatch 3 · nonisolated(unsafe)
“Like other uses of the word ‘unsafe’ in Swift, this puts the burden on you to ensure safety for this variable. This should be a last resort.”
documented WWDC24 10169 · Migrate your app to Swift 6 (2024), 14:45–15:16.
The interview framing for all three
Each moves a proof obligation from the compiler to you. Name which obligation — “I am asserting that every access to this field goes through self.lock” — and name how you would check it. inference Saying “I used @unchecked Sendable” without naming the obligation is the answer that fails.
8 · Priority, donation & cancellation
Urgency you declare, cancellation you honour.
QoS is not a speed dial. It is a declaration of intent that the system uses to make scheduling and energy tradeoffs, and — crucially — to decide whose priority to raise when a dependency is visible to it. Cancellation, symmetrically, is not a kill: Swift’s model is cooperative, so a task that never checks is a task that never stops.
| QoS class | Means | Typical UI-frameworks use | What goes wrong |
|---|---|---|---|
.userInteractive | The user is waiting on this frame. | Main thread, hit testing, layout, drawing on the display path. | Used as a default. Everything urgent means nothing is. |
.userInitiated | The user asked and is waiting for a result. | Opening a document, running a search the user typed. | Work that outlives the interaction keeps a high-priority thread busy. |
.utility | Long-running, progress is visible. | Thumbnail generation, indexing, export. | Reasonable default; forgotten when the user starts waiting on it. |
.background | Not user-visible; the system may defer it. | Cache trimming, prefetch, maintenance. | Something user-interactive waits on it → priority inversion. |
Apple frames QoS as importance and energy, not raw speed: “Because higher priority work is performed more quickly and with more resources than lower priority work, it typically requires more energy.” (DispatchQoS). The four-way split in the third and fourth columns is our teaching summary, not Apple text.
Swift raises priority for you — sometimes
Apple documents two elevations: when a higher-priority task is enqueued on a busy actor, “the actor’s current task is temporarily elevated”; and when a higher-priority task awaits a task’s value, “the priority of this task increases until the task completes.” Apple’s stated purpose for both: “priority elevation helps you prevent a low-priority task from blocking the execution of a high priority task, which is also known as priority inversion.”
Source: TaskPriority.
…and cannot, across a semaphore
Apple is explicit: with dispatch_semaphore_wait and dispatch_group_wait, “the system can’t automatically propagate priority from the higher-priority thread to the lower-priority thread.” The dependency is invisible to the runtime, so no boost happens. This is exactly what Thread Performance Checker reports.
Source: Diagnosing performance issues early.
Inheritance rules worth memorising
A child task inherits its parent’s priority. Task { } inherits “the same actor isolation, priority, and task-local state as the current task.” Task.detached inherits none of it, and Apple notes a detached task “executes only with .medium priority, by default.”
Cancellation is Boolean and cooperative
“Cancellation is a purely Boolean state; there’s no way to include additional information like the reason for cancellation.” Responding means throwing CancellationError, returning nil, or returning partial work — your choice, made per call site.
Source: Task — Task Cancellation.
Swift · three shapes of cooperative cancellation
import Foundation
import Synchronization
struct PageRenderer {
/// `checkCancellation()` throws, which unwinds the whole job.
func renderAll(_ pages: [URL]) async throws -> [Data] {
var output: [Data] = []
for page in pages {
try Task.checkCancellation()
output.append(try await render(page))
}
return output
}
/// `isCancelled` when partial work is more useful than an error.
func renderBestEffort(_ pages: [URL]) async -> [Data] {
var output: [Data] = []
for page in pages {
if Task.isCancelled { return output }
guard let data = try? await render(page) else { return output }
output.append(data)
}
return output
}
private func render(_ page: URL) async throws -> Data { Data() }
}
/// A handler fires the moment `cancel()` arrives, while the body is still running,
/// so the shared handle must itself be synchronized.
final class Downloader: Sendable {
private let session = Mutex<URLSessionDataTask?>(nil)
func data(from url: URL) async throws -> Data {
try await withTaskCancellationHandler {
try await withCheckedThrowingContinuation { continuation in
let task = URLSession.shared.dataTask(with: url) { data, _, error in
if let error { continuation.resume(throwing: error) }
else { continuation.resume(returning: data ?? Data()) }
}
session.withLock { $0 = task }
task.resume()
}
} onCancel: {
session.withLock { $0?.cancel() }
}
}
}Note the Mutex inside Downloader. Apple warns that “because the task is still running when the cancellation handler starts, avoid sharing state between the task and its cancellation handler, which could create a race condition.” Where you must share a handle, the handle itself has to be synchronised — this is a real case where Swift concurrency and a lock belong in the same type.
8AThe hang that looks like it was fixedTask inherits isolation
Interview explanation: Wrapping synchronous work in Task { } from main-actor context does not move it off the main thread. Apple’s own words: the task “inherits the context from its enclosing context … which means it can only execute on the main actor and does still block the main actor for a long amount of time. This just delays the hang until after the immediate button action finishes.” The fix is either to make the work nonisolated and async, or to detach.
Practical proof: Run the button action with the Thread Performance Checker enabled and profile with the Swift Concurrency instrument; the work shows up executing on the main actor in both the naive and the Task { } version, and only leaves it in the detached or nonisolated async version. Source: Improving app responsiveness.
Swift · the trap and its two repairs, in AppKit terms
import AppKit
@MainActor
final class DocumentWindowController {
let label = NSTextField(labelWithString: "")
// WRONG: this Task inherits @MainActor, so the synchronous work still blocks the main thread.
func reindexIncorrectly() {
Task {
let summary = Self.buildIndexSynchronously() // hangs the main actor
label.stringValue = summary
}
}
// RIGHT (a): make the heavy work nonisolated and async; only the UI hop returns to the main actor.
func reindexWithAsyncWork() {
Task {
let summary = await Self.buildIndex()
label.stringValue = summary
}
}
// RIGHT (b): if it must stay synchronous, detach so it cannot inherit the main actor.
func reindexDetached() {
Task.detached(priority: .utility) {
let summary = Self.buildIndexSynchronously()
await MainActor.run { self.label.stringValue = summary }
}
}
nonisolated static func buildIndexSynchronously() -> String { "indexed" }
nonisolated static func buildIndex() async -> String { "indexed" }
}The rule behind every donation answer
Ownership decides whether inversion can be fixed.
Not speed, not fairness, not recursion. The first question about any primitive is whether the runtime knows which single thread will release it — because that is the only thread there is to raise.
Simplified diagram. The three-way taxonomy is Apple’s own: “Symmetric primitives with a single known owner can do that, like pthread_mutex_t or the most efficient, os_unfair_lock. Asymmetric primitives like pthread conditional variables or dispatch_semaphore don’t have this ability, because the runtime doesn’t know which thread will signal it. Keep this feature in mind when choosing a synchronization primitive, and favor symmetric primitives for mutually exclusive access.” documented (Tech Talk 110147 · Tune CPU job scheduling for Apple silicon games, 2022 — the page exposes no per-sentence timestamp anchors, so this is cited to the session)
| Primitive under test | Holder priority, no waiter | Holder priority while a .userInteractive thread waits | Same critical section took |
|---|---|---|---|
os_unfair_lock | 4 — 100% on E cores | 31 for 90% of its time — 90% on P cores | 277 ms |
pthread_mutex | 4 — 100% on E cores | 31 for 90% — 91% on P cores | 271 ms |
DispatchSemaphore(1) | 4 — 100% on E cores | 4 — still 100% on E cores, unchanged | 1017 ms |
measured here Two independent measurements agree. Reading the holder’s current scheduling priority directly through thread_info(mach_thread_self(), THREAD_EXTENDED_INFO, …).pth_curpri gives 4 → 31 for both locks on 3 of 3 trials and 4 → 4 for the semaphore on 0 of 3 — re-run and reproduced on 2026-09-22. Recording the same shape under System Trace gives the priority-band and core-placement percentages above. The same critical section took 277 ms boosted and 1017 ms unboosted — a 3.7× longer hold, every millisecond of it with the user-interactive thread blocked. That is priority inversion with a price tag, and the price is paid entirely by choosing an owner-less primitive.
The mechanism, in one sentence
A lock records who owns it, so the kernel has a thread to raise. A semaphore is a bare counter with no owner — there is no thread to raise, so a high-priority waiter simply waits at the low-priority holder’s pace. inference The trace never uses the word “donation”; the boost plus its absence in the no-waiter control is what makes the reading strong.
A false lead worth not repeating
pthread_get_qos_class_np() reports the QoS a thread requested, not its effective priority. Under full donation it still returned QOS_CLASS_BACKGROUND while pth_curpri read 31. negative result Measuring donation with that API produces a confident, wrong “no donation here” conclusion.
pthread_join donates too
A joining thread donates its priority to the thread it joins — a real dependency the runtime can see, exactly like lock ownership. measured here inference No Apple text states it, but it gives a concrete second reason for Apple’s advice: “Don’t synchronize the main thread with a background thread, or make the main thread join a background thread.” An experiment that joins the thread under test silently invalidates itself.
Temper the textbook story
The classic “high-priority thread starves indefinitely behind a low-priority holder” does not reproduce on modern Darwin with a donating lock — an attempt to show it measured only a 1.5× penalty, because donation works. negative result The residual risk lives with primitives that cannot donate, which is precisely why OSSpinLock was deprecated in favour of os_unfair_lock.
9 · Preemption
Oversubscription costs latency, and the receipt is involuntary switches.
The kernel takes a core away from a running thread when its turn ends or when something more urgent becomes runnable. That is preemption, it is not optional, and its cost is measurable directly.
9AThroughput is flat; latency is notmeasured · fixed total work, 1× to 4× the core count
threads wall_ms involuntary switches ms per unit of parallelism
14 185.2 2344 185.2
28 359.1 4474 179.6
42 541.0 7564 180.3
56 704.5 9597 176.1Two readings, and both matter. Wall time rises almost exactly linearly with thread count, which is expected: each thread does a fixed amount of work, so four times the threads is four times the work on the same fourteen cores. Normalised for that, the last column is flat — 185, 180, 180, 176 ms. Oversubscribing a purely CPU-bound workload did not cost measurable throughput.
What it did cost is visible in the third column. Involuntary context switches rose from 2,344 to 9,597 — roughly linearly with the oversubscription factor. Every one of those is a thread that was running and had its core taken away, and every one of those is latency for whoever was waiting on that thread.
The interview form of this result: oversubscription is not primarily a throughput problem on a modern scheduler; it is a latency and energy problem. If someone argues “we measured it and throughput was fine”, they measured the wrong thing — and section 11’s tail measurement is the right one.
The same result, through a second instrument. The switch counter above is the cheap view. Section 2 shows what those switches look like from a trace: at 2× oversubscription, more than half of all thread-time sat in Preempted. And the threading half of this chapter reaches it a third way, measuring preemption share rise from 19.7% to 71.0% over the same 1×–4× sweep with Instruments’ own System Trace — see 3 · Scheduling & cores. Three independent instruments, one conclusion; that agreement is the evidence, not any one number.
measured here Apple M4 Pro (10P + 4E), macOS 26.3, getrusage(RUSAGE_SELF). Voluntary context switches (ru_nvcsw) reported 0 throughout, in this and every other fixture — see the negative result below.
NEGATIVE RESULT · A METRIC THAT DOES NOT WORK ON THIS PLATFORM. getrusage(2)’s ru_nvcsw field — voluntary context switches, the count that should rise when threads block — reported 0 on macOS 26.3 in every fixture measured here, including one whose threads block on a condition variable thousands of times. ru_nivcsw (involuntary) is populated and is used above. So do not try to demonstrate “this version blocks instead of spinning” with ru_nvcsw; measure CPU time instead, which shows the same thing unambiguously. negative result
The tick, for scale
measured here sysctl kern.clockrate reports hz = 100, tick = 10000 on macOS 26.3 — a 10 ms statistics tick. inference That is not the scheduling quantum and must not be quoted as one; it bounds the resolution of tick-based accounting, which is one reason sampling profilers and tick counters disagree with high-resolution timers.
Timers are coalesced, on purpose
measured here kern.timer.coalescing_enabled is 1, with per-tier scaling sysctls. The practical consequence measured here: a usleep-based 5 ms heartbeat was consistently about 1 ms late at the median regardless of system load, so a timer-wakeup measurement could not distinguish a contended machine from an idle one. If you need to measure scheduling latency, measure work, not wakeups.
Why more threads rarely helps
Once every core is busy, an extra runnable thread cannot add throughput; it can only add switches. The exception is a workload that blocks — then extra threads cover the blocked ones. “Are my threads blocking?” is therefore the question that decides whether a bigger pool is a fix or a regression.
The energy argument
A preemption is cache-hostile: the incoming thread finds the caches full of somebody else’s data. Oversubscription therefore costs energy even when wall time looks unchanged, which on a laptop is a user-visible outcome even though no timer shows it.
10 · Inversion and donation
Urgency has to travel across the wait.
When an urgent thread waits for something a less urgent thread holds, the urgent thread now runs at the holder’s speed. The kernel can fix this — but only when it knows who to boost, and that knowledge is a property of the primitive you chose.
The mechanism, in one sentence
A primitive that records an owner lets the kernel find a specific thread to raise. A primitive that records only a count or a condition has nobody to raise, so the waiter simply waits.
Which primitives can donate
Ownership-recording locks — os_unfair_lock, pthread_mutex_t, NSLock, and Mutex built on them — plus a synchronous dispatch onto a queue, and pthread_join. Counting and signalling primitives cannot: a semaphore’s permits and a condition variable’s predicate name no thread.
The rule that follows
Do not express “wait for this work” with a semaphore or a condition when a lock or a synchronous dispatch would express the same thing. You are not choosing between two equivalent spellings; you are choosing whether the kernel is able to help you.
Inversion without a lock
The chapter’s exercise 04 is an inversion with no lock anywhere: a latency-sensitive thread is simply outnumbered on the run queue by undeclared CPU-bound work. Nothing is owned, so nothing can be donated, and the only repair is to change the band the other work runs in.
Two shapes people call “priority inversion”, and only one has a donation fix
Shape one — bounded, ownership-based. High-priority thread H waits on a lock held by low-priority thread L, and medium-priority M keeps preempting L. Classic. Donation raises L to H’s priority for the duration of the hold, L finishes, H proceeds. The fix is structural: use a primitive with an owner, and keep the hold short.
Shape two — unbounded, load-based. H is not waiting on anything at all. It is simply runnable behind a great deal of equally-ranked work. There is no owner, so there is nothing to donate to, and no lock discipline will help. The fix is to rank the competing work correctly, which is a QoS decision, or to reduce it.
Naming which shape you have is the first move, and the discriminator is simple: is the slow thread blocked, or merely runnable? If it is blocked, find the owner. If it is runnable, find the competition. inference
The threading half of this chapter covers donation across Dispatch and Swift concurrency in detail — including Apple’s statement that a semaphore wait cannot carry priority, the pthread_get_qos_class_np false lead, and where the textbook “indefinite starvation” story overstates the case. See 8 · Priority & donation.
11 · Diagnosis
Measure the tail, and measure work rather than wakeups.
Two methodological rules do most of the work here, and both were forced by measurements that failed before they succeeded.
Rule one · the mean will exonerate the bug
measured here Under 56 undeclared CPU-bound threads on 14 cores, a latency-sensitive thread’s work unit showed:
p50 inflation p99 inflation max inflation
undeclared QoS 1.02 79.47 100.32
QOS_CLASS_BACKGROUND 1.04 2.66 5.37The median moved by 2%. Any profiler reporting a mean, a total, or “time spent in function” sees nothing. The p99 moved by a factor of 79, which at 60 Hz is a dropped frame about once a second — precisely what a user calls “juddering”.
When a user says judder, hitch, stutter, or “occasionally”, fix the measurement before the code: percentiles, a histogram, or a worst case. A mean is the wrong instrument for a tail problem, and reporting one is how a real defect gets closed as not-reproducible.
Rule two · measure work, not wakeups
negative result The first version of that experiment measured how late a usleep-driven 5 ms heartbeat woke. It showed nothing: about 1 ms of median lateness with and without the load, because timer coalescing put a floor under the measurement well above the effect being looked for. Only the maximum moved.
Rewriting it to time a fixed unit of CPU work on the latency-sensitive thread — and to calibrate against the same machine’s idle baseline moments earlier — produced the 79× signal above. The defect never changed; the instrument did. That is worth carrying into an interview: “my first measurement showed nothing, and here is why it was the wrong measurement” is a stronger answer than a clean number with no story.
| Symptom | First discriminator | If it is “busy” | If it is “waiting” |
|---|---|---|---|
| App stops responding | sample <pid> during the freeze | CPU Profiler; find the hot frame | Read the blocking frame; find the owner |
| Occasional hitch, mean is fine | Percentiles of a fixed work unit, against an idle baseline | Tail is inflated → competition or QoS | Tail is inflated → an intermittent wait |
| Fans spin with no visible work | CPU seconds per wall second | > 1 core held while idle → a spin | ≈ 0 → look elsewhere |
| Adding threads made it slower | Involuntary switches, and whether threads block | Switches up, throughput flat → oversubscription | Contention → see chapter 6 |
| Async work never finishes | Is a task blocking a pool thread? | — | Pool exhausted → remove the block |
System Trace is the picture of this chapter
Its thread-state lanes show each thread’s state as bands over time, next to every other thread. That is the one view in which “ready but not running” is directly visible rather than inferred — a gap in your thread’s lane while dozens of other lanes are solid. Apple demonstrates the instrument and its Narrative view, which reports why a thread was blocked and the exact syscall backtrace that blocked it.
documented WWDC23 10248 · Analyze hangs with Instruments, 37:07–38:22. https://developer.apple.com/videos/play/wwdc2023/10248/ measured here The state names are not Apple’s wording — that session never says “runnable”. The eight-value vocabulary in section 2 comes from this page’s own headless export.
Prefer CPU Profiler to Time Profiler
Apple’s reason matters here: “using a timer to sample call stacks suffers from a problem called aliasing … when some periodic work on the system happens at the same cadence as the sampling timer”, so the wrong functions become over-represented. “You should prefer CPU Profiler over Time Profiler for CPU optimization because it’s more accurate and more fairly weights software consuming CPU resources.” Periodic work at a fixed cadence is exactly the shape a scheduling problem has.
documented WWDC25 308 · Optimize CPU performance with Instruments, 9:37–10:17. https://developer.apple.com/videos/play/wwdc2025/308/
Unprivileged, and enough
sample <pid> needs no elevation and separates busy from blocked immediately. getrusage in-process gives CPU seconds and involuntary switches. thread_info gives live per-thread state. All three are available without Instruments, and all three were the basis for this chapter’s numbers.
Ship the measurement
A latency-sensitive loop can record its own percentiles cheaply with an in-process histogram and a signpost. A tail regression that only appears on a customer’s loaded machine will never be found by a profiler you are not running; it will be found by a number your app already collects.
https://developer.apple.com/documentation/os/ossignposter
What this chapter did not do. No Instruments window was opened and no screenshot appears anywhere — but the traces behind sections 2 and 4 are real Instruments recordings, made with xctrace and exported as XML (section 11A), so thread states, core clusters and scheduler priorities here are observations rather than quotations. What remains quoted rather than observed is the user interface: System Trace’s lane rendering and the Narrative view’s presentation. Nothing here used dtrace or any privileged tracing; spindump was only ever run far enough to record that it refuses without root. No real-time thread (THREAD_TIME_CONSTRAINT_POLICY) was created or measured, so this chapter makes no claim about hard-deadline scheduling beyond naming it as out of scope.
11A · Instruments without the window
Real traces from a terminal, and where privilege actually stops you.
“That needs Instruments, and I do not have a GUI” is the most common reason a scheduling question gets answered from memory instead of from evidence. It is also wrong. xctrace is the command-line front end to the same instruments: it records, it exports XML, it needs no elevation, and it opens no window. Every thread-state and core-placement figure in this chapter came through it.
$ xcrun xctrace version
xctrace version 26.0 (17C529)
# record: launches the target and stops when it exits
$ xcrun xctrace record --template 'System Trace' \
--output run.trace --launch -- ./yourprog
# what tables does the recording contain?
$ xcrun xctrace export --input run.trace --toc
# pull one table out as XML
$ xcrun xctrace export --input run.trace \
--xpath '/trace-toc/run[@number="1"]/data/table[@schema="thread-state"]' \
--output ts.xmlWhat a row carries
A thread-state row names the thread, the state, the duration, the core and its cluster (CPU 5 (P Core)), the scheduler priority, and a narrative sentence. A cpu-profile row adds a full backtrace with source lines. Everything this chapter asserts about bands and placement is read off those two columns.
measured here
Templates that matter here
System Trace for thread states, context switches and syscalls; CPU Profiler for per-core sampled backtraces; Swift Concurrency, File Activity, Allocations, Leaks, Logging, Network and Tailspin are all present too. xctrace list templates is the authoritative list on your machine.
measured here
The export is large, and that is fine
A six-second System Trace of 28 threads exported a 60 MB thread-state table. Aggregate it with a twenty-line script rather than reading it; the value is that the aggregation is reproducible and can go in a test, which a screenshot cannot.
measured here inference on the testability argument.
What it still is not
It is a recorder and an exporter, not the app. There is no flame graph, no lane rendering, no click-through to source. For “show me the shape” the window is still better; for “give me the number, in CI, on a machine with no display”, this is the only option.
inference
The privilege boundary, measured rather than assumed
$ spindump <pid> 1 -o /tmp/sd.txt
spindump must be run as root when sampling the live system
$ sample <pid> 1 -file /tmp/s.txt
Sampling process 15739 for 1 second with 1 millisecond of run time between samples
Sampling completed, processing symbols...
$ fs_usage -w -f filesys
'fs_usage' must be run as root...The rule to carry: sample is the unprivileged one and is usually enough — it names threads and separates busy from blocked immediately. spindump is the whole-system one and refuses without root even for a single named process. xctrace sits between them: real trace data, no elevation. measured here
11BThe Thread Performance Checker, and the four defects it will not finda tripwire, not coverage
Apple: “The Thread Performance Checker tool detects priority inversions and non-UI work on the main thread. It doesn’t require any recompilation.” And, in the same document: “The Thread Performance Checker tool is currently supported only on macOS and iOS. Resolution of certain performance issues may require significant code refactoring or redesign of the underlying logic.” documented https://developer.apple.com/documentation/xcode/diagnosing-performance-issues-early
Two detectors, not a profiler. Priority inversion, and non-UI work on the main thread. Line those up against what this chapter is about and the gap is the answer: it will not report oversubscription, efficiency-core placement, a wrong QoS band, or cooperative-pool starvation — none of the four defects in sections 2 to 5. inference from the documented detector list.
It runs under Xcode, surfacing issues in the Issue navigator and the source editor. It is not a command-line tool, and in tests its findings are warnings by default — they only fail the test after Runtime API Checking is set to “On (as Failure)” in the test plan. A team that enabled it and never changed that setting has a tripwire that never trips the build. documented
And it comes with a direct Apple citation for section 10. Apple’s own precaution list names the mechanism: “Don’t use dispatch_semaphore_wait and dispatch_group_wait to emulate synchronous behavior … the system can’t automatically propagate priority from the higher-priority thread to the lower-priority thread.” That is the documented statement of the rule section 10 measures: a semaphore has no owner, so there is nobody to donate to. documented
Not exercised here. The Thread Performance Checker itself was never run — it needs Xcode, and no GUI session was opened for this page. Everything in this disclosure is Apple’s text plus the inference about what its stated detectors cannot cover.
12 · Fixing exercises
Two broken programs. Diagnose, repair, prove.
Exercises 03 and 04 of the same bundle. Neither one deadlocks, races, or leaks; both produce completely correct output. That is the point — these are the scheduling defects that survive code review and profiling.
Same bundle as the heap chapter
os-memory-io-ipc-exercises.tar.gz — 44 files, 55,129 bytes. SHA-256 0a04e5544c047fc5376919d91fdcf5943a1c1316aa38996d93bac105b23ba13f · raw path labs/os-memory-io-ipc-exercises.tar.gz
tar xzf os-memory-io-ipc-exercises.tar.gz
cd os-memory-io-ipc-exercises
./run-all.sh 03 0403 · The idle pipeline that drains the battery
An ingest pipeline takes frames from a capture device at 200 frames per second and hashes each one on a pool of worker threads, one per core. The field reports fans spinning up and the battery draining even when almost nothing is arriving, with several cores pinned near 100%. Every frame is processed exactly once and the checksum is right. Find the cost, remove it, and tell me the one case where the original shape would have been the better choice.
clang -O2 -g -Wall -Wextra -pthread broken/pipeline.c -o /tmp/pipeline_broken
/tmp/pipeline_brokenExpected signal. cpuPerWallCore=12.64 — the equivalent of 12.6 cores held continuously to process 200 small frames — with 23,437 involuntary context switches in 1.4 seconds and zero useful work in any of them. The fixed build reports cpuSeconds=0.003, cpuPerWallCore=0.00, 433 involuntary switches, an identical checksum, and a slightly shorter wall time. measured here
Success criterion. cpuPerWallCore at most 0.25, every frame still processed, checksum unchanged, holding on two consecutive runs. Your wait sits inside a while loop testing the predicate, and you can say why an if would be a bug even on a platform that never spuriously wakes. You can name the situation in which spinning is right, with a rough figure for the crossover.
Progressive hints
- What does the worker do when there is no work? Read the inner loop and ask what instruction it executes when nothing has been published. Then ask what the scheduler is supposed to do with a thread that is always ready to run.
- You need two things, not one. Leaving the CPU means asking the kernel to stop scheduling you until a condition becomes true. That needs a way to announce the change and a way to test it without racing the announcement. One primitive does not give you both; a matched pair does.
- The predicate and the lock must be the same lock. Otherwise you can test “no work”, have work published, then go to sleep, and never wake. That is a lost wakeup, and it is much harder to debug than what you started with.
Solution
Block instead of spinning, with the predicate guarded by the mutex the condition variable releases:
pthread_mutex_lock(&m);
while (claimed >= published && !shutting_down)
pthread_cond_wait(&work, &m); /* releases m and parks in the kernel */
mine = ++claimed;
pthread_mutex_unlock(&m);
/* producer */
pthread_mutex_lock(&m);
published = f;
pthread_cond_signal(&work); /* one frame, one waiter */
pthread_mutex_unlock(&m);Three details that are not optional. while, not if — a wakeup is a hint that the predicate may hold, not a promise. The mutex must guard the predicate, because the condition variable’s atomicity guarantee is defined in terms of that mutex; this is why published and claimed stopped being free-standing atomics. And signal for one item, broadcast for shutdown — one frame can only be consumed once, so waking all of them is a thundering herd.
What it costs. A mutex acquisition per frame and a kernel round trip per wakeup, a few microseconds each, against 5,000 µs of idle time per frame. Here it is free, and the wall time improved because fourteen spinning threads had been competing with the producer for cores.
When spinning is right. When the expected wait is shorter than the cost of parking and waking — roughly a few microseconds on this machine, where a kernel-mediated round trip measures about 4.5 µs. That is why os_unfair_lock and friends spin briefly and then fall back to blocking: they are betting on a short critical section and they have a plan for losing the bet. An unbounded spin with no fallback, waiting on work that arrives every 5,000 µs, is the bet nobody should take. inference
04 · The meter that judders while the mean stays perfect
A media app refreshes a level meter on a latency-sensitive thread: a small fixed unit of work that must finish before the next frame is due. A library re-indexes the user’s collection on a pool of CPU-bound threads. QA reports juddering — and reports that they profiled it, the meter’s average frame cost is completely normal, nothing is blocked, and there is no lock anywhere near it. Find the defect, fix it, and tell me exactly what your fix costs.
clang -O2 -g -Wall -Wextra -pthread broken/heartbeat.c -o /tmp/hb_broken
/tmp/hb_brokenExpected signal. p50Inflation=1.02 and p99Inflation=79.47. The fixture measures itself twice — once with the pool idle to find this machine’s floor, once under load — so every number is a ratio against its own baseline rather than a figure from another laptop. The fixed build reports p99Inflation around 2.7, and hogBatches falling from 35,540 to 1,554. measured here
Success criterion. p99Inflation at most 8.0, p50Inflation still around 1 and you can explain why that is expected rather than a sign the fix did nothing, the class set on each pool thread rather than on the thread that created them, and the cost in background throughput stated with the number.
Progressive hints
- Count the threads. 56 CPU-bound threads on 14 cores, and not one of them has said anything about how important it is. What does the scheduler have to go on?
- The meter already asked. It calls
pthread_set_qos_class_self_np(QOS_CLASS_USER_INTERACTIVE, 0)before measuring anything, and it did not help. Why would it? A priority is only meaningful relative to what else is in the queue. - QoS is a property of a thread. Setting a class affects the calling thread only, and it is not inherited through
pthread_create. Where is the earliest point in each pool thread’s life at which it could declare itself?
Solution
One line, on each pool thread, as its first act:
static void *hog(void *unused) {
(void)unused;
pthread_set_qos_class_self_np(QOS_CLASS_BACKGROUND, 0);
...
}What that line does. It is not a speed dial and it does not slow the pool’s instructions down. It is a placement decision: it tells the scheduler which band this work belongs in, so the interactive thread stops taking turns against 56 peers. Apple’s description of BACKGROUND — work “not initiated by the user and that the user may be unaware of the results”, to be run “in the most energy and thermally-efficient manner” — is exactly a library re-index, which is why the pool was in the wrong band to begin with.
What it costs, stated plainly. hogBatches drops about 20×. The re-index is now much slower whenever the user is doing something else. If the re-index has a deadline of its own, demoting it is the wrong repair and the right one is to make it smaller or interruptible. Say which you are choosing and why — the fixture prints the price so you cannot report the win without it.
The repairs that do not work. Raising the meter further: already done, necessary and not sufficient. Fewer pool threads: helps, and hides the lesson — with 14 instead of 56 the tail is smaller but the pool still competes in the same band, so the defect is the band, not the count. setpriority(2)/nice: process-wide and coarse, and it does not express what the work is for, which is the vocabulary a QoS-scheduled system is built on. inference
13 · Interview questions
Fifteen questions, with the follow-up that comes next.
Answer out loud before opening each one.
ExplainWhat states can a thread be in, and why does the distinction matter?
Running on a core, runnable and waiting for one, or blocked waiting for something that is not a core. The distinction matters because each has a different cost and a different repair: running costs CPU and energy, runnable costs latency and shows up in no CPU profile, and blocked costs a stack and a scheduler slot.
The one people collapse is runnable. A thread that is ready and not scheduled is not “slow code” and is not “blocked on something”; it is losing to competition, and no amount of optimising its own instructions will help.
Follow-up: “Can the kernel tell you what a blocked thread is waiting for?” — No. Measured here, a thread sleeping on a timer, one parked on a mutex and one waiting on a read all report the same TH_STATE_WAITING. You need the stack as well as the state, which is why sample and Thread State Trace always pair them.
DiagnoseThe app judders, but the average frame time is normal and nothing is blocked. Where do you look?
At the tail, and at what else is runnable. If nothing is blocked then the thread is not waiting for a resource — it is waiting for a core, and that is a competition problem.
measured here under 56 undeclared CPU-bound threads on 14 cores, a latency-sensitive work unit’s p50 inflated by 2% while its p99 inflated by 79×. Every mean-based measurement exonerates that bug.
The repair is to rank the competing work: declare it BACKGROUND or UTILITY so it stops competing in the band the user can see. That brought p99 inflation to about 2.7×.
Follow-up: “What did that cost?” — the demoted pool completed about 20× less work in the same interval. Every QoS repair is a trade, and an answer that does not name the price is incomplete.
ExplainIs QoS a priority number?
No — it is a declaration of intent that the system maps to a scheduling band, and it is better thought of as a placement decision than a speed dial. The classes describe what the work is for: interactive, user-initiated, utility, background. Apple is explicit that DEFAULT “is not intended to be used as a work classification” — it is the absence of a decision.
It is also per-thread, not per-process, and not inherited through pthread_create. A pool must declare its class on each worker, as that worker’s first act.
Follow-up: “Is it binary — foreground or background?” — No. Measured here, the same load at UTILITY produced about 3× tail inflation against 79× undeclared and 2.7× at BACKGROUND. Utility is a real intermediate position, and knowing that is the difference between guessing and choosing.
ExplainWhat is priority inversion and how does the system fix it?
An urgent thread waits on something a less urgent thread holds, so the urgent work now proceeds at the holder’s speed. The kernel can fix it by donating the waiter’s priority to the holder for the duration — but only if it knows who the holder is.
That is a property of the primitive. A lock records an owner, so there is a specific thread to boost; so does a synchronous dispatch and pthread_join. A semaphore records a count and a condition variable records a predicate — neither names a thread, so neither can donate.
The design rule that follows: do not express “wait for this work” with a counting or signalling primitive when an ownership-recording one says the same thing.
Follow-up: “Is every case fixable by donation?” — No, and this is the important half. If the slow thread is merely runnable behind a lot of equally-ranked work, nothing is owned and there is nothing to donate to. That inversion is fixed by ranking the competing work, not by lock discipline. The discriminator is whether the slow thread is blocked or runnable.
DiagnoseOur fans spin up and the work is trivial. What is your first measurement?
CPU seconds divided by wall seconds — how many cores the process actually held. Measured here, a pipeline that spun while idle reported 12.64 on a 14-core machine to handle 200 small frames, against 0.00 for the same pipeline blocking properly.
If that number is above one while the app is idle, something is running that should be waiting, and the giveaway in the profile is that the hottest function computes nothing.
Follow-up: “Would voluntary context switches show the fix?” — Not on macOS. getrusage’s ru_nvcsw measured 0 in every fixture here, including ones that block thousands of times. ru_nivcsw is populated. Use CPU time; it is unambiguous.
ExplainWhat is the difference between preemptive and cooperative scheduling here?
The kernel preempts threads: a core can be taken away at essentially any instruction, whether or not the thread cooperates. The Swift runtime schedules tasks cooperatively: a task keeps its pool thread until it reaches a suspension point.
The consequence is asymmetric. A thread that runs a long computation is preempted and everyone else still progresses. A task that runs a long computation without suspending holds a pool thread, and the pool — sized near the core count on the assumption that tasks make progress or suspend — does not grow to compensate.
So “blocking” means something more expensive in task code than in thread code, and the failure mode is a stall rather than slowness.
Follow-up: “Does marking a function async make its body non-blocking?” — No. Between two awaits the task owns its thread outright, so a synchronous computation inside an async function blocks exactly as much as it would anywhere else.
DiagnoseWe added threads and it got slower. What happened?
Two candidates, and they are distinguished by whether the threads block.
If the work is CPU-bound and nothing blocks, extra threads cannot add throughput once the cores are full — they can only add preemptions. Measured here on 14 cores, going from 14 to 56 CPU-bound threads left throughput-normalised time flat (185 → 176 ms) while involuntary context switches rose from 2,344 to 9,597. That costs latency and energy, not throughput.
If it got genuinely slower rather than merely later, suspect contention — a shared lock whose hold time is now multiplied by the number of contenders. That is chapter 6’s territory.
Follow-up: “When does a bigger pool help?” — when threads block, because then the extra threads cover the blocked ones. “Do my threads block?” is the question that decides whether a bigger pool is a fix or a regression.
ChooseA thread must never miss its deadline. How do you set it up?
A dedicated thread, not a task on a shared cooperative pool whose other occupants you do not control. Declare QOS_CLASS_USER_INTERACTIVE on that thread, and — the part people miss — make sure everything it competes with has declared something lower, because the declaration only means something relative to the queue.
Then remove waits from its path: no locks shared with background work, no allocation that could hit a slow path, no synchronous I/O, and nothing that can block for an unbounded time.
And measure it as a percentile with a bound, not as an average, because the failure is always in the tail.
Follow-up: “What about real-time scheduling?” — macOS does expose time-constraint policies for genuinely hard-deadline work such as audio rendering. That is a different contract with admission requirements, and it is out of scope for this chapter; nothing here was measured with one.
ExplainWhy did a timer-based latency measurement show nothing?
Because timer coalescing puts a floor under it. Measured here, a usleep-driven 5 ms heartbeat was about 1 ms late at the median regardless of load — kern.timer.coalescing_enabled is 1 by design, to let the system batch wakeups and save energy. The measurement’s noise floor was far above the effect being looked for, and only the maximum moved.
Rewriting it to time a fixed unit of CPU work, calibrated against the same machine’s idle baseline, produced a 79× signal from the identical defect.
Measure work, not wakeups, when the question is about scheduling.
Follow-up: “Is coalescing a bug?” — No, it is an energy feature, and it is why an app that wants precise timing should say so through the appropriate API rather than assuming sleep is precise.
DesignYour framework runs on the client’s threads. What do you promise about scheduling?
Three things, and all three are contract.
Which thread your callbacks arrive on, and whether that is a guarantee or a current implementation detail. If a client may be on the main thread when you call back, blocking inside your call is their hang.
Whether you create threads, and how many. Apple’s guidance is to size pools from the core count and explicitly not to scale with workload; a per-framework pool is a defect even when each one is individually reasonable, because the client has several of you.
What class your work runs at. If you spawn background work, declare it — an undeclared pool competes in the user-visible band, which is the whole of this chapter’s exercise 04.
Follow-up: “What if the client calls you from a high-priority thread?” — then your work inherits that urgency, which is usually right and occasionally catastrophic if you then do something long. Say in the documentation which calls are cheap and which should not be made from a latency-sensitive thread.
ExplainName the states a thread can be in — as a profiler reports them, not as a textbook does.
Three at the kernel level: running on a core, ready but not running, and blocked on something that is not a core. The strong answer knows the tool splits the middle one and does not use this section’s vocabulary: Preempted is a thread that was running and lost its core; Runnable is reserved for one just woken and not yet dispatched. Interrupted, Idle and Terminated make up the rest.
Why it matters: told to “look for runnable”, you open the Thread State Trace, find that lane essentially empty, and conclude the machine is fine. Measured on 28 CPU-bound threads at USER_INTERACTIVE on 14 cores, 52.6% of thread-time was Preempted and Runnable recorded a single interval.
Follow-up: “Can the kernel tell you what a blocked thread is waiting for?” — no. thread_info reports the same TH_STATE_WAITING for a timer, a mutex and a read. The state says a thread is not runnable; you always need the stack as well.
ChooseYou are about to declare BACKGROUND on a batch re-index. What does it cost?
It is the only class that moves work to the efficiency cores. Measured, five threads one per class competing on 14 cores: the top three classes all resolve to priority 31 with ~98% performance-core samples; UTILITY to 20, still ~91% P; BACKGROUND to 4, with only 10% P-core samples and 512.6 ms of processor time against ~343 ms for the same work in the other classes. On an otherwise idle machine the demotion cost about 2.4× wall time.
So the answer has a price attached, which is the graded part. If the batch has a deadline of its own, demotion is the wrong repair — make it smaller or interruptible instead.
Follow-up: “Is it pinned to the efficiency cores?” — no. Placement is a preference, not a partition: the BACKGROUND thread still took 489 samples on performance cores. And never branch on 31/20/4 — those are observations from a trace on one OS build, not API.
DiagnoseAsync work stops finishing entirely once we scale up. No deadlock, CPU near zero.
Cooperative-pool exhaustion: enough tasks are blocked in the kernel that no pool thread is left, and the pool does not grow to replace a thread parked in a syscall, because it grows to meet runnable demand. Measured, the cliff is exactly at activeProcessorCount — 13 blocking jobs always completed, 14 always stalled — and pre-warming the pool does not move it.
The repair is to keep the blocking and change where it happens: hand the synchronous work to a Dispatch queue and suspend the task with withCheckedContinuation. State its price: the repaired build used 41 kernel threads at 200 jobs, so it converts a hard stall into real thread growth that needs its own bound.
Follow-up: “Make it reproduce on a laptop with a different core count.” — LIBDISPATCH_COOPERATIVE_POOL_STRICT=1 pins the pool to width one, so a single blocking task starves deterministically on any machine. Run async test suites under it.
DiagnoseSame symptom, but it stalls with one job rather than fourteen.
That is not the pool, it is actor serialisation. Task { … } written inside @main’s static func main() async inherits MainActor isolation, so the work serialises on one actor instead of spreading across the pool — and then one blocking job stops everything, warmed or not.
The discriminator is the question this chapter keeps returning to: “which thread is this on?” and “which executor is this isolated to?” have different answers and different failure modes.
Follow-up: “How would you confirm it?” — switch to Task.detached, which does not inherit isolation. If the threshold jumps up to the core count, it was isolation. If it stays at one, keep looking.
Choose a toolA hitch you cannot reproduce, and no Instruments window is available.
sample <pid> first, unprivileged, to split busy from blocked. Then xctrace record --template 'System Trace' and export the thread-state table, which gives Preempted share, core cluster per sample and scheduler priority — real Instruments data, headless, no elevation.
Know the privilege boundary: spindump refuses without root even for a single named process; sample does not; fs_usage also needs root. And know what the Thread Performance Checker will not tell you: it detects priority inversions and non-UI work on the main thread, which is none of oversubscription, E-core placement, a wrong QoS band or pool starvation.
Follow-up: “What do you measure?” — percentiles of a fixed unit of work against the same machine’s idle baseline. A mean exonerates a tail bug: measured, p50 moved 2% while p99 moved 79×.
Drill
Answer first, then read the explanation.
One defensible first move each.
Scenario 01 · The judder nobody can reproduce
Users report occasional stutter. Your profile shows a normal average frame cost and no blocked threads. First move?
Scenario 02 · The QoS that did nothing
You called pthread_set_qos_class_self_np(QOS_CLASS_BACKGROUND, 0) before creating your worker pool and the tail did not improve. Why?
Scenario 03 · Two waits, one donation
An urgent thread waits behind background work. In which case can the kernel raise the background thread’s priority?
Scenario 04 · Idle and hot
Your daemon is idle and holding two cores. Which measurement identifies the cause fastest?
Record your answer · 1
Rank one piece of background work.
Pick real background work in your own code. Write: what it is for → the class you would declare → what the user loses if it is too low → what the user loses if it is too high → the measurement that would settle it.
Record your answer · 2
Rehearse one latency diagnosis.
For a hitch you have actually seen, write the chain: symptom → running, runnable or blocked → the measurement that decided it → what would have falsified it → the smallest fix → what that fix cost.
Primary sources for this chapter
Apple documentation, WWDC sessions, and Darwin headers
Raw URLs are printed beside each title so they can be copied without following a link.
| Source | URL (copyable) | Used in this chapter for |
|---|---|---|
macOS 26.3 SDK header <sys/qos.h> | $(xcrun --show-sdk-path)/usr/include/sys/qos.h | The six QoS classes, their raw values, and Apple’s description of each |
| DispatchQoS | https://developer.apple.com/documentation/dispatch/dispatchqos | The Swift-facing spelling of the same classes |
| Thread.QualityOfService | https://developer.apple.com/documentation/foundation/thread/qualityofservice | Declaring a class on a Foundation thread |
| ProcessInfo.activeProcessorCount | https://developer.apple.com/documentation/foundation/processinfo/activeprocessorcount | Sizing a pool from what is actually available rather than advertised |
| Tech Talk 110147 · Tune CPU job scheduling for Apple silicon games | https://developer.apple.com/videos/play/tech-talks/110147/ | Timestamps verified against the published transcript: the P and E cores use a similar microarchitecture 2:36–3:02; “do not scale your thread count based on your workload … query CPU information to size your thread pool” 15:51–16:13; which primitives can resolve an inversion, and that dispatch_semaphore and condition variables cannot — “the runtime doesn’t know which thread will signal it” — 29:50–31:47 |
| WWDC23 10248 · Analyze hangs with Instruments | https://developer.apple.com/videos/play/wwdc2023/10248/ | Thread State Trace and the Narrative view, 37:07–38:22 (verified). Timestamp gap: the phrase “separating busy from blocked” is this page’s summary, not a quotation, and has no single transcript location |
| WWDC25 308 · Optimize CPU performance with Instruments | https://developer.apple.com/videos/play/wwdc2025/308/ | Timer aliasing, and “you should prefer CPU Profiler over Time Profiler”, 9:37–10:17 (verified) |
| WWDC25 226 · Profile and optimize power usage in your app | https://developer.apple.com/videos/play/wwdc2025/226/ | Background reading on the energy cost of avoidable CPU time. Timestamp gap: nothing on this page quotes it, so no transcript location was sought |
| OSSignposter | https://developer.apple.com/documentation/os/ossignposter | Shipping your own latency measurement |
macOS 26.3 manual pages getrusage(2), getpriority(2), sample(1) | man 2 getrusage · man 2 getpriority · man 1 sample | Context-switch counters; why nice is the wrong vocabulary; separating busy from blocked without Instruments |
Darwin interfaces thread_info / THREAD_BASIC_INFO, pthread_set_qos_class_self_np | $(xcrun --show-sdk-path)/usr/include/mach/thread_info.h · .../pthread/qos.h | Live per-thread run states; setting a class on the calling thread |
How to read the labels on every claim in this chapter
documented — Apple documentation, an official WWDC or Tech Talk transcript, Swift Evolution, or a Darwin manual page or SDK header on macOS 26.3. Quoted verbatim with the source named beside it.
measured here — observed on one machine: Apple M4 Pro (10 performance + 4 efficiency cores), macOS 26.3 (25D125), 48 GB, 16 KiB pages, Swift 6.2.4, Apple clang 17.0.0, Xcode 26.3. Evidence of a mechanism, never a benchmark. Reproduce; do not quote.
inference — our reasoning joining the two above. Marked so you can disagree with the reasoning without doubting the evidence.
negative result — something that was tried and did not show what was expected. These bound what a chapter is allowed to claim, and they are kept deliberately visible.
correction — a correction to an earlier reading published on this page, naming what was wrong and why. This page’s whole method is labelled provenance, so its own errata carry a label too rather than disappearing into a rewrite.
version-sensitive — observed behaviour of this OS build that must be re-measured on another before it is relied on.
Scope of the evidence, stated plainly. The labels above mean the same thing here as everywhere else on this page: documented is Apple’s words or a Darwin header, measured here is one machine — Apple M4 Pro (10P + 4E), macOS 26.3 (25D125), on 2026-09-23 — inference is our reasoning joining them, and negative result is something that did not show what was expected. Four things this chapter does not have: no Instruments GUI session and no screenshot, so System Trace and CPU Profiler behaviour is documentation rather than observation; no real-time or time-constraint thread was created, so nothing here describes hard-deadline scheduling beyond naming it out of scope; the kern.sched, kern.clockrate and timer-coalescing sysctls are observed and are not documented API, so they are labelled as inference and must never be branched on in shipping code; and ru_nvcsw is reported as unusable on this platform rather than quietly omitted, because a reader who tries it deserves to know why it returns zero.
14 · Mistakes, follow-ups & recap
What goes wrong, what gets asked next, and the chapter in eight plain sentences.
No new terms, no numbers to memorise. If one of these does not yet feel obvious, the section that earns it is named beside it.
Where people go wrong
- Reading a mean when the complaint is a tail“We measured it and throughput was fine” is the single most common wrong answer in this chapter. Under 56 undeclared threads on 14 cores the median moved by 2% and the 99th percentile moved by a factor of 79 — the same run, two conclusions. Percentiles of a fixed unit of work against an idle baseline, or you have not measured the defect.
- Adding threads to a problem that is not blockingExtra threads only help when the existing ones wait. Once every core is busy, another runnable thread cannot add throughput; it can only add preemptions. Involuntary switches rose from 2,344 to 9,597 across a 1×–4× sweep while normalised throughput stayed flat.
- Declaring everything user-interactiveThe top three quality-of-service classes all resolve to the same priority band and the same cores. Marking everything urgent does not raise your work; it removes the only signal you had for the work that really is urgent.
- Blocking a cooperative-pool threadA Swift task that blocks does not free its thread the way a blocked thread frees a core. Enough of them and the pool is gone, with nothing held and no cycle anywhere — that failure is chapter 7’s, and the cause is written here.
- Sampling with a timer and trusting itTimer-driven sampling aliases against periodic work at the same cadence, and timer coalescing puts a floor under any wakeup-latency measurement. Measure work done, not wakeups; prefer the CPU profiler over the time profiler.
The follow-up you will get
How many threads should this pool have?
Size it from what is actually available rather than what is advertised — activeProcessorCount, which can shrink under you on Apple silicon — and only then ask whether the work blocks. Non-blocking work wants roughly the core count; blocking work wants more, and the honest answer names the measurement that would tell you which you have. Apple’s own guidance is not to scale thread count with workload size.
The main thread is stuck. What do you run first, and why that?
sample <pid> during the freeze, because it needs no elevation and separates busy from blocked immediately. If the stack is deep in your own code, it is a profiling problem. If it is parked in a wait, read the blocking frame and find the owner. Naming the discriminator before the tool is the part being scored.
Would raising the priority of the slow thread fix it?
Only if it is blocked on something with an owner the kernel can raise. If it is merely runnable behind a lot of equally-ranked work there is nothing to donate to, and the repair is to lower the competition rather than raise the victim. That distinction — bounded ownership-based inversion versus unbounded load-based inversion — is section 10.
Recap — what to carry out of this chapter
1. A process owns the memory and the resources; the threads inside it own only a stack and a place in the queue, and share everything else. (1)
2. A thread is always in one of three states — running, ready to run, or waiting — and “ready to run, no core available” is where latency hides. (2)
3. Concurrency is how much work is in flight; parallelism is how much is executing this instant, and the second number belongs to the hardware. (3)
4. Quality of service is not a priority number you set. It is a statement of intent the system maps onto bands and core types, and declaring everything urgent means nothing is. (4)
5. A queue is a place work goes; a task is a piece of work that can suspend and resume. They solve the same problem with different failure modes. (5–6)
6. An actor protects state by serialising access to it, but other work interleaves at every suspension point — which is exactly what a lock does not allow. (7)
7. Urgency travels across a wait only when the thing being waited on knows whose priority to raise. Nothing without an owner can donate. (8, 10)
8. Before changing any code, separate busy from blocked. Nearly every wrong scheduling fix comes from skipping that one measurement. (11–11A)
Next: chapter 6 asks what protects shared state once more than one of these threads reaches it, and chapter 7 asks what it looks like when they stop making progress. The evidence for all three lives in the execution lab.
Chapter 6 · locks and synchronization
Locks and synchronization.
Once more than one thread can reach the same state, something has to say who touches it and when. This chapter is the shelf of things that can say it — and, more usefully at interview, what each one is bad at. It builds on chapter 5’s execution model and hands over to chapter 7 the moment a choice here stops the program making progress.
The problem, in plain words
Two threads share the same memory. If both change the same thing at the same time, the result is whatever the machine happened to do that microsecond — and it is different on the next run. A lock is a way of saying “only one of you at a time, and the rest of you wait here”.
The part people skip is what is being protected. It is almost never a single variable; it is a rule that has to stay true — a balance that must never go negative, a count that must match a list. The lock has to be held across every step that would briefly break that rule, or it protects nothing.
What breaks without it. Totals come out slightly wrong, only under load, only on fast machines, and only in production. A test that runs the same code once and gets the right answer has proved nothing at all.
How to recognise it
The complaint is about a value, not about time: a total that is “a bit low”, a balance that went negative, a cache that returned something stale, a counter that drifts. Chapter 5 is about work not happening; this chapter is about work happening in an order nobody intended.
The question that sorts it is: which rule was briefly false, and who could have looked while it was? If you can name the rule and the window, you have found the critical section. If you cannot name a rule — only a variable — you are about to reach for an atomic and fix nothing.
The tell: “we made it atomic and it still loses money.” An atomic makes one location indivisible; it does not make two operations indivisible. Whenever the invariant spans a check and an act, or two separate fields, the answer is a lock around both — or a design where the pair cannot be observed apart.
The idea
Almost every shared-state bug is a read, modify, write that got interleaved. The thread reads a value, works out the new one, and writes it back — and in the gap between the read and the write, someone else did the same thing from the same starting point. One of the two updates disappears with no error anywhere.
A lock closes that gap by making the whole sequence look instantaneous to everybody else. That is why the interesting question is always how wide the locked region has to be, not which primitive to use. Too narrow and the rule is still observable while false; too wide and you have serialised work that did not need it.
Two different promises are hiding in the word “atomic”. Atomicity means nobody sees a half-written value. Ordering means nobody sees your writes in a different sequence than you made them. A lock gives you both. A relaxed atomic gives you only the first.
Watch it run
Two deposits of 10, and only one of them survives
The balance starts at 100. Two threads each add 10. Every line of this is correct code; nothing crashes, nothing warns, and the answer is wrong.
this thread acts now
this thread is between steps
the damage
the repair
Thread 1about to read
balance += 10 — three machine steps, not oneThread 2about to readthe same three steps, on the same field
balance100the rule: every deposit must appear in the total
Step 1. Both threads want to add 10. The correct answer is 120.
Thread 1reads 100now it computes 100 + 10 = 110 in a register
Thread 2not yet—
balance100unchanged; thread 1 has not written yet
Step 2. Thread 1 reads. Nothing is wrong yet — but the rule is now in flight somewhere no one else can see.
Thread 1holding 110preempted, or simply slower than the other core
Thread 2reads 100it also computes 110 — from the same stale start
balance100still 100, and now two threads intend to write 110
Step 3. This single interleaving is the entire bug. Thread 2 read before thread 1 wrote, so both deposits start from the same number.
The window is a handful of nanoseconds, which is why it shows up under load and never in a unit test.
Thread 1writes 110correct, as far as thread 1 can tell
Thread 2writes 110also correct, as far as thread 2 can tell
balance110two deposits of 10 against a start of 100
Step 4. The second write lands on top of the first. Ten currency units have vanished, with no error, no crash and no warning.
Thread 1locks, reads 100, writes 110, unlocksthe whole sequence, not just the write
Thread 2waits, then reads 110it starts from what actually happened
balance120the rule held at every moment another thread could look
Step 5. The repair is not “make the write atomic”. It is to hold the lock across the read and the write, so the pair can never be observed apart.
Section 1 measures this exact program; the execution lab’s exercise 02 is the version that ships with a sanitizer transcript.
The code
Thirteen primitives, each shown in Swift with the trap it hides, live in section 2. The runnable programs for this chapter are:
How this chapter is built. The same eight steps as every other chapter: overview in plain words (above) → mental model and vocabulary (1) → Darwin mechanism and where the public contract stops (2) → deeper internals (2A–2F, 3) → Mac lab (C in the execution lab) → failure story and diagnosis workflow (A, B, D · 02) → interview questions (F, G) → mistakes, follow-ups and recap (4).
Which invariant is protected by what, how wide the critical section has to be, and what the failure looks like to the user.
Show the interleaving that breaks the rule, then the same program with the rule restored — and name the instrument that would have caught it.
futex implementation, C++ memory-model formalism, lock-free queue proofs, the fairness algorithm inside any particular primitive.
The sentence to have ready
“A lock protects an invariant, not a variable.” Almost every follow-up question in this chapter — granularity, ordering, whether an actor can replace the lock, why an atomic counter did not help — resolves faster once you name the invariant first and the storage second.
1 · Races & ordering
What “atomic” buys, and what it does not.
A data race is two threads accessing the same location with no ordering between them, at least one of them writing. The interview trap is the step after that: people reach for an atomic, the counter stops tearing, and the bug survives — because the invariant spanned more than one location.
Simplified diagram: real hardware may reorder and cache far more aggressively than a two-lane timeline can show. The point it is drawn to make is that value += 1 is three operations, not one.
$ swiftc -swift-version 5 -O racedemo.swift -o racedemo && ./racedemo
trial 1: expected 100000 | unguarded 56332 | Mutex 100000
trial 2: expected 100000 | unguarded 46626 | Mutex 100000
trial 3: expected 100000 | unguarded 59097 | Mutex 100000Swift · the exact program that produced the run above
import Foundation
import Synchronization
final class UnsafeCounter: @unchecked Sendable {
private var value = 0
func increment() { value += 1 }
var current: Int { value }
}
final class SafeCounter: Sendable {
private let value = Mutex<Int>(0)
func increment() { value.withLock { $0 += 1 } }
var current: Int { value.withLock { $0 } }
}
let n = 100_000
for trial in 1...3 {
let unsafe = UnsafeCounter()
DispatchQueue.concurrentPerform(iterations: n) { _ in unsafe.increment() }
let safe = SafeCounter()
DispatchQueue.concurrentPerform(iterations: n) { _ in safe.increment() }
print("trial \(trial): expected \(n) | unguarded \(unsafe.current) | Mutex \(safe.current)")
}This is a real run captured on 2026-09-21 while writing this chapter, on a 14-core Apple M4 Pro under macOS 26.3 with Swift 6.2.4, using DispatchQueue.concurrentPerform over 100,000 iterations. The numbers are machine- and run-specific; reproduce it yourself rather than quoting these figures. The point that generalises is the shape: roughly half the increments vanished, so a race is not a rare tail event under real contention.
Atomic ≠ thread-safe
An atomic makes one operation on one location indivisible. If your invariant is “count equals items.count”, two atomics cannot express it, because nothing orders them with respect to each other. Ask “what is the invariant?” before choosing the primitive.
Memory ordering, at interview depth
Relaxed: indivisible, but publishes nothing. Acquire (load): everything the releasing writer did before its store is visible after this load. Release (store): everything written before it becomes visible to an acquiring reader. Sequentially consistent: additionally, a single total order all threads agree on. This is a working model, not the formal memory model.
Swift makes the asymmetry a type error
AtomicLoadOrdering has three members (.relaxed, .acquiring, .sequentiallyConsistent) and AtomicStoreOrdering three (.relaxed, .releasing, .sequentiallyConsistent). AtomicUpdateOrdering has five — it adds both .releasing and .acquiringAndReleasing, so a read-modify-write can be purely acquiring or purely releasing as well. You cannot write a “releasing load”; the compiler refuses. Swift’s own reason: “By modeling these as separate types, we can ensure that unsupported operation/ordering combinations (such as an atomic ‘releasing load’) will lead to clear compile-time errors.” documented
Sources: AtomicLoadOrdering, AtomicStoreOrdering, AtomicUpdateOrdering.
When you actually need orderings
Almost never in framework-client code. A lock or an actor gives you release-on-unlock and acquire-on-lock for free. Reach for explicit orderings only for a hot counter, a one-shot flag, or a published-once pointer — and say so in the interview rather than performing expertise.
Swift · Atomic with each ordering, and why each one was chosen
import Synchronization
final class RequestMeter: Sendable {
private let served = Atomic<Int>(0)
private let ready = Atomic<Bool>(false)
private let claimed = Atomic<Bool>(false) // separate from `ready`
private let payload = Mutex<[String]>([])
// `.relaxed` - a pure counter. It publishes no other memory.
func recordServed() { served.wrappingAdd(1, ordering: .relaxed) }
func servedCount() -> Int { served.load(ordering: .relaxed) }
// Release/acquire pair: the writer fills the payload, THEN sets the flag.
// A reader that acquires `true` is guaranteed to see the payload writes.
func publish(_ lines: [String]) {
payload.withLock { $0 = lines }
ready.store(true, ordering: .releasing)
}
func consume() -> [String]? {
guard ready.load(ordering: .acquiring) else { return nil }
return payload.withLock { $0 }
}
// Compare-and-exchange is how a lock-free state machine advances exactly once.
// It runs on its OWN flag: consuming the claim must not retract publication.
func claimOnce() -> Bool {
let (exchanged, _) = claimed.compareExchange(
expected: false, desired: true, ordering: .sequentiallyConsistent)
return exchanged
}
}
// The type system enforces the asymmetry: a load cannot be `.releasing`,
// a store cannot be `.acquiring`. Only read-modify-write takes both.correction An earlier version of this snippet ran claimOnce() on the same ready flag that publish/consume use as their release/acquire channel. It compiled and the API use was correct, but a successful claim made consume() return nil forever — silently teaching that a publication flag and a one-shot claim token can be the same variable. They cannot. Type-checked after the fix: 0 errors, 0 warnings.
Distinguish the three claim types in your answer
API guarantee: Mutex “offers non-recursive exclusive access to the state it is protecting by blocking threads attempting to acquire the lock” — Apple’s own sentence. Teaching simplification: the acquire/release summary above. Inference: “this cache is read-mostly, so a readers-writer lock should win” — that is a hypothesis you owe a measurement, not a fact. Every claim in this chapter carries one of those labels: documented measured here inference negative result
The distinction interviews actually test
A data race and a race condition are different bugs.
One is unsynchronised memory access, and a sanitizer finds it. The other is a correct program reaching an impossible state, and no sanitizer will ever find it. Confusing them is the most common way a candidate gives a confident wrong answer about Swift 6.
| Branch | Synchronisation | Result | Thread Sanitizer |
|---|---|---|---|
| A · data race unsynchronised value += 1 | None | 8 threads × 200,000 increments → 900,219 of 1,600,000 (re-run 2026-09-22; a second run lost even more) | 3 warnings, including Swift’s own “Swift access race” |
| B · race condition check-then-act across two critical sections | Every access correctly locked | 5 seats, 40 buyers → 40 sold, 35 oversold | Nothing. Zero bytes. |
| C · the repair one critical section spans check and act | Same lock, wider section | 5 sold, 0 oversold, invariant held | Nothing (correctly) |
measured here Re-run in full on 2026-09-22; the fixture suite asserts TSan’s silence on branch B as a passing condition, because that silence is the lesson. The repair is not a different lock — the lock never changed. Only the boundary of the critical section changed.
Swift · correctly locked, and still loses money (deliberately-wrong code, clearly marked)
import Foundation
// Every access is locked. There is no data race and Thread Sanitizer is silent.
// It still goes negative, because the INVARIANT spans two critical sections.
final class Ledger: @unchecked Sendable {
private let lock = NSLock()
private var balance = 100
func canWithdraw(_ n: Int) -> Bool { lock.lock(); defer { lock.unlock() }; return balance >= n }
func withdraw(_ n: Int) { lock.lock(); defer { lock.unlock() }; balance -= n }
// The repair is not a different lock. It is one critical section that both
// decides and acts, so no other thread can act in the gap.
func withdrawAtomically(_ n: Int) -> Bool {
lock.lock(); defer { lock.unlock() }
guard balance >= n else { return false }
balance -= n
return true
}
}
// Caller, from two threads. `if canWithdraw { withdraw }` is the bug.
func brokenCaller(_ ledger: Ledger) {
if ledger.canWithdraw(100) { ledger.withdraw(100) }
}“I ran it and got the right answer” is not evidence
The same unsynchronised counter, same machine, same session, built four ways: measured here
| Build | Observed total (expected 1,600,000) | Thread Sanitizer |
|---|---|---|
-Onone, no sanitizer | Loses a large and varying fraction, every run | — |
-O, no sanitizer | 1,600,000 — exactly correct, 3 runs of 3 | — |
-Onone + TSan | 622,675 | 3 warnings |
-O + TSan | 573,620 | 3 warnings |
At -O the optimiser collapses the increment loop into a single add, the window essentially vanishes, and the program prints the right answer every time. The race is still there — TSan proves it at both optimisation levels. And note the second lesson, which candidates miss: enabling TSan changed the observable behaviour too, because instrumentation defeats the collapse. A sanitizer is not a passive observer, so never reason about timing from a sanitized run.
1AABA, and the primitive Swift ships to defeat itsingle-word CAS · WordPair · the tag can still wrap
The problem, in Swift’s own words: “A freshly allocated object often happens to be placed at the same memory location as a recently deallocated one. Therefore, two successive loads of a simple atomic pointer may return the exact same value, even though the pointer may have received an arbitrary number of updates between the two loads, and the pointee may have been completely replaced.” documented (SE-0410 · Low-Level Atomic Operations, implemented in Swift 6.0) A compare-and-exchange therefore succeeds on stale reasoning.
The repair Swift ships: WordPair (macOS 15) gives double-wide atomics, so “the second word can be used to augment atomic values with a version counter (sometimes called a ‘stamp’ or a ‘tag’).” It is not universal: “This type only conforms to AtomicRepresentable on platforms that support double wide atomics.” And it does not solve ABA — the tag can wrap. It makes ABA improbable, not impossible.
=== 1. plain single-word CAS is blind to A -> B -> A ===
observed=100 then value went 100 -> 200 -> 100
compareExchange(expected: 100, desired: 999) -> exchanged=true, original=100
=> the CAS SUCCEEDED even though the slot was mutated twice in between. This is ABA.
=== 2. the same sequence with a WordPair (value, tag) ===
observed=(v:100, tag:0); now (v:100, tag:2)
compareExchange -> exchanged=false
=> the CAS FAILED. The value matches but the tag does not, so re-read and retry.The CAS form most candidates never name. weakCompareExchange “is allowed to spuriously fail; it is designed to be called in a loop until it indicates a successful exchange has happened” — because of “some transient condition … such as an incoming interrupt during a load-link/store-conditional instruction sequence.” documented And a real CAS loop takes two orderings, successOrdering:failureOrdering:, where the failure ordering is an AtomicLoadOrdering. The asymmetry is the point: a failed CAS performed no write, so it cannot be a release.
Swift · ABA, the tagged repair, and a correct CAS loop
import Synchronization
// Single-word CAS is blind to A -> B -> A.
let plain = Atomic<Int>(100)
func blindClaim() -> Bool {
let observed = plain.load(ordering: .acquiring) // reads 100
// ... another thread sets 200, then back to 100 ...
let (exchanged, _) = plain.compareExchange(
expected: observed, desired: 999, ordering: .sequentiallyConsistent)
return exchanged // true - history invisible
}
// WordPair (macOS 15) carries a version tag in the second word. The tag can
// still wrap: this makes ABA improbable, not impossible.
let tagged = Atomic<WordPair>(WordPair(first: 100, second: 0))
func taggedClaim() -> Bool {
let observed = tagged.load(ordering: .acquiring)
let (exchanged, _) = tagged.compareExchange(
expected: observed,
desired: WordPair(first: 999, second: observed.second &+ 1),
ordering: .sequentiallyConsistent)
return exchanged
}
// A real CAS loop uses the weak variant and SEPARATE orderings. A failed CAS
// performed no write, so its failure ordering cannot be a release.
let counter = Atomic<Int>(0)
func incrementByCAS() {
var current = counter.load(ordering: .relaxed)
while true {
let (exchanged, original) = counter.weakCompareExchange(
expected: current, desired: current &+ 1,
successOrdering: .acquiringAndReleasing,
failureOrdering: .acquiring)
if exchanged { return }
current = original
}
}Lock-free is a claim about blocking, not speed
Swift requires “a lock-free implementation on every platform” but explicitly not wait-freedom: “if no direct instruction is available for an operation, then it must still be implemented, e.g. by mapping it to a compare-exchange loop.” documented So an individual thread can retry indefinitely. Lock-freedom converts waiting into repeated work.
And under contention that is often a loss
A weakCompareExchange increment loop over 1,000,000 increments across 14 cores took 2,187,428 retries — about 2.2 failed attempts per success — and ran 6.7× slower than an unfair lock. measured here “Use an atomic because a lock is expensive” is not supported by this data.
No default ordering, on purpose
SE-0410: “we require an explicit ordering argument on all atomic operations. The intention here is to force developers to carefully think about what ordering they need to use … making it far less likely that an unintended default .sequentiallyConsistent ordering slips through code review.” documented
One ordering has no Swift spelling
std::memory_order_consume. SE-0410 lists it as “not yet adopted”. documented A candidate who names consume as a Swift ordering is wrong; one who knows why it is absent is ahead. Also worth carrying into any diagnosis answer: SE-0410 notes Thread Sanitizer “does not support fences and may report false-positive races for data protected by a fence.”
2 · Primitive matrix
Thirteen primitives, and when each one is wrong.
You will be asked to choose, and then asked why not one of the others. Learn the whole shelf so the choice is visibly a choice — then say out loud that on a modern macOS target your defaults are Mutex for new synchronous state and an actor for anything that touches async code. The column that earns points is the last one: these are not interchangeable, and naming what a primitive is bad at is how you show you chose rather than reached.
| Primitive | Owner? / fair? / recursive? | Blocks | Guards state, or counts permits | Reach for it when | Do not use when |
|---|---|---|---|---|---|
Mutex<Value>Synchronization · macOS 15 | Single owner · not fair · not recursive | a thread | State lives inside the lock | Default for new synchronous shared state on macOS 15+ | You need to hold it across an await, or re-enter it — re-entry kills the process (SIGKILL) |
OSAllocatedUnfairLockos · macOS 13 | Single owner · not fair · not recursive | a thread | Optional in-lock state | Back-deployment below macOS 15; repairing existing os_unfair_lock Swift code | Same as above; and never the bare lock()/unlock() pair in async code — the compiler now refuses it |
os_unfair_lockC · macOS 10.12 | Single owner · not fair · not recursive | a thread | Nothing — you own the state | C and Objective-C | From Swift at all. It is a value type with no stable address: “the system may lock or unlock the wrong object” |
pthread_mutex_tPOSIX | Single owner · fairness selectable · recursive only with PTHREAD_MUTEX_RECURSIVE | a thread | Nothing | Portable C; when you need ERRORCHECK self-deadlock detection or an explicit fairness policy | You want fairness for its own sake — fair handoff cost ~180× throughput here |
NSLockFoundation · macOS 10.0 | Single owner · not fair (first-fit) · not recursive | a thread | Nothing | Objective-C interop; pre-macOS 13 targets | New Swift code — the two rows above are faster and keep the state with the lock. Re-entry “will lock up your thread permanently” |
NSRecursiveLockFoundation · macOS 10.0 | Single owner · not fair · recursive | a thread | Nothing | Untangling a legacy re-entrant call graph | You are reasoning about queues or actors — recursion is tracked per thread, and neither guarantees one |
pthread_rwlock_tPOSIX | No single owner · writer-preferring · multiple read locks allowed, read-while-write undefined | a thread | Nothing | Read-mostly data and a read critical section long enough to amortise the bookkeeping — crossover measured at ~4,000 element-reads | The critical section is short. At 100% reads with a 1-element section it was 30× slower than a plain exclusive lock — and it has no owner, so no inversion resolution |
NSConditionFoundation · macOS 10.5 | Lock: single owner · wakeups: no owner | a thread | A predicate — carries no permits | A thread must wait for a state, not just for exclusivity | You want a stored permit. A signal() sent while nobody is parked is gone forever |
DispatchSemaphoreDispatch · macOS 10.6 | No owner (asymmetric) | a thread | Stored permits — the count survives | Bounding concurrency (“at most 4 decoders”) from code that is already asynchronous | Making async code synchronous. The compiler refuses wait() in async, and it hides the dependency so priority cannot be donated |
Serial DispatchQueueDispatch · macOS 10.6 | Single owner · strict FIFO · not recursive | a thread on sync; nothing on async | Neither — it serialises work | Serialising a subsystem; targeting several queues onto one root | A hot path — sync cost ~40× an unfair lock uncontended, ~150× contended. And sync onto your own lineage traps |
Concurrent queue + .barrierDispatch · macOS 10.7 | Multiple owners | a thread on sync | Neither | A private concurrent queue you created, where readers overlap and writers must not | The queue is global or serial — the barrier silently degrades to a plain async (measured: 38–40 “barriers” at once) |
actorSwift 5.5, back-deployed to macOS 10.15 | Owner: its executor · not FIFO, reordered by priority · re-entrant at every await | a task, not a thread | Isolated state the compiler checks | Anything already async; several fields sharing one invariant | The invariant must survive an await, or the accessor must stay synchronous |
Atomic<Value>Synchronization · macOS 15 | n/a — lock-free, not wait-free | nothing | One location, indivisibly | A hot counter, a one-shot flag, a published-once pointer | The invariant spans more than one location; or contention is high — measured 3.5–3.9× slower than the unfair lock across 14 cores |
documented Availability figures are each symbol’s own DocC platforms block; quoted sentences from Mutex, OSAllocatedUnfairLock, os_unfair_lock_lock, NSLock, NSRecursiveLock, NSCondition, DispatchSemaphore, dispatch_barrier_async, and man 3 pthread_rwlock_rdlock on macOS 26.3. measured here for every ratio. inference The last two columns are our guidance, not Apple’s. Two availability caveats worth keeping honest: the actor keyword is a Swift 5.5 language feature — the macOS 10.15 figure is the DocC availability of the Actor protocol that back-deployment makes reachable — and the DispatchQueue/DispatchSemaphore class pages carry no platforms block at all, so those figures come from the nearest member that does.
1 · Ownership, not speed, is the first question
Only a primitive with a single known owner can have a priority inversion resolved for it. Apple: “the runtime knows which thread will unlock the lock next. We can take advantage of that power to automatically resolve priority inversions in your app between the waiters and the owners of the lock. And even enable other optimizations, like directed CPU handoff to the owning thread.”
documented WWDC17 706 · Modernizing Grand Central Dispatch Usage (2017), 15:29–15:47.
2 · A semaphore counts; a condition tests
DispatchSemaphore stores permits — a signal() before any wait() is remembered. NSCondition stores nothing — a signal() with no parked waiter is lost, reproduced 3 of 3. measured here Conflating the two is the most common primitive-choice error at interview.
3 · A lock parks a thread; an actor parks a task
Apple: “Actors are also nonblocking. In this situation, the weather feed actor will be suspended and the thread it was executing on is now freed up to do other work.”
documented WWDC21 10254 (2021), 32:05–32:14.
The sentence that closes the matrix
Every “do not use when” above reduces to one of three failures: the primitive cannot be told who owns it, it stores the wrong kind of thing (a permit where you needed a predicate, or the reverse), or it cannot survive the shape of your call site (an await, a deinit, a re-entry). Name which one, and the choice defends itself. inference
2AUnfair, and why that is a featureownership · inversion · starvation
Interview explanation: “Unfair” is a precise claim, not a warning label. Apple: the lock “does not enforce fairness or lock ordering — for example, an unlocker could potentially reacquire the lock immediately, before an awoken waiter gets an opportunity to attempt to acquire the lock. This may be advantageous for performance reasons, but also makes starvation of waiters a possibility.” It also “does not spin on contention, but instead waits in the kernel to be awoken by an unlock” — which is why it replaced OSSpinLock.
The part that earns the point: unfair locks do record ownership: “Locks contain thread-ownership information that the system may use to attempt to resolve priority inversions.” A semaphore has no owner, so it can offer nothing equivalent. That single contrast explains most of Apple’s guidance about which primitive to reach for. Unfairness and ownership are independent properties — saying so is what separates a memorised answer from an understood one.
What fairness actually costs, measured. Darwin’s pthread_mutex_t is the only primitive here with a selectable fairness policy, which makes it the cleanest way to price the trade. Two threads hammering one mutex for 300 ms under each policy: measured here
| Policy | Acquisitions in 300 ms | Involuntary context switches | Switches per acquisition |
|---|---|---|---|
PTHREAD_MUTEX_POLICY_FAIRSHARE_NP | ~193k–255k | ~103k–121k | ~0.42 |
PTHREAD_MUTEX_POLICY_FIRSTFIT_NP (default since 10.14) | ~43.9M–46.5M | ~73k–100k | ~0.002 |
≈ 180× the throughput and ≈ 190× fewer context switches per acquisition, same code, only the fairness policy changed. Both policies split the work near-evenly between the two threads (50–59% busiest share), so in this symmetric case the cost of fairness is throughput, not distribution. Apple’s own man page explains the mechanism: FIRSTFIT “allows acquisition of the mutex to occur in any order … new contending acquirers may obtain ownership of the mutex ahead of existing waiters”, while FAIRSHARE “guarantees that ownership of a contended mutex will be granted to waiters on a strictly ordered first-in, first-out basis.” documented (man 3 pthread_mutexattr_settype, macOS 26.3)
A free bonus finding. A sample report of a deadlocked NSLock shows _pthread_mutex_firstfit_lock_slow in the stack — direct evidence that Foundation’s NSLock on macOS 26.3 is a pthread_mutex_t running the unfair first-fit policy. measured here
2BRecursive locks and the re-entrancy they hidesame thread ≠ same queue
Interview explanation: A recursive lock lets one thread acquire the same lock repeatedly. That is not a concurrency feature — it is a way to let a call graph that re-enters itself keep working. The hidden cost is that your invariant is broken in the middle of the outer critical section, and the inner call now observes that broken state. NSRecursiveLock is Apple’s answer to NSLock’s “calling the lock method twice on the same thread will lock up your thread permanently.”
The trap to name: recursion is tracked per thread. A serial dispatch queue does not promise a stable thread, so “I’m on my own queue, so the recursive lock will let me back in” is not a valid argument. Neither is “I’m in the same actor” — an actor re-entering after an await may be on a different thread entirely.
2CCondition variables: the predicate is the contractwait · signal · spurious wakeups
Interview explanation: A condition variable is a lock plus a parking lot. Apple’s documented sequence is: lock, test a Boolean predicate, wait() while it is false, re-test on wake, do the work, optionally update predicates and signal, unlock. The while loop is not defensive style — Apple states that “signaling a condition does not guarantee that the condition itself is true. There are timing issues involved in signaling that may cause false signals to appear.”
What wait() actually does: “When a thread waits on a condition, the condition object unlocks its lock and blocks the thread … The condition object then reacquires its lock before returning from the wait() or wait(until:) method.” Being able to say the atomic unlock-and-sleep is the whole reason the primitive exists is a strong senior signal. man 3 pthread_cond_wait says the same: it “atomically blocks the current thread waiting on the condition variable specified by cond, and releases the mutex specified by mutex.”
An adjacent trap worth one sentence. “It is advised that PTHREAD_MUTEX_RECURSIVE mutexes are not used with condition variables. This is because of the implicit unlocking done by pthread_cond_wait(3).” documented A recursive lock releases only one level, so the wait can sleep while still holding the mutex.
Simplified diagram. measured here The right-hand panel is reproduced 3 of 3: a producer sets the predicate and calls signal(), and three seconds later the broken consumer is still parked in wait() while the correct consumer — which tests the predicate before waiting — never waits at all. Every rule about condition variables follows from this one distinction.
| Run | What changed | Observed |
|---|---|---|
while predicate loop | Correct | 120 consumed, 0 underflows, 618 spurious wakeups absorbed |
if instead of while | One keyword | 120 consumed, 208 underflows — each one a removeFirst() trap in real code |
signal(), 8 waiters, one predicate flip | Wake one | 1 proceeded |
broadcast(), 8 waiters, one predicate flip | Wake all | 8 proceeded — seven bounce off the lock and re-park: the thundering herd |
measured here Re-run 2026-09-22. Spurious wakeups are injected — a broadcast that changes no state — so the bug is deterministic instead of a once-a-month production mystery. The genuine dilemma to state at interview: signal() risks a lost wakeup when several waiters could proceed or when waiters test different predicates on the same condition; broadcast() risks the herd. The safe default is broadcast() plus a strict while loop, paying wakeup cost for correctness.
2DWhat Swift 6 now refuses in an async context — and the two it still allows@_unavailableFromAsync
“Do not block a cooperative thread” has stopped being advice and become enforcement — for most primitives. Compiling a single async function that touches each one, with swiftc -swift-version 6 -typecheck on the macOS 26.3 SDK: measured here
Primitive in an async context | Result | Compiler message |
|---|---|---|
NSLock.lock() / .unlock() | error | “Use async-safe scoped locking instead” |
NSCondition.lock() / .wait() | error | “Use async-safe scoped locking instead” |
DispatchSemaphore.wait() | error | “Await a Task handle instead” |
DispatchGroup.wait() | error | “Use a TaskGroup instead” |
Thread.sleep(forTimeInterval:) | error | “Use Task.sleep(until:clock:) instead.” |
OSAllocatedUnfairLock.lock() / .unlock() | error | “Use withLock for scoped locking” |
OSAllocatedUnfairLock.withLock { } | allowed | — |
Mutex.withLock { } | allowed | — |
DispatchQueue.sync { } | allowed | — hole 1 |
DispatchWorkItem.wait() | allowed | — hole 2 |
Where the truth lives. The SDK’s own Dispatch.swiftinterface carries @_unavailableFromAsync on exactly two symbols — DispatchGroup.wait and DispatchSemaphore.wait — and DispatchQueue.sync and DispatchWorkItem.wait carry no such annotation. Foundation’s NSLock.h marks the lock family NS_SWIFT_UNAVAILABLE_FROM_ASYNC. documented
Why withLock is exempt, from Swift’s own proposal: “Calling withLock in an asynchronous function is okay because the same thread that calls lock() will be the same one that calls unlock() because there will not be any suspension points between the calls.” documented (SE-0433 · Synchronous Mutual Exclusion Lock)
The teachable rule: DispatchQueue.sync is the one the runtime covers instead, by trapping rather than hanging. DispatchWorkItem.wait is covered by neither, and is the quietest remaining way to block a cooperative thread. inference
A caveat on how easy this is to mis-measure. A naive probe of DispatchWorkItem.wait() in an async function does produce errors — but they are Sendability errors, not noasync errors. Passing the work item in as a parameter compiles with 0 errors and 0 warnings, which is the result that settles it. negative result
Swift · the one synchronous frame that defeats the guardrail
import Foundation
// Swift 6 makes the NAIVE spelling a compile error:
// error: instance method 'wait' is unavailable from asynchronous contexts;
// Await a Task handle instead
//
// ONE synchronous helper frame defeats that guardrail, and that is how the bug
// actually reaches production - the blocking call is usually two or three frames
// down inside a legacy API.
func legacyBlockingRead(_ sem: DispatchSemaphore) -> Bool { // <-- the escape hatch
sem.wait(timeout: .now() + 2) == .success
}
func wedgesTheCooperativePool(_ sem: DispatchSemaphore) async -> Bool {
legacyBlockingRead(sem) // compiles cleanly; occupies a pool thread
}
// The repair: express the dependency in the task graph so the runtime can see it.
func repaired(_ produce: @Sendable @escaping () async -> Int) async -> Int {
async let value = produce()
return await value
}2EThe readers-writer lock is a bet, and it usually losesthe crossover, measured
100% reads across 14 cores, varying only the length of the read critical section (“span” = array elements summed while holding the read lock): measured here
| Span | Operations | pthread_rwlock | OSAllocatedUnfairLock | rwlock vs exclusive |
|---|---|---|---|---|
| 1 | 200,000 | 122.0 ms | 4.1 ms | 30.0× worse |
| 8 | 200,000 | 117.0 ms | 7.7 ms | 15.2× worse |
| 64 | 200,000 | 125.0 ms | 8.2 ms | 15.3× worse |
| 512 | 200,000 | 147.0 ms | 27.9 ms | 5.3× worse |
| 4,096 | 20,000 | 10.3 ms | 21.4 ms | 0.48× — rwlock wins |
| 32,768 | 20,000 | 23.1 ms | 208.7 ms | 0.11× — rwlock wins |
“Read-mostly” alone is not the criterion. A separate sweep varying the write ratio at a one-element critical section found the rwlock 10.8×–14.1× worse at every ratio from 0% to 50% writes. The decision rule: a readers-writer lock is a bet that concurrent readers will overlap enough to pay for its own bookkeeping. If the read critical section is a dictionary lookup, they cannot — and you have bought a slower exclusive lock with no owner, and therefore no priority-inversion resolution. inference
On starvation, be precise. The man page says “To prevent writer starvation, writers are favored over readers” documented, so readers are de-prioritised by design. But four continuous writers against one reader for 500 ms still let the reader complete 43,871 reads. measured here On Darwin, in this shape, the reader is de-prioritised, not permanently starved. Say “permits starvation”, not “causes” it.
One more correction to the matrix row. Recursive read locking is explicitly allowed: “A thread may hold multiple concurrent read locks. If so, pthread_rwlock_unlock() must be called once for each lock obtained.” It is read-while-write that is undefined, and taking a write lock while already holding either returns EDEADLK. documented
2FWhat the primitives actually costand the benchmarking trap that invalidates most numbers
1,000,000 increments of one shared counter, built -O -wmo. These exist to show shape, not to be quoted. measured here
| Primitive | Uncontended (1 thread) | Contended (14 cores) |
|---|---|---|
OSAllocatedUnfairLock.withLock | 1.72 ms · 1.0× | 21.9 ms · 1.0× |
Mutex.withLock | 1.84 ms · 1.0× | 25.0 ms · 1.1× |
NSLock.lock/unlock | 5.83 ms · 3.4× | 38.4 ms · 1.5× |
Atomic.wrappingAdd(.relaxed) | 1.79 ms · 1.0× | 76.5 ms · 3.5–3.9× |
Atomic weak-CAS loop | 1.87 ms · 1.0× | 145.8 ms · 6.7–7.1× (2,187,428 retries) |
DispatchQueue.sync (serial) | 71.9 ms · ~40× | 3237 ms · ~150× |
Three readings a candidate can defend. (1) Uncontended, the lock is free — Mutex, the unfair lock and a relaxed atomic add are indistinguishable at ~1.8 ns/op, so “use an atomic because a lock is expensive” is not supported. (2) Contended, the unfair lock beats the atomic. inference The plausible mechanism is the one Apple describes for convoying: the unfair lock lets one thread reacquire and batch while staying on CPU, whereas every atomic read-modify-write bounces the cache line between all 14 cores — a hypothesis consistent with the data, not a measured cache analysis. (3) A serial queue is not a lock. It buys ordering, FIFO fairness and priority donation; it is not cheap mutual exclusion.
The benchmarking caveat that must travel with any of these figures. An earlier version of this benchmark, built with plain -O (no -wmo), reported Mutex.withLock at 50 ms uncontended — about 28× the true figure — because the generic Mutex<Int> never specialised. Rebuilt with -O -wmo it matched the unfair lock exactly, and -Onone inflated every primitive to ~82 ms. correction Any micro-benchmark of Synchronization types in a Swift script is measuring the optimiser, not the lock.
Swift · Mutex holding the state it protects
import Synchronization
final class SafeCounter: Sendable {
private let value = Mutex<Int>(0)
func increment() { value.withLock { $0 += 1 } }
var current: Int { value.withLock { $0 } }
/// Compound operations belong INSIDE one critical section.
func incrementAndRead() -> Int {
value.withLock { stored in
stored += 1
return stored
}
}
}Swift · OSAllocatedUnfairLock with an explicit state machine
import Foundation
import os
enum LoadState {
case idle
case loading
case ready(Data)
}
final class AssetBox: Sendable {
private let state = OSAllocatedUnfairLock(initialState: LoadState.idle)
func beginLoading() -> Bool {
state.withLock { current in
guard case .idle = current else { return false }
current = .loading
return true
}
}
func finish(with data: Data) {
state.withLock { $0 = .ready(data) }
}
}Verified against the compiler while writing this page: the bare lock() / unlock() methods on OSAllocatedUnfairLock exist only where State == (). Once the lock carries state you must go through withLock, which is exactly the safety Apple intends. Apple also warns that the manual pair “must call unlock() from the same thread you use to call lock(). Because of this, it’s unsafe to use this approach across an await suspension point.”
Swift · NSCondition with the mandatory predicate loop
import Foundation
final class BoundedQueue<Element>: @unchecked Sendable {
private let condition = NSCondition()
private var items: [Element] = []
private let capacity: Int
init(capacity: Int) { self.capacity = capacity }
func put(_ item: Element) {
condition.lock()
defer { condition.unlock() }
while items.count == capacity { // `while`, never `if`: signals can be spurious
condition.wait()
}
items.append(item)
condition.broadcast()
}
func take() -> Element {
condition.lock()
defer { condition.unlock() }
while items.isEmpty {
condition.wait()
}
let item = items.removeFirst()
condition.broadcast()
return item
}
}Swift · a POSIX readers-writer lock for a read-mostly index
import Foundation
final class ReadMostlyIndex: @unchecked Sendable {
private var lock = pthread_rwlock_t()
private var table: [String: Int] = [:]
init() { pthread_rwlock_init(&lock, nil) }
deinit { pthread_rwlock_destroy(&lock) }
func value(for key: String) -> Int? {
pthread_rwlock_rdlock(&lock)
defer { pthread_rwlock_unlock(&lock) }
return table[key] // concurrent with other readers
}
func set(_ value: Int, for key: String) {
pthread_rwlock_wrlock(&lock)
defer { pthread_rwlock_unlock(&lock) }
table[key] = value // exclusive
}
}3 · Replacing locks
When a serial queue or actor is the better lock — and when it is not.
“Use an actor instead of a lock” is true often enough to be a cliché and false often enough to be an interview question. The honest answer is a boundary condition: serialisation mechanisms replace locks only where the protected work can become asynchronous and the invariant does not have to hold across a suspension.
Replace the lock
Serial queue or actor wins
- The callers are already
async, or can become so without leaking through a synchronous public API. - The protected work is long enough that parking a whole thread would be wasteful — an actor suspends a task, not a thread.
- You want the compiler to check isolation rather than relying on a comment saying “call with the lock held”.
- Several pieces of state share one invariant. An actor makes the boundary a type; a lock leaves it a convention.
- You need priority to work through the boundary: a busy actor’s current task is elevated when higher-priority work arrives.
Keep the lock
Serialisation is the wrong shape
- The accessor must stay synchronous — a computed property, a
deinit, an AppKit override, a C callback, aSendableconformance on an existing class … unless you can prove you are already on that actor’s executor, in which caseassumeIsolatedgives synchronous access with a fatal-error precondition. - The critical section is nanoseconds. A lock’s uncontended cost is far below an executor hop.
- The invariant must survive across what would be an
await. Actor reentrancy explicitly does not protect that span. - You are inside a cancellation handler, a signal-safe path, or any context where you cannot suspend.
- You are bridging to Objective-C or C that has no concept of Swift isolation.
| Property | Lock (Mutex) | Serial dispatch queue | Actor |
|---|---|---|---|
| Blocks a thread on contention | Yes | Only if you call sync | No — suspends the task |
| Callable from synchronous code | Yes | Yes (sync, with deadlock risk) | No, except from inside the same isolation |
| Ordering among waiters | Unspecified; unfair locks may starve | Strict FIFO | Not FIFO; reordered by priority |
| Invariant holds across a suspension | n/a — cannot suspend while held | Yes within one block | No — reentrant at every await |
| Enforced at compile time | Only if the state lives inside the lock | No | Yes |
| Priority handled by the runtime | Ownership may resolve inversions | Boosts queued work ahead of it | Elevates the running task |
3AThe supported synchronous escape from an actorassumeIsolated · a precondition, not a conversion
What it is. “Assume that the current task is executing on this actor’s serial executor, or stop program execution otherwise … If that is the case, the operation is invoked with an isolated version of the actor, allowing synchronous access to actor local state without hopping through asynchronous boundaries. If the current context is not running on the actor’s serial executor … this method will crash with a fatal error.” documented (Actor.assumeIsolated, macOS 10.15)
Two things make it interview-grade rather than trivia. First, it is a precondition, not a conversion — it does not make a synchronous caller safe, it asserts the caller already is, and kills the process if not. The classic use is a @MainActor type reached from an AppKit callback the framework guarantees is on the main thread but the compiler does not know about. Second, the check is against the serial executor, not the actor identity: “if another actor uses the same serial executor … this check will succeed, as from a concurrency safety perspective, the serial executor guarantees mutual exclusion of those two actors.” A candidate who knows the check is executor-scoped understands what actor isolation actually is.
Do not confuse it with its debug sibling. assertIsolated is documented as: “In -O builds (the default for Xcode’s Release configuration), the isolation check is not performed and there are no effects.” documented That is a development assertion, not a shipping guarantee. preconditionIsolated is the always-on form.
Swift · reaching main-actor state from a framework callback
import AppKit
@MainActor
final class InspectorController {
private var rowCount = 0
// A framework callback the compiler does not know is main-thread-only.
// `assumeIsolated` is a PRECONDITION, not a conversion: it asserts the
// caller already is on this actor's serial executor and kills the process
// if not. The check is against the EXECUTOR, not the actor identity.
nonisolated func tableViewDidReload(_ table: NSTableView) {
MainActor.assumeIsolated {
rowCount = table.numberOfRows
}
}
}Row sources: Mutex, DispatchQueue, TaskPriority, os_unfair_lock_lock, TSPL — Concurrency, and WWDC21 · Behind the scenes for FIFO-versus-reentrancy. The table is our synthesis of those sources.
4 · Mistakes, follow-ups & recap
What goes wrong, what gets asked next, and the chapter in seven plain sentences.
No new terms. The section that earns each line is named beside it; the evidence behind every claim is in the execution lab, and the sources for this chapter are in H.
Where people go wrong
- Protecting the variable instead of the ruleLocking each field separately, or making each field atomic, leaves the moment when one has been updated and the other has not. A correctly-locked ledger that takes the lock twice — once to check, once to act — still goes negative. Hold it across both, or there is no invariant.
- Assuming a clean sanitizer means correctThread Sanitizer finds data races: unsynchronised access to the same location. It cannot see a race condition: correctly locked operations in a harmful order. A program that is clean under the sanitizer and still loses money is the standard interview trap, and it is reproducible on demand.
- Reaching for a reader-writer lock because reads are frequentIt is not “the lock for read-mostly data”. Below a read critical section of roughly a microsecond it is far slower than a plain mutex, because the bookkeeping costs more than the work it protects. Above it, it wins. Which side you are on is a measurement, not a preference.
- Calling
synconto a queue you may already be onA serial queue is not re-entrant the way a recursive lock is. The same thread re-entering the same lineage deadlocks instantly, and the diagnostic you get depends entirely on which primitive you chose — some name the problem, one hangs silently, two kill the process. - Doing expensive work inside the lockThe hold time of a lock is multiplied by every thread that wants it. Decode, transform and allocate outside; take the lock only to publish the result. This is the single change that fixes most “it got slower when we added workers” reports.
The follow-up you will get
Could an actor replace this lock?
Only if the protected work can become asynchronous and the rule does not have to hold across a suspension. An actor serialises access, but it lets other work interleave at every await — so an invariant that must survive the whole operation is not safe there, even though nothing races. Section 3 is the boundary condition, with a worked example that is data-race free and still wrong.
Which lock would you pick on a modern macOS target, and why not the others?
A mutex for new synchronous state, an actor for anything touching asynchronous code — and then name what you gave up. The unfair lock is cheaper and can starve a waiter by design; a semaphore excludes correctly but has no owner, so it cannot pass priority on; a recursive lock hides re-entrancy you would rather find. Saying what each one is bad at is how you show the choice was a choice.
How would you prove the fix, rather than assert it?
Reproduce first: a deterministic interleaving, a repeat count and a failing assertion. Then show the same program under the sanitizer, and finally the repair changing the evidence — no warning, the invariant held, and the throughput cost you accepted. A fix that only makes the symptom rarer is not a fix, and an interviewer will ask how you know.
Recap — what to carry out of this chapter
1. A lock protects a rule that must stay true, not a variable. Name the rule first and the storage second. (1–2)
2. A data race is two threads touching one location with no ordering between them and at least one writing. Making that one location atomic can leave the bug exactly where it was. (1)
3. Atomicity and ordering are two different promises. An atomic gives you the first; you only get the second if you ask for it. (1)
4. There is no best primitive, only a shelf. The answer that earns points names what each one is bad at, which is how you show you chose rather than reached. (2)
5. On a modern macOS target the defaults are a mutex for new synchronous state and an actor for anything that touches asynchronous code. Everything else needs a reason. (2)
6. A serial queue or an actor replaces a lock only where the protected work can become asynchronous and the rule does not have to hold across a suspension. (3)
7. The cheapest lock is the one held for the least time. Do the expensive transform outside it, then take the lock to publish the result. (2F, 3)
How to read the labels on every claim in this chapter
documented — Apple documentation, an official WWDC or Tech Talk transcript, Swift Evolution, or a Darwin manual page or SDK header on macOS 26.3. Quoted verbatim with the source named beside it.
measured here — observed on one machine: Apple M4 Pro (10 performance + 4 efficiency cores), macOS 26.3 (25D125), 48 GB, 16 KiB pages, Swift 6.2.4, Apple clang 17.0.0, Xcode 26.3. Evidence of a mechanism, never a benchmark. Reproduce; do not quote.
inference — our reasoning joining the two above. Marked so you can disagree with the reasoning without doubting the evidence.
negative result — something that was tried and did not show what was expected. These bound what a chapter is allowed to claim, and they are kept deliberately visible.
correction — a correction to an earlier reading published on this page, naming what was wrong and why. This page’s whole method is labelled provenance, so its own errata carry a label too rather than disappearing into a rewrite.
version-sensitive — observed behaviour of this OS build that must be re-measured on another before it is relied on.
Next: chapter 7 is what happens when these choices stop the program making progress — nine distinct pathologies, nine different repairs.
Chapter 7 · deadlocks, livelocks and starvation
Deadlocks, livelocks, and starvation.
Chapter 6 chose the primitives. This chapter is every way that choice can stop a program making progress — and, because “it hangs” is the same sentence for all of them, the single observation that tells each one apart from the other eight.
The problem, in plain words
Three different things all look like “the app is stuck”, and they need opposite repairs.
Deadlock is waiting in a circle: you hold something I need, I hold something you need, and neither of us will let go. Nobody uses any processor time, and nobody ever will again.
Livelock is the opposite picture with the same result: everybody is busy, everybody keeps politely backing off and trying again at the same moment, and no one ever gets through. The fans spin up and nothing finishes.
Starvation is one participant never getting a turn while the others keep going. Nothing is broken; the rules simply never favour that one.
What breaks without it. Without the distinction, every one of these gets the same wrong fix — usually “add a timeout” or “add more threads” — and the real one comes back under a different load.
How to recognise it
Every report in this chapter arrives as the same sentence: it hangs. What separates the nine mechanisms behind it is not the symptom but a single cheap observation, and taking that observation before you theorise is the whole skill.
Start with processor time. Zero CPU means everyone is parked: look for a cycle, or for a pool with nothing left to run on. High CPU with no progress means nobody is parked at all: that is a livelock, and no amount of lock discipline will help. Normal CPU with one participant never finishing is starvation, and the lock is behaving exactly as documented.
The tell: take a sample during the freeze and look at two threads at once. The same frame in every sample on two threads, each waiting on what the other holds, is a deadlock and nothing else. If the stacks churn instead — different frames every sample, all of them retry and back-off — you are looking at a livelock wearing a deadlock’s complaint.
The idea
A deadlock needs four conditions at once, and breaking any one of them removes it: the resources are exclusive, holders keep what they have while waiting for more, nothing can be taken away by force, and the waiting forms a circle. Almost every practical repair attacks the last one, because it is the only one you can state as a rule people can follow: always take these locks in this order.
Attacking “no preemption” instead — grab a lock, fail, drop everything, retry — is what turns a deadlock into a livelock. Two threads doing that in lockstep make no progress and look busy while doing it, which is strictly worse, because now the symptom no longer even points at a lock.
And not every wedge involves a lock at all. Enough blocked tasks can consume every thread in a cooperative pool with nothing held and no cycle anywhere: the fix is to express the dependency in the task graph, not to find the lock that is not there.
Watch it run
Two threads, two locks, opposite order
A money transfer locks the source account, then the destination. Two transfers run in opposite directions. Both pass review; both pass their tests.
acquired, and held
asking for a lock
parked forever
the repair
Transfer 1wants A then Bmoving money from account A to account B
Transfer 2wants B then Amoving money from account B to account A
LocksA free · B freenothing held; every acquisition below succeeds or waits
Step 1. Neither thread is wrong on its own. The defect is that they disagree about the order, and nothing in the language or the review checklist notices.
Transfer 1holds Anow asks for B
Transfer 2not started—
LocksA held · B freeon most runs transfer 1 simply finishes here
Step 2. Almost every execution goes this way, which is exactly why the bug reaches production: it needs the two threads to interleave.
Transfer 1holds Aabout to ask for B
Transfer 2holds Babout to ask for A
LocksA held · B heldboth conditions are now in place
Step 3. The interleaving that matters. Each thread holds what the other is about to need, and neither will release before it gets it.
Transfer 1parked on Bheld by transfer 2, which will never release it
Transfer 2parked on Aheld by transfer 1, which will never release it
Evidence0% CPUthe same wait frame in every sample, on both threads
Step 4. The circle closes. Nothing crashes, nothing logs, nothing times out; the process simply stops, and it will still be stopped tomorrow.
That signature — no processor time, and one identical frame on two threads at once — is what distinguishes this from every other entry in the catalogue.
Transfer 1takes A, then Bproceeds and releases both
Transfer 2waits for Athen takes A, then B, and proceeds
Ruleone total orderevery site acquires in the same order, always
Step 5. The repair is an ordering rule, not a timeout. Both threads now take the lower-numbered account first, so no circle can form — and the tradeoff is that the order becomes a global rule nothing checks, which a third lock added later can quietly break.
Section 1 catalogues the other eight mechanisms and their distinguishing evidence; the execution lab’s exercise 01 ships this deadlock with its backtrace.
The code
Three of the six broken programs in the execution lab are this chapter’s, each with a deterministic symptom and a watchdog so a wedged run ends by itself:
How this chapter is built. The same eight steps as every other chapter: overview in plain words (above) → mental model and vocabulary (1, the wait-for-graph and Coffman diagrams) → Darwin mechanism and where the public contract stops (1, the nine-pathology table and the self-deadlock trap matrix) → deeper internals (1A) → Mac lab (C, and exercises 01, 04 and 05 in D) → failure story and diagnosis workflow (A, B, E) → interview questions (F, G) → mistakes, follow-ups and recap (2).
Which of the nine pathologies this is, and the one observation that rules out the other eight.
Reproduce it deterministically, then show the repair changing the evidence — not just the symptom going quiet.
Formal deadlock-detection algorithms, the banker’s algorithm, wait-for-graph tooling, model checkers.
The sentence to have ready
“‘It hangs’ is a symptom, not a diagnosis.” Say which mechanism you think it is and, in the same breath, the observation that would distinguish it from the others — zero CPU versus high CPU, a cycle versus no cycle, someone parked versus nobody parked. That one sentence is the difference between a mid-level and a senior answer.
1 · Failure catalogue
Nine pathologies, nine different repairs.
“The app froze” is a symptom shared by many distinct mechanisms. An answer that names which one, and the observation that distinguishes it from the others, is the difference between a mid-level and a senior response. Every row below follows the same shape: symptom → mechanism → evidence → repair → tradeoff.
| Pathology | Mechanism | Evidence that identifies it | Repair | Tradeoff of the repair |
|---|---|---|---|---|
| Deadlock (ABBA) | A cycle in the wait-for graph. All four Coffman conditions hold. | 0% CPU, and the same frame in 100% of samples on two threads at once. measured here 92 of 92 samples in __psynch_mutexwait. | One total lock order, obeyed at every acquisition site. | The order becomes a global invariant nothing checks. A third lock added later can reintroduce the cycle. |
| Self-deadlock / re-entry | One thread asks for something it already holds, directly or through a target chain. | Depends entirely on the primitive — see the trap matrix below. Some trap with a named diagnostic, one hangs silently, two kill the process. | Never sync onto a lineage you may already own; use a lock, not a queue, for a synchronous accessor. | None worth the name. A trap with a diagnostic is strictly easier than a hang. |
| Cooperative-pool wedge | N blocked tasks where N ≥ core count. No lock is held and no cycle exists. | measured here 18 detached tasks blocking: exactly 14 got threads (= activeProcessorCount), 4 never scheduled, the releaser never ran. | Express the dependency in the task graph — await a child, a TaskGroup, or a continuation. | The compiler blocks the naive form, so this arrives through a synchronous helper that hides the wait. |
| Livelock | Threads acquire-fail-release-retry in lockstep. Breaking “no preemption” with a naive trylock loop is the classic cause. | High CPU, no forward progress, no thread ever parked. Stacks churn but only through retry/back-off frames. | Blocking acquisition in a stated total order; if you must retry, randomised back-off and a bounded attempt count. | Back-off adds latency variance and makes the code non-deterministic to test. |
| Starvation | An unfair lock permits it by design; a writer-preferring rwlock de-prioritises readers by design. | One participant’s work never appears while others complete. negative result Measured on Darwin, a reader against four continuous writers still completed 43,871 reads in 500 ms. | A fair primitive (serial queue = strict FIFO, or FAIRSHARE), or remove the hot path from the shared lock. | Fairness cost ~180× throughput here. Say “permits starvation”, not “causes” it. |
| Priority inversion | Low-priority work holds a resource high-priority work needs, and the primitive has no owner to raise. | Thread Performance Checker reports it directly. In a trace: holder stays at band 4 on E cores while a band-31 thread is Blocked. measured here 277 ms vs 1017 ms for the same critical section. | Prefer owner-bearing primitives; or ensure the waiter’s QoS is ≤ the signaller’s. | Donation makes the lock fast, not the design right — boosted background work still burns the user’s battery on a P core. |
| Convoying | Fair handoff forces a context switch on every single acquisition; the unlocker cannot reacquire. | High involuntary context-switch count, short on-CPU bursts, throughput far below what the critical section’s length predicts. measured here ~0.42 switches per acquisition vs ~0.002. | Prefer the unfair lock; shorten hold time; do the expensive transform outside the lock. | Unfairness “makes starvation of waiters a possibility” — you trade a bounded-wait guarantee for throughput. |
| Lost / missed wakeup | A condition variable stores no permits. A signal() delivered while nobody is parked is discarded. | measured here 3 of 3: three seconds after signal() the broken consumer is still parked in wait(). | Lock → while !predicate { wait() } → act. The predicate is the state. | One predicate re-test per wakeup — free, compared to a lost wakeup. |
| Thundering herd | broadcast() wakes every parked waiter; all contend for one lock; all but one re-park. | measured here 8 waiters, one predicate flip: signal() → 1 proceeded, broadcast() → 8 proceeded. | signal() when the change can satisfy exactly one waiter; broadcast() when it can satisfy an unknown number. | The real dilemma: signal() risks a lost wakeup when waiters test different predicates; broadcast() pays the herd. |
Simplified diagram. “Lock ordering” means a total order you can state in one sentence and enforce at every acquisition site — for example, always by object identifier, or always outer-container before inner-element.
Simplified diagram. measured here The deadlock is made deterministic — not a race you loop to hit — by a hand-rolled rendezvous gate every thread reaches after taking its first lock and before reaching for its second, so it reproduces 100% of the time. sample needed no elevated privileges for a same-user process.
| Re-entry case | Documented behaviour | Observed on macOS 26.3 | Exit | Diagnostic string in the crash report |
|---|---|---|---|---|
q.sync { q.sync { } } on a serial queue | “results in deadlock” | SIGTRAP | 133 | BUG IN CLIENT OF LIBDISPATCH: dispatch_sync called on queue already owned by current thread |
child.sync { root.sync { } } through a target chain | “results in deadlock” | SIGTRAP | 133 | same — note the two call sites name different queue objects |
DispatchQueue.main.sync { } from the main thread | “results in deadlock” | SIGTRAP | 133 | same |
NSLock().lock(); lock() | “will lock up your thread permanently” | hangs forever | watchdog | none |
NSRecursiveLock().lock(); lock() | Permitted | completes | 0 | — |
OSAllocatedUnfairLock.withLock re-entered | “triggers a runtime exception” | SIGKILL | 137 | BUG IN CLIENT OF LIBPLATFORM: Trying to recursively lock an os_unfair_lock |
Mutex.withLock re-entered | “platform dependent … panic the process, deadlock, or leave this behavior unspecified” | SIGKILL | 137 | identical to the row above — on Apple platforms it is the panic |
Main thread in DispatchSemaphore.wait(), only main.async can signal | — | silent permanent hang | watchdog | no trap, no crash report |
measured here documented This is the chapter’s most consequential modernisation, and it is a divergence, not a replacement. Apple’s prose still says “results in deadlock”; macOS 26.3 traps with a named diagnostic. Report both. Trapping is an observation, not a documented contract, and could change. inference The reason libdispatch can do it is ownership: a serial queue has a single known owner, so the runtime can see that the calling thread already owns the queue it is synchronising onto. It cannot see a cycle mediated by an ownerless primitive — which is exactly why the last row is silent.
Simplified diagram. All three panels are reproduced by bounded fixtures. measured here The third is the one the chapter most needs: a wedge with no lock and no cycle at all. Apple’s statement of the contract it violates: “Swift concurrency requires tasks to make forward progress when they’re running … In extreme cases, when the entire thread pool is occupied by blocked tasks, and they’re waiting on something that requires a new task to run on the thread pool, the concurrency runtime can deadlock.” documented (WWDC22 110350 · Visualize and optimize Swift concurrency, 2022)
| # | Coffman condition | What it means | How Apple-platform code breaks it |
|---|---|---|---|
| 1 | Mutual exclusion | The resource cannot be shared | Make the data immutable and swap a pointer atomically; use a value type and copy |
| 2 | Hold and wait | A holder requests more while holding | Acquire everything at once, or copy out under lock 1, release, then take lock 2 |
| 3 | No preemption | The system cannot revoke a held resource | withLockIfAvailable / trylock plus full back-off and restart — careful: this trades deadlock for livelock |
| 4 | Circular wait | A cycle exists in the wait-for graph | The practical one. Impose one total order over all locks and obey it at every acquisition site |
All four must hold simultaneously; breaking any one prevents deadlock. inference This is textbook operating-systems material (Coffman, Elphick and Shoshani, 1971), not an Apple claim, and is labelled as such. The interview answer that lands: “I break circular wait, because it is the only one of the four I can state as a one-sentence rule and enforce at every call site — for example ‘always take the lower object identifier first’ — and I can assert it in debug builds with os_unfair_lock_assert_owner, which Apple documents as: if the lock ‘is unlocked or owned by a different thread, this function asserts and terminates the process.’” documented
1ABoth repairs, in C, with the cost of eachtotal order vs drop-everything backoff
Both were measured on the same fixture. A total order completed 2000 paired acquisitions per thread with no stall. Trylock-with-backoff also completed 2000, with only 1–3 backoffs. measured here That small number is the interesting part: it is why the backoff repair looks attractive, and it is still the weaker choice, because its worst case is unbounded and it can livelock, whereas a total order is a static property you can assert.
Apple’s warning about the naive form of repair 2, which is the whole reason it is second choice: “Do not attempt to call this function within a retry loop; os_unfair_lock_lock accomplishes the same task, without hiding the lock waiter from the system or preventing resolution of priority inversions.” documented The second half of that sentence is the deeper point — a spin-retry loop also destroys the ownership information the system needs to fix a priority inversion.
What happened when the same repair was attempted in Swift. Nesting two state-carrying scoped locks does not compile: error: mutable capture of 'inout' parameter 'first' is not allowed in concurrently-executing code. measured here inference Read that as the type system telling you something true: hold-and-wait across two scoped locks is a shape to restructure, not to spell more cleverly.
C · the deadlock and both repairs (deliberately-wrong code, bounded by a watchdog)
/* UNSAFE BY DESIGN. Bounded: a watchdog thread forces _exit(75) after 5s. */
#include <pthread.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
static void rendezvous(void); /* hand-rolled two-thread gate */
static pthread_mutex_t lock_a = PTHREAD_MUTEX_INITIALIZER;
static pthread_mutex_t lock_b = PTHREAD_MUTEX_INITIALIZER;
/* DEADLOCK: opposite acquisition orders. The gate makes it fire 100% of the
time instead of being a race you loop to hit. */
static void *thread_a(void *_) {
pthread_mutex_lock(&lock_a);
rendezvous(); /* both threads hold their first lock */
pthread_mutex_lock(&lock_b); /* <-- blocks here forever */
pthread_mutex_unlock(&lock_b); pthread_mutex_unlock(&lock_a);
return NULL;
}
static void *thread_b(void *_) {
pthread_mutex_lock(&lock_b);
rendezvous();
pthread_mutex_lock(&lock_a); /* <-- blocks here forever */
pthread_mutex_unlock(&lock_a); pthread_mutex_unlock(&lock_b);
return NULL;
}
/* REPAIR 1 - total order (breaks circular wait). Order by address: any stable
total order works, and this one needs no extra field on the object. */
static void transfer_ordered(pthread_mutex_t *x, pthread_mutex_t *y) {
pthread_mutex_t *first = x < y ? x : y, *second = x < y ? y : x;
pthread_mutex_lock(first); pthread_mutex_lock(second);
/* ... both held: the paired update is atomic ... */
pthread_mutex_unlock(second); pthread_mutex_unlock(first);
}
/* REPAIR 2 - trylock with drop-everything backoff (breaks hold-and-wait).
Measured: 2000 paired acquisitions with only 1-3 backoffs. Still the weaker
repair: its worst case is unbounded and it can livelock. */
static int transfer_backoff(pthread_mutex_t *x, pthread_mutex_t *y, int *backoffs) {
for (;;) {
pthread_mutex_lock(x);
if (pthread_mutex_trylock(y) == 0) {
/* ... both held ... */
pthread_mutex_unlock(y); pthread_mutex_unlock(x);
return 0;
}
pthread_mutex_unlock(x); /* drop EVERYTHING, do not hold and wait */
(*backoffs)++;
usleep(50 + (unsigned)(random() % 200)); /* randomised, or you lockstep */
}
}
/* The real two-thread gate lives in the E1 fixture; this stub keeps the
excerpt self-contained so it compiles on its own. */
static void rendezvous(void) { }Double-checked locking — do not hand-roll it
The unlocked first check reads a pointer with no ordering, so a reader can see a non-nil pointer whose pointee writes are not yet visible. Swift ships the supported primitive: AtomicLazyReference (macOS 15) — “these values can be set (initialized) exactly once, but read many times.” Below macOS 15, a Swift let global already gives it: “Global variables in Swift are initialized lazily.” documented
Tradeoff: AtomicLazyReference can construct the value more than once under a race and discard the loser, so the initialiser must be side-effect-free.
Check-then-act is the same bug without the laziness
Two individually-atomic calls with a gap between them. The repair is always the same: make the decision and the action one critical section. The API lesson is sharper — do not export a predicate whose truth cannot survive the caller’s next statement. Export the compound operation. inference
Make a forward-progress violation deterministic
Setting LIBDISPATCH_COOPERATIVE_POOL_STRICT=1 narrows the cooperative pool to a single thread, so the first blocking task wedges immediately instead of only on a loaded 14-core machine. measured here It turns “a hang our CI sees once a month” into “a test that fails every time”.
Label this carefully. The behaviour is demonstrated and the name is verified in this machine’s dyld shared cache, but it appears in no Apple documentation page. It is a development and CI aid, not an API, and not something to build a workflow on.
GCD does not avoid the wedge — it postpones it
The same blocking shape on a GCD global queue survives to 64 waiters and fails at 128, paying for the postponement with 70 live threads. Swift’s pool fails at 14 instead, but it fails early and loudly. measured here inference Neither model makes blocking safe; only one makes it detectable in development.
Swift · a two-lock deadlock and the ordering that removes it
import Foundation
final class Account: @unchecked Sendable {
let id: Int
let lock = NSLock()
private var balance: Int // guarded by `lock`
init(id: Int, balance: Int) { self.id = id; self.balance = balance }
func adjustLocked(_ delta: Int) { balance += delta } // caller must hold `lock`
}
/// Deadlocks when two transfers run in opposite directions at the same time:
/// thread 1 holds A and wants B, thread 2 holds B and wants A.
func transferDeadlockProne(_ amount: Int, from a: Account, to b: Account) {
a.lock.lock(); defer { a.lock.unlock() }
b.lock.lock(); defer { b.lock.unlock() }
a.adjustLocked(-amount)
b.adjustLocked(amount)
}
/// Same work under one global order: always take the lower id first.
/// A cycle in the wait-for graph is now impossible.
func transfer(_ amount: Int, from a: Account, to b: Account) {
precondition(a.id != b.id, "self-transfer would re-enter a non-recursive lock")
let (first, second) = a.id < b.id ? (a, b) : (b, a)
first.lock.lock(); defer { first.lock.unlock() }
second.lock.lock(); defer { second.lock.unlock() }
a.adjustLocked(-amount)
b.adjustLocked(amount)
}Granularity: one dial, two failure modes
One coarse lock is easy to reason about and serialises everything. Many fine locks reduce contention and multiply the ordering rules you must never break. Sharding — N locks chosen by key hash — is the usual middle, and it works only because a shard is never allowed to take another shard’s lock.
Hold time beats lock count
Contention is a function of how long the lock is held, not how many threads exist. Copying data out under the lock and doing the expensive transform outside it is usually a bigger win than splitting the lock. Measure the held interval before you redesign the ownership.
Never call out under a lock
Do not invoke a delegate, a completion handler, a notification, or unknown client code while holding a lock. You do not control what it acquires, so you cannot reason about ordering. Capture what you need, unlock, then call out.
Priority inversion has a concrete fix list
Apple’s: do not use dispatch_semaphore_wait or dispatch_group_wait “to emulate synchronous behavior when calling an asynchronous internal method or API,” and where no synchronous variant exists, “ensure that the QoS of the waiting thread is the same as or lower than the QoS of the signaling thread.”
Source: Diagnosing performance issues early.
Swift · the semaphore anti-pattern Thread Performance Checker flags, and its repair
import Foundation
enum Loader {
/// WRONG: blocks the caller's thread waiting on lower-priority work.
/// The system cannot propagate the waiter's priority across a semaphore.
static func settingsSynchronously() -> Data {
let semaphore = DispatchSemaphore(value: 0)
var result = Data()
DispatchQueue.global(qos: .background).async {
result = fetchSettings()
semaphore.signal()
}
semaphore.wait()
return result
}
/// RIGHT: stay asynchronous; the dependency is expressed in the task graph,
/// so the runtime can escalate the priority of the work being awaited.
static func settings() async -> Data {
await withCheckedContinuation { continuation in
DispatchQueue.global(qos: .userInitiated).async {
continuation.resume(returning: fetchSettings())
}
}
}
private static func fetchSettings() -> Data { Data() }
}Compiled while writing this page: under Swift 6 language mode the incorrect branch also draws warning: mutation of captured var 'result' in concurrently-executing code. The compiler is telling you about the data race; the Thread Performance Checker tells you about the inversion. They are two different defects in six lines.
2 · Mistakes, follow-ups & recap
What goes wrong, what gets asked next, and the chapter in seven plain sentences.
No new terms. The evidence behind every claim is in the execution lab, and the sources for this chapter are in H.
Where people go wrong
- Adding a timeout and calling it fixedA timeout converts a permanent hang into an intermittent failure and destroys the evidence that would have identified which of the nine mechanisms you had. Use one as a watchdog in a fixture, never as a repair in production code.
- Breaking “no preemption” with a retry loopGrab, fail, release everything, try again is the classic route from a deadlock to a livelock. Two threads doing it in step burn processor time forever and look busy while doing it. If you must retry, randomise the back-off and bound the attempts — but a stated total order is better.
- Assuming a hang means a lockEnough blocked tasks exhaust a cooperative pool with nothing held and no cycle anywhere. Eighteen detached tasks blocking on a semaphore: exactly as many as there are cores got threads, and the one that would have released them never ran.
- Signalling a condition variable with nobody waitingA semaphore stores permits; a condition variable stores nothing. A signal delivered into an empty waiting room is gone forever, and the consumer parks for good. The predicate loop is not defensive style — it is the contract.
- Treating starvation as a defect in the lockAn unfair lock permits starvation by design, and that unfairness is what makes it fast. The repair is a fair primitive or a shorter shared path, chosen deliberately with the throughput cost named — not a bug report against the primitive.
- Diagnosing from one threadA single backtrace shows you a thread that is waiting, which is not news. The diagnosis lives in the relationship between two of them, so read the whole process and ask who holds what.
The follow-up you will get
Your lock order works. What happens when someone adds a third lock next year?
That is the honest weakness of the repair, and saying so is the senior answer. A total order is a global rule that nothing in the type system or the compiler checks, so it survives only as long as everyone keeps obeying it. Write it down next to the locks, add an assertion that fires in debug when acquisition happens out of order, and prefer designs with one lock over designs that need a documented ordering at all.
How would you tell a livelock from a deadlock without a debugger?
Processor time, then two samples a second apart. A deadlock is at zero and the stacks are identical between the samples. A livelock is busy and the stacks move — but only through retry and back-off frames, never through the work the program is supposed to be doing. Both are “it hangs”; the two observations separate them in under a minute.
Could this have been prevented by design rather than discipline?
Usually yes, and it is worth offering. Take one lock instead of two by making the pair of accounts a single guarded object; or move the operation behind one serial queue so no second acquisition exists; or express the dependency as a task graph the runtime can see, which is what removes the cooperative-pool wedge as well. The repairs in this chapter are the ones you need when the design is already shipped.
Recap — what to carry out of this chapter
1. “The app froze” is a symptom shared by nine distinct mechanisms, and each one has a different repair. Naming the mechanism is the answer; naming a fix is not. (1)
2. A deadlock is a cycle of waiting: no processor time at all, and the same frame in every sample on two threads at once. (1)
3. A livelock is the opposite reading of the same symptom: plenty of processor time, no forward progress, and nobody ever parked. (1)
4. Starvation is not a defect in the lock. An unfair lock permits it by design, and the repair is a fair primitive or a shorter shared path — not a bug report. (1)
5. Not every wedge involves a lock. Enough blocked tasks can consume the whole cooperative pool with nothing held and no cycle anywhere. (1)
6. A condition variable stores no permits: a signal delivered while nobody is waiting is simply gone. That is why the predicate loop is mandatory rather than defensive. (1)
7. Repair the wait, not the symptom — one total lock order, a primitive that has an owner to raise, or the dependency expressed in the task graph. A timeout hides all three. (1, 1A)
How to read the labels on every claim in this chapter
documented — Apple documentation, an official WWDC or Tech Talk transcript, Swift Evolution, or a Darwin manual page or SDK header on macOS 26.3. Quoted verbatim with the source named beside it.
measured here — observed on one machine: Apple M4 Pro (10 performance + 4 efficiency cores), macOS 26.3 (25D125), 48 GB, 16 KiB pages, Swift 6.2.4, Apple clang 17.0.0, Xcode 26.3. Evidence of a mechanism, never a benchmark. Reproduce; do not quote.
inference — our reasoning joining the two above. Marked so you can disagree with the reasoning without doubting the evidence.
negative result — something that was tried and did not show what was expected. These bound what a chapter is allowed to claim, and they are kept deliberately visible.
correction — a correction to an earlier reading published on this page, naming what was wrong and why. This page’s whole method is labelled provenance, so its own errata carry a label too rather than disappearing into a rewrite.
version-sensitive — observed behaviour of this OS build that must be re-measured on another before it is relied on.
Next: the execution lab — the diagnosis workflow, six trace signatures, nine runnable experiments, six broken programs to repair, a four-stage incident drill and twenty interview questions, shared by chapters 5, 6 and 7.
Shared block · chapters 5, 6 and 7
The execution lab.
One diagnosis workflow, six trace signatures, nine runnable experiments, six broken programs to repair, a four-stage incident drill, twenty interview questions, a drill and the primary sources — serving threading and scheduling, locks and deadlocks together. They are together on purpose: in a real incident you do not know which of the three you have until you have measured, and the tool order is the same for all of them. Nothing here is numbered into one chapter. The sections are lettered A to H, and all three chapters link into them.
Which chapter sends you where
From chapter 5 · threading and scheduling: experiments E3 (bounded parallelism), E6 (priority donation) and E8 (thread explosion); exercises 03 and 06; trace signatures C1, C4, C5 and C6.
From chapter 6 · locks and synchronization: experiments E2 (a data race and a race condition), E4 (the predicate loop) and E9 (contention scaling); exercise 02; trace signatures C2 and C3.
From chapter 7 · deadlocks, livelocks and starvation: experiments E1 (ABBA), E4 (lost wakeup) and E7 (the task-and-semaphore deadlock); exercises 01, 04 and 05; trace signatures C2 and C3, and the whole of the incident drill.
A · Diagnosis
Choose the instrument from the symptom.
Every tool below answers exactly one question. Saying which question you are asking — before naming a tool — is the behaviour the interview is scoring.
| Question you are asking | Tool | Cost, limit, or caveat |
|---|---|---|
| Is the main run loop unresponsive, and for how long? | Instruments — Hangs (an instrument, present in the Time Profiler, CPU Profiler and Hitches templates) | 250 ms default; Instruments 26 records the setting literally as Include Microhangs (>250ms). Requires Instruments 14 / macOS 13 or later. |
| Where is CPU time going? | CPU Profiler — prefer this over Time Profiler | Apple: “You should prefer CPU Profiler over Time Profiler for CPU optimization because it’s more accurate and more fairly weights software consuming CPU resources.” Time Profiler samples on a timer and suffers aliasing plus a bias against fast-clocking cores. |
| Exactly which instructions ran, with no sampling bias? | Processor Trace | M4 / A18 silicon only; Instruments 16.3+; ~1% overhead; must be enabled in Privacy & Security → Developer Tools. A specialist tool, not a default. |
| Is a thread running, ready-but-waiting, or blocked — and who woke it? | Thread State Trace (in the System Trace template) | The only tool that shows Runnable and Preempted at all. Its Narrative names the waking thread and core, or “a timer expiration”. |
| Which actor or task is this work on? | Swift Tasks / Swift Actors (Swift Concurrency template) | Reads task and actor identity, not thread identity. Needs a build with the concurrency runtime. |
| Is anything high-priority waiting on anything low-priority? | Thread Performance Checker | macOS and iOS only; no recompile; on by default for the Run action. Suppress known cases with PERFC_SUPPRESSION_FILE; disable it before you trust profile numbers. |
| Do two threads touch this memory unsynchronised? | Thread Sanitizer | “Increases memory usage by five to ten times … 2x to 20x slowdown.” macOS or Simulator only. It is not a passive observer — it changes observable timing. And it does not support fences, where it may report false positives. |
| Is AppKit/UIKit being touched off the main thread? | Main Thread Checker | “Adds 1–2% CPU overhead … increases process launch time by no more than 100 milliseconds.” |
| Is the main thread busy or blocked, right now, without Instruments? | sample | “Suspends the process at specified intervals (by default, every 1 millisecond), records the call stacks of all threads.” Sample count measures how long the thread existed, not how much CPU it used — read the leaf frame, not the number. |
| Who is on the other end, system-wide? | spindump -onlyBlocked / -onlyRunnable | Needs privilege: run unprivileged against a live target it exits 77 (EX_NOPERM) and writes no report at all, not a partial one. Use -onlyTarget for faster sampling rates. |
| How long is my critical section held, versus waited for? | OSSignposter, two separate intervals | You must instrument the code; an interval “consists of one begin call and one end call only”. |
| Can I make a forward-progress violation reproduce every time? | LIBDISPATCH_COOPERATIVE_POOL_STRICT=1 | Narrows the cooperative pool to one thread. Undocumented, development and CI only — the behaviour is demonstrated and the name verified on this machine, but it appears in no Apple documentation page and may change. |
documented Quoted behaviour from Diagnosing memory, thread, and crash issues early, Diagnosing performance issues early, Improving app responsiveness, OSSignposter, WWDC25 308 · Optimize CPU performance with Instruments (2025, ?time=530 for the CPU Profiler passage and ?time=845 for Processor Trace), and the macOS 26.3 manual pages sample(1) and spindump(8). measured here for the spindump exit code, the tool inventory (xcrun xctrace list templates / list instruments on Instruments 26.0), and the strict-pool behaviour.
Hangs are three different phenomena, not one
Apple splits them, and each has its own signature and its own next tool: a busy main thread hang (“the main thread will show a bunch of CPU activity”), an asynchronous hang (work correctly moved off the main thread but then awaited on it), and a blocked main thread hang (“there will be little to no CPU activity on the main thread”). documented
And the caveat a busy/blocked split needs, or you will call an idle app a hang: “Here, the main thread is just asleep because there was no user input. From the operating system’s perspective, it is blocked, but it’s just saving resources … So to determine whether a blocked thread is a responsiveness issue or not, look to the Hangs instrument, not the thread states instrument. So a blocked main thread does not imply an unresponsive main thread. Similarly, High CPU Usage also doesn’t imply that the main thread is unresponsive. But if the main thread is unresponsive, that means it was either blocked or the main thread was busy.” documented (WWDC23 10248 · Analyze hangs with Instruments, 2023 — busy ?time=555, asynchronous ?time=1552, blocked ?time=2157, this caveat ?time=2435)
Where the 250 ms number comes from: “Most of Apple’s developer tools start reporting issues when the period of unresponsiveness for the main run loop exceeds 250 ms … Hang reporting only measures the time on the main run loop. The time necessary for event delivery … and the time to render the new screen … can add 10 ms to 50 ms to the overall delay.” documented
Leaf frame in a sample report | What that thread is doing |
|---|---|
__psynch_mutexwait under _pthread_mutex_firstfit_lock_wait | Blocked on a pthread_mutex — and therefore also on NSLock |
_dispatch_semaphore_wait_slow · in libdispatch | Blocked on a DispatchSemaphore. The different library is the whole tell. |
__ulock_wait | os_unfair_lock, or pthread_join |
__semwait_signal under nanosleep | Sleeping on a timer — not blocked on anything you own |
__workq_kernreturn | An idle Dispatch worker thread parked waiting for work |
| A frame in your own binary, with a source line | Actually on a core |
measured here The trap this table exists to defuse. In one run, four blocked threads each showed 277 samples — the full window — while the one genuinely busy thread showed 137. Sample count measures thread lifetime, not CPU use. Read the leaf frame. One free practical detail: pthread_setname_np names do appear in sample output (Thread_14646510: MUTEXWAIT) but did not appear in the Instruments Thread State Trace export. Name your threads anyway — it is free, and it is the difference between a readable and an unreadable report.
# 1. Is it busy or blocked? 3 seconds at 1 ms, all threads, to a file.
$ sample FrameworkLab 3 1 -file /tmp/fl-hang.txt
# 2. Who is on the other end? System-wide, blocked stacks only.
$ sudo spindump FrameworkLab 5 10 -onlyBlocked -noBinary -o /tmp/fl-spin.txt
# 3. Confirm a suspected race, macOS target or Simulator only.
$ xcodebuild test -scheme FrameworkLab -enableThreadSanitizer YES
# 4. Re-measure without the checker's own overhead in the stacks.
# Product > Profile, or launch from Instruments, not Run.Honest scope: the sample invocation above was run against a throwaway process on this machine to confirm the flag syntax and that it produces a report without elevated privileges; spindump was run the same way and exited without a report when unprivileged, which is why step 2 shows sudo. No Instruments session, no Thread Sanitizer run, and no profiling of a real AppKit app was performed for this page, and no timings are claimed beyond the race demonstration above.
Swift · signpost the wait and the held span separately
import Foundation
import os
import Synchronization
let signposter = OSSignposter(subsystem: "com.example.FrameworkLab", category: "locking")
final class TimelineStore: Sendable {
private let rows = Mutex<[String]>([])
func append(_ row: String) {
let id = signposter.makeSignpostID()
let waitState = signposter.beginInterval("lock-wait", id: id)
rows.withLock { storage in
signposter.endInterval("lock-wait", waitState)
let heldState = signposter.beginInterval("lock-held", id: id)
storage.append(row)
signposter.endInterval("lock-held", heldState)
}
}
}Two intervals, not one. A long wait means contention — someone else holds it too long. A long held span means your critical section is doing too much. The repairs are opposite, so measuring them together tells you nothing.
B · Trace signatures
Six things a report can look like, and what each one means.
Every case below is symptom → mechanism → evidence → repair → tradeoff, with an explicit split between what the tool shows and what you infer. That split is the thing being scored: a tool never says “priority inversion”, and a candidate who claims it did is guessing.
C1Runnable but unschedulednothing is hot, everything is slow
Symptom. Wall-clock latency is several times the CPU time. Profiles look “fine” — nothing is hot.
Mechanism. More runnable threads than cores. Each thread’s share of a core shrinks proportionally.
Evidence — what the tool shows. Thread State Trace. At 14 threads on 14 cores, Preempted is 19.7% of interval time and (Preempted + Runnable) ÷ Running is 24.7%. At 42 threads on the same machine it is 71.0% and 262.1%, with Running essentially unchanged. measured here This is the one case sample cannot see at all — it has no concept of “ready but not running”.
Evidence — what you infer. That the process is oversubscribed rather than slow. The tool shows ratios; the conclusion that the fix is fewer threads is inference. inference
Repair. Size the pool to activeProcessorCount; stop creating a pool per subsystem; let Swift concurrency, Dispatch or OperationQueue own the pool.
Tradeoff. Total throughput will not improve — it was already flat. What improves is per-thread latency and the tail: the fastest-to-slowest spread went from ~1.4× back to ~1.06×. Promise latency, not throughput.
C2Lock wait__psynch_mutexwait
Symptom. A thread is idle for a long, variable interval that correlates with another thread’s work.
Mechanism. pthread_mutex_lock on a lock another thread owns.
Evidence — what the tool shows. Two independent signatures. sample: leaf __psynch_mutexwait under _pthread_mutex_firstfit_lock_wait under _pthread_mutex_firstfit_lock_slow. Thread State Trace: a single long Blocked interval ended by made runnable by 0xda8048 (statelab, pid: 96444) running on CPU 9 (P Core). measured here
Evidence — what you infer. That the named thread is the owner. The trace attributes the wakeup, not ownership. They coincide for a mutex; for a condition variable or semaphore they need not. inference
Repair. Shrink the critical section, or split the lock. Signpost the wait and the held span separately: a long wait means someone else holds it too long; a long held span means your own section does too much.
Tradeoff. Splitting one lock into two creates a lock order you must now document and obey at every site.
C3Semaphore waitdifferent library, same silence
Symptom. Indistinguishable from C2 in a wall-clock view.
Mechanism. dispatch_semaphore_wait with no owner the runtime can name.
Evidence — what the tool shows. The sample leaf is _dispatch_semaphore_wait_slow in libdispatch, not in libsystem_pthread. The different library is the whole tell. measured here
Evidence — what you infer. That no priority donation is happening. The trace does not label “donation”; you infer it from the holder’s priority not changing (C4). inference
Repair. Replace the semaphore with a dependency the runtime can see: a lock for mutual exclusion, async let or a task group for a result, a continuation to bridge a callback.
Tradeoff. Semaphores remain the right tool for counting — bounded concurrency, backpressure — where there genuinely is no single owner. Do not let “semaphores are bad” become the rule.
C4Priority inversionread the holder’s band, not the waiter’s
Symptom. A user-interactive path stalls for hundreds of milliseconds while CPU sits idle on the performance cores.
Mechanism. A background-QoS thread holds the resource. Whether the system fixes it depends entirely on whether the primitive has a known owner.
Evidence — what the tool shows. The holder thread, same fixed work under the lock: with os_unfair_lock or pthread_mutex and a user-interactive waiter, the holder’s band moves 4 → 31 and its Running time moves 100% E cores → ~90% P cores, hold time 277 / 271 ms. With DispatchSemaphore(1) and the identical waiter, the holder stays at 4, stays 100% on E cores, hold time 1017 ms, and the waiter is Blocked at band 31 for 1028 ms. Controls with no waiter: all three stay at 4. measured here
Evidence — what you infer. That the boost is donation caused by the waiter. The priority change plus its absence in the no-waiter control is strong, but the trace never uses the word. inference
Repair. Prefer owner-bearing primitives — Apple: “favor symmetric primitives for mutually exclusive access.” Where a semaphore is unavoidable: “Ensure that the QoS of the waiting thread is the same as or lower than the QoS of the signaling thread.” documented
Tradeoff. Donation makes the lock fast, not the design right. A background job boosted to user-interactive for 277 ms is still 277 ms of background work on a performance core, charged to the user’s battery.
C5Main-thread hangand the absence that identifies it
Symptom. The UI stops responding; the app is not crashed.
Mechanism. One of three: busy main thread, asynchronous hang, or blocked main thread.
Evidence — what the tool shows. The Hangs instrument marks the interval. Then: CPU activity on the main thread ⇒ busy; little or none ⇒ blocked, and Thread State Trace’s Narrative names the blocking syscall and its exact backtrace. Without Instruments: sample <pid> 3 1 -file out.txt and read the main thread’s leaf frame. documented
The recognise-on-sight case. A Task { } plus semaphore.wait() inside @MainActor code produces two threads in the entire process: the main thread in semaphore_wait_trap, and a watchdog. The diagnostic gem is the absence — there are no cooperative-pool worker threads at all. The async work was never slow; it never started. measured here
Evidence — what you infer. Nothing about unresponsiveness from the thread state alone. A blocked main thread does not imply an unresponsive one — an idle app’s main thread is Blocked by definition. Only the Hangs instrument makes that call. documented
Repair. Busy: move the work off @MainActor — nonisolated async, or Task.detached if it must stay synchronous. Blocked: remove the wait; use the async API.
Tradeoff. Task { } from main-actor context inherits the main actor and only delays the hang; Task.detached inherits neither priority nor context and “executes only with .medium priority, by default.”
C6Excessive thread creationcount the threads, do not read the stacks
Symptom. Memory and context-switch counts climb; latency degrades under load; the whole machine feels sluggish, not just the app. Output stays correct, so no sanitizer helps.
Mechanism. Blocking work items on concurrent queues grow the pool; many private serial queues each request an overcommit thread.
Evidence — what the tool shows. This case needs a different reading skill: the thread list, not the stacks. sample listed 145 thread entries on a 14-core machine, and the “Sort by top of stack” summary showed 659 samples in __semwait_signal and 493 in __workq_kernreturn — scores of threads, almost none running. measured here Measured ceilings: 71 for any amount of blocking work on one queue, 201 for 200 untargeted serial queues, 513 for 600.
Evidence — what you infer. That the sysctl names map to those ceilings. The numbers match exactly, but no Apple text states the mapping. inference
Repair. Target your serial queues onto a shared root — measured 201 → 71 on a concurrent root, 201 → 2 on a serial root. Or move the blocking work to an async API.
Tradeoff. A serial root serialises everything targeted at it. Two threads instead of 201 is only the right answer if the work genuinely has no parallelism to exploit.
# Busy or blocked, one process, no privileges needed.
$ sample FrameworkLab 3 1 -file /tmp/fl-hang.txt
# Thread COUNT rather than thread stacks - the C6 reading.
$ grep -c 'Thread_' /tmp/fl-hang.txt ; sysctl -n hw.ncpu
# Runnable vs Preempted vs Blocked, and who woke whom. Headless, no GUI.
$ xcrun xctrace record --template "System Trace" --output st.trace --launch -- ./yourtool
$ xcrun xctrace export --input st.trace --xpath '/trace-toc/run[@number="1"]/data/table[@schema="thread-state"]' --output thread-state.xmlmeasured here One non-obvious fact that makes xctrace export usable at all: the exported XML interns its values. A value is written once with id="N" and thereafter referenced as ref="N", so any reader that does not resolve those references sees every column after the first row as empty.
C · Experiments
Nine things you can run, and what each one proves.
Every number on this page came from one of these. They are written to be pasted into an empty directory and compiled alone — no package manifest, no sibling file. Six of them deadlock, hang or starve on purpose.
SAFETY · READ BEFORE RUNNING. Several experiments below are deliberately broken, and several of those hang by design. Each one carries two independent bounds: an in-process watchdog thread that calls _exit(75) after a fixed budget — reporting through write(2) and exiting through _exit(2) rather than printf/exit, because a deadlocked process may hold a lock that stdio or an atexit handler would need — and an external hard timeout that SIGKILLs at roughly twice that budget. Exit code 75 means the watchdog fired, and for the deadlock modes that is the expected, correct result. A deadlock fixture that completed would be the failure. None of this belongs in production, and several broken forms compile without a single warning under Swift 6 strict concurrency — which is exactly why they are worth running.
Every figure below is one machine, one OS build, one workload. Orderings were stable across repeated runs; absolute numbers move by roughly ±30%, and two of them moved materially between the first run and the re-run — which is itself the lesson. Reproduce these; do not quote them.
E1Deterministic ABBA deadlock, with backtrace evidenceC · modes: deadlock, ordered, trylock
Build and run. clang -O0 -g -Wall -Wextra -pthread e01_abba_deadlock.c -o e01_abba, then ./e01_abba deadlock | ordered | trylock.
Observed.
E01 mode=deadlock (UNSAFE BY DESIGN; watchdog budget 5s)
thread A: holds lock_a, waiting at the gate
thread B: holds lock_b, waiting at the gate
thread B: gate open, now reaching for lock_a <-- blocks here forever
thread A: gate open, now reaching for lock_b <-- blocks here forever
WATCHDOG: no progress within budget - forcing _exit(75).
E01 mode=ordered result=COMPLETED 2000 paired acquisitions per thread, no stall
E01 mode=trylock result=COMPLETED 2000 paired acquisitions, 1-3 backoffsTimeout behaviour. Watchdog budget 5 s; the external driver SIGKILLs at ~10 s and asserts exit 75.
What it proves. That the deadlock is a property of the acquisition order, not of timing: a rendezvous gate makes it fire 100% of the time. And that a sample report identifies it from three things at once — 92 of 92 samples in one frame (so the thread never moved), __psynch_mutexwait at the top of stack, and two threads mutually parked.
The repair. A total lock order removes circular wait. Backoff removes hold-and-wait but can livelock.
Limitation. The backoff count (1–3) is small only because contention here is low; its worst case is unbounded.
E2A data race, a race condition, and the build flag that hides oneSwift · 6 modes
Build and run. swiftc -swift-version 6 -Onone e02_data_race.swift -o e02_race for the plain build, and swiftc -swift-version 6 -Onone -g -sanitize=thread e02_data_race.swift -o e02_tsan for the sanitized one. Thread Sanitizer works under the command-line toolchain — no Xcode project needed.
Observed (re-run 2026-09-22):
E02 mode=race expected=1600000 observed=900219 lost=699781 correct=false
E02 mode=race expected=1600000 observed=629777 lost=970223 correct=false (second run)
E02 mode=fixed expected=1600000 observed=1600000 lost=0 correct=true
WARNING: ThreadSanitizer: Swift access race (pid=89798)
WARNING: ThreadSanitizer: data race (pid=89798)
ThreadSanitizer: reported 3 warnings
E02 mode=checkthenact gapMs=2 seats=5 buyers=40 sold=40 oversold=35 dataRace=false
-> Thread Sanitizer output: 0 bytesWhat it proves. Three separate things. (1) Swift programs produce TSan’s Swift-specific “Swift access race” report — an exclusivity violation — alongside the ordinary “data race” report; engineers who have only seen the C output do not recognise the first one. (2) At -O the optimiser collapses the loop and the same program prints the right answer 3 times out of 3 — “I ran it and got the right answer” is not evidence. (3) A correctly-locked check-then-act sells 35 seats it does not have with the sanitizer completely silent.
The repair. For the data race, a Mutex. For the race condition, not a different lock — one critical section spanning check and act.
Limitation. The burst mode is deliberately flaky (it failed 7 of 8 runs originally; in the re-run one burst was clean and one oversold by 2). The driver asserts its exit code, never its outcome, because asserting a specific oversell count would be asserting a race.
E3Bounded parallelism, with the bound measuredSwift · semaphore vs task group vs timeout
E03 mode=unbounded tasks=64 limit=none peakConcurrency=64 completed=64
E03 mode=semaphore tasks=64 limit=4 peakConcurrency=4 completed=64 boundHeld=true
E03 mode=taskgroup tasks=64 limit=4 peakConcurrency=4 completed=64 boundHeld=true
E03 mode=timeout tasks=16 limit=2 budgetMs=150 acquired=2 timedOut=14
shedLoadInsteadOfHanging=trueWhat it proves. The bound holds exactly, and the async TaskGroup sliding window reaches the identical bound without blocking a thread. The timeout mode exercises the branch most semaphore code omits: a finite deadline that sheds load instead of queueing forever. Using .distantFuture here is how a stall becomes a hang.
A design detail worth stealing. Peak concurrency is measured by a meter that increments, records the high-water mark and decrements under one lock — deliberately, because tracking inFlight and peak in two separate atomics would make the measurement itself a check-then-act race.
E4Why the predicate loop is mandatorySwift · while vs if, and the lost wakeup
E04 mode=correct expected=120 consumed=120 underflows=0 spuriousAbsorbed=618 correct=true
E04 mode=broken expected=120 consumed=120 underflows=208 correct=false
note=each underflow would be a removeFirst() crash in real code
E04 mode=lostwakeup brokenConsumerWokeUp=false correctConsumerSawState=true
lesson=the predicate is the state; the signal is only a hint
E04 mode=timeout returned=false elapsedMs=405 boundedByDeadline=trueWhat it proves. The only difference between the two consumers is while versus if. Spurious wakeups are injected — a broadcast that changes no state — so the bug is deterministic rather than a once-a-month production mystery. ~618 wakeups absorbed by the loop, versus ~208 underflows without it. The lostwakeup mode isolates the other half of the contract: a producer signals with nobody waiting and the signal evaporates.
An honesty note about the fixture. Each underflow is a removeFirst() on an empty array — a trap in real code. The fixture returns nil instead, purely so it can count failures rather than crash the run, and says so in its own source.
E5Actor reentrancy: compiles clean, TSan clean, still wrongSwift · broken + three repairs
E05 mode=broken callers=6 succeeded=6 finalBalance=-500 dataRace=false invariantHeld=false
E05 mode=recheck callers=6 succeeded=1 finalBalance=0 dataRace=false invariantHeld=true
E05 mode=reserve callers=6 succeeded=1 finalBalance=0 refunds=0 invariantHeld=true
E05 mode=serialize callers=6 succeeded=1 finalBalance=0 maxInFlight=1 invariantHeld=true
$ ./e05_tsan broken 2>&1 | grep -c ThreadSanitizer
0What it proves. Thread Sanitizer reports absolutely nothing while the balance reaches −500, and the driver asserts that silence as a passing condition. The actor is doing its job perfectly: one job at a time touches balance, no torn reads, no lost updates, no data race. The invariant dies at the suspension, where the actor is released and five other jobs run to completion before the first one resumes.
The repairs, in preference order. Re-check is cheapest but the suspended work still ran. Reserve is stronger but needs a correct compensation path. Serialize works but reintroduces queueing, ordering and starvation questions — last resort.
E6Priority donation, measured directlyC · the best single artifact here
E06 primitive=os_unfair_lock uncontended=4 contended=31 donatedTrials=3/3
E06 primitive=pthread_mutex uncontended=4 contended=31 donatedTrials=3/3
E06 primitive=dispatch_semaphore uncontended=4 contended=4 donatedTrials=0/3How it is measured. A QOS_CLASS_BACKGROUND thread holds a resource; a QOS_CLASS_USER_INTERACTIVE thread blocks behind it. The holder reads its own current scheduling priority through thread_info(mach_thread_self(), THREAD_EXTENDED_INFO, …).pth_curpri — the same number sample and spindump print as “priority N”.
What it proves. It turns “don’t use a semaphore as a mutex” from a rule into a number. The holder’s priority jumps 4 → 31 the instant a user-interactive thread blocks on a lock, and drops back on release. Behind a semaphore it never moves. Perfectly reproducible, and reproduced again on re-run.
A false lead worth not repeating. pthread_get_qos_class_np() reports the QoS a thread requested, not its effective priority — under full donation it still returned QOS_CLASS_BACKGROUND while pth_curpri read 31. negative result
E7The Task { } + semaphore deadlockSwift · and the control that names the real culprit
E07 mode=deadlock -> WATCHDOG _exit(75) (permanent hang)
E07 mode=detached -> result=42 completed=true
E07 mode=asyncmain -> result=42 completed=true
# the trace signature - the WHOLE process:
93 Thread_14606672 DispatchQueue_1: com.apple.main-thread (serial)
+ 93 main
+ 93 _dispatch_semaphore_wait_slow
+ 93 semaphore_wait_trap (in libsystem_kernel.dylib)
93 Thread_14606675 [the watchdog, sleeping]What it proves. Two threads in the entire process, and no cooperative-pool workers at all. The async work was never slow — it never started. Task { } inherits the enclosing isolation, so inside @MainActor code the child is main-actor-isolated and sem.wait() blocks the one thread main-actor work can run on.
The detached mode is the control that identifies the culprit. Same semaphore, same blocking wait, but Task.detached inherits no isolation, runs on the cooperative pool, and completes. The semaphore was never the bug; isolation inheritance was. It is still poor code — it burns a thread — and belongs only at a genuine process boundary. The real fix is asyncmain: await instead of blocking.
Provenance worth stating. This experiment exists because the bug was written by accident while building E3. That is how common it is.
E8Thread explosion, and the “fix” that fixes nothingSwift · the counterintuitive middle row
E08 mode=explode peakThreads=73 threadsPerCore=5.2
E08 mode=bounded peakThreads=73 limit=4
note=the permit bounds CONCURRENCY, not thread creation
E08 mode=bounded-submit peakThreads=8 limit=4
note=the wait moved OUT of the work item, so unstarted work holds no thread
E08 mode=structured peakThreads=13 threadsPerCore=0.9What it proves — and this is the teaching point. Wrapping the work-item body in a semaphore is the reflex fix, and it does nothing at all for thread explosion: 73 threads either way. A blocked permit-wait occupies a Dispatch worker thread exactly like blocked work does; libdispatch cannot tell a blocked item from a slow one, so it brings up another thread to keep the queue’s width occupied. The permit bounds how many items do work; it does not bound how many threads exist.
The repair. Move the same wait to the submission site — peak drops from 73 to 8, because unstarted work never occupies a thread. Structured concurrency reaches the same place by suspending rather than blocking.
E9Contention scaling: adding threads made it slowerC · median of 5 runs per cell
threads 1 2 4 8 16 (milliseconds)
pthread_mutex 19 22 64 40 35
os_unfair_lock 7 50 108 29 26
atomic_relaxed 6 15 30 79 142
sharded_x16 16 8 4 12 11
E09 strategy=pthread_mutex oneThreadMs=19 sixteenThreadMs=35 slowdown=2.2x
E09 strategy=os_unfair_lock oneThreadMs=7 sixteenThreadMs=33 slowdown=5.0x
E09 strategy=atomic_relaxed oneThreadMs=6 sixteenThreadMs=156 slowdown=24.0x
E09 strategy=sharded_x16 oneThreadMs=16 sixteenThreadMs=11 slowdown=0.7xThree results that contradict common intuition. (1) The atomic is the worst performer under contention — 24× slower at 16 threads. Sixteen cores fighting for exclusive ownership of one cache line ping-pong it endlessly, and unlike a lock there is no parking: every loser retries immediately and burns a core doing it. Lock-free removes the lock, not the contention. (2) os_unfair_lock degrades worse than pthread_mutex in this shape, even though it is much cheaper uncontended — which is exactly why it is the default recommendation, and exactly why fairness is what bounds the tail. (3) Only sharding scales, and it is the only strategy faster at 16 threads than at 1.
A detail that decides the result. The shards carry 128-byte padding. Without it they share a cache line and you measure false sharing instead — a classic way to “prove” that sharding does not help.
Limitation, and why this table is printed with its re-run. The first run gave slowdowns of 2.8×, 5.7×, 22.9× and 0.7×; the re-run above gave 2.2×, 5.0×, 24.0× and 0.7×. The ordering is stable; the absolute figures are not. That is the honest shape of a wall-clock measurement on a busy laptop, and it is why the instruction is to reproduce rather than quote.
The honest reading. The repair for contention is fewer acquisitions or a smaller shared surface — never more threads, and not reflexively an atomic.
D · Fixing exercises
Six broken programs. Diagnose, repair, prove.
The experiments above are things to run. These are things to fix. Each one ships a deliberately broken program with a deterministic symptom, an interview-style prompt, the evidence you are expected to collect, a bounded success criterion, progressive hints, and a separate solution — plus a fixed source file, a patch from one to the other, and a check that proves both halves. The broken starting points are immutable. Copy one, repair your copy, and leave the original for the next time you want the drill cold.
SAFETY · READ BEFORE RUNNING. Several of these programs deadlock, stall or oversubscribe the machine on purpose. They cannot wedge it. Every one carries an in-process watchdog thread that calls _exit(75) after a fixed budget — reporting through write(2) and exiting through _exit(2) rather than printf/exit, because a deadlocked process may hold a lock that stdio or an atexit handler needs — and every check.sh adds an external hard timeout that SIGKILLs at several times that budget. Exit status 75 means the watchdog fired, and for exercise 1 that is the expected, correct result: a run that completed would be the failure. None of the broken code belongs in production, and most of it compiles without a single warning under Swift 6 strict concurrency — which is exactly why it is worth fixing by hand.
Get the bundle, then run one command
Download threads-locks-exercises.tar.gz — 34 files, 56,651 bytes. SHA-256 f8e1692ee5f1b791c8371f6898acc12ea7b60c0d2837f4b918d55783761f61d1
The archive is built deterministically from the same sources linked below — sorted member list, pinned timestamps, zeroed ownership, gzip -n — so rebuilding it reproduces that hash byte for byte. It carries no build products and no absolute paths, and the packer refuses to write an archive in which one appears.
tar xzf threads-locks-exercises.tar.gz
cd threads-locks-exercises
./run-all.shFor each exercise it compiles the broken and the fixed source each alone in a fresh temporary directory and fails on any unexpected warning; applies solution.patch to a copy of the broken file and requires the result to equal the fixed file byte for byte; runs the broken build and asserts its documented symptom; runs the fixed build and asserts its documented success; and repeats every timing-sensitive measurement. measured here From a clean extract on the reference machine: 120 assertions, 0 failures, 85 seconds. ./run-all.sh --quick skips the Thread Sanitizer builds; ./run-all.sh 03 05 runs only the named exercises.
Where a measurement is deterministic the check asserts it exactly. Where it is timing-sensitive it asserts a ratio with a generous bound, so the exercise still validates on hardware that behaves quite differently. Every absolute figure below is one machine under one load — reproduce them, do not quote them.
01 · The transfer that freezes
A money-transfer feature locks the source account, then the destination account, then moves the balance. It passed review and it passes its unit tests. In production it freezes a few times a week, always under load. Reproduce it deterministically, prove from the process what it is waiting on, and fix it. Then tell me what your fix costs.
clang -O0 -g -Wall -Wextra -pthread broken/transfer.c -o /tmp/transfer_broken
/tmp/transfer_broken ; echo "exit status: $?"Expected signal. Two threads each announce that they hold one account and are reaching for the other, then nothing until the watchdog fires. A sample of the live process shows two or more threads parked in __psynch_mutexwait, one frame holding every sample, and the wait mutual. measured here The rendezvous gate in the fixture is test scaffolding, not the bug: it removes the timing luck that normally hides an ordering defect.
Success criterion. Your repaired build exits 0 (never 75), prints conserved=true and transfers=100002, and does so twice in a row — a deadlock fix that works once has not been tested. You can name which of the four Coffman conditions you removed.
Progressive hints
- Where to look. Print the account id each thread locks first and the one it locks second. The two threads disagree, and that disagreement is the entire bug. Nothing about the amount, the direction, or the width of the critical section matters.
- The four conditions. A deadlock needs mutual exclusion, hold-and-wait, no preemption, and circular wait. Three of those are properties of using locks at all. Only one is a property of your code, and it is the one you can delete.
- The shape of the repair. The direction money moves and the order locks are acquired do not have to be the same thing. Sort the two accounts by something stable every call site can compute independently, and always take the lower one first. There is a second, weaker repair — keep the inconsistent order but never block while holding — and you should know why it is weaker before reaching for it.
Solution
Impose a total order over the locks and obey it at every acquisition site:
static void transfer(account_t *from, account_t *to, long amount) {
account_t *first = from->id < to->id ? from : to;
account_t *second = from->id < to->id ? to : from;
pthread_mutex_lock(&first->m);
pthread_mutex_lock(&second->m);
from->balance -= amount;
to->balance += amount;
pthread_mutex_unlock(&second->m);
pthread_mutex_unlock(&first->m);
}Why it works. A deadlock requires a cycle in the wait-for graph. If every thread acquires in increasing id order, a thread can only ever wait on a lock with a higher id than every lock it holds, so following waits strictly increases the id — and a strictly increasing sequence cannot return to where it started. Circular wait is not merely unlikely; it is impossible.
What it costs. Two comparisons at run time. The real cost is a discipline you now have to maintain: every future call site must agree, including one written by somebody who has never read this file. Write the order down beside the lock declarations and treat “acquires two locks” as a review trigger.
The weaker repair. Take the first lock, trylock the second, and on failure drop everything and retry after a backoff. That removes hold-and-wait rather than circular wait. It is sometimes the only option when one of the locks lives inside a framework you do not control, but its worst case is unbounded — two threads can livelock — and it needs a backoff, which needs jitter, which is now a tuning parameter. Prefer the total order when you control the code. inference
02 · The totals that are always a bit low
A download manager tallies bytes and chunks from several transfer threads. QA reports the totals are “a bit low, but only on fast connections”. The engineer who wrote it says they have run it twenty times and it looks fine. Prove the defect exists with something better than a total that looks wrong. Fix it. Then explain why their twenty runs proved nothing, and why a build flag can change the answer.
swiftc -swift-version 6 -Onone broken/stats.swift -o /tmp/stats_broken
/tmp/stats_broken ; /tmp/stats_broken # two runs disagree
swiftc -swift-version 6 -Onone -g -sanitize=thread broken/stats.swift -o /tmp/stats_tsan
/tmp/stats_tsan ; echo "exit status: $?"Expected signal. Keep two kinds of evidence apart. The lost-update output is nondeterministic — 493,500 then 575,481 of an expected 1,600,000 on the reference machine, a different figure every run. The detector evidence is not: WARNING: ThreadSanitizer: Swift access race and data race, on every run, with both conflicting stacks. Exit status 134 is SIGABRT — once the sanitizer has reported a warning it aborts at exit, which is why a sanitized CI job fails loudly even when the program’s own output looks plausible. measured here
Then build it a third way, at -O, and be ready to explain what you see: the optimiser collapses each thread’s 200,000 increments into one addition and the program prints the same wrong number every single time. measured here That is the answer to “I ran it twenty times”: a stable number is not a correct number, and an unsynchronised program’s behaviour is a property of the build rather than of the source. negative result
Success criterion. Two consecutive runs print correct=true and lostChunks=0; built with -sanitize=thread the fixed build produces no WARNING: ThreadSanitizer line and exits 0; and you did not reach for @unchecked Sendable.
Progressive hints
- What the hardware does.
bytesReceived += bytesis a load, an add and a store. Two threads can both load 41, both add, and both store 42. One increment is simply gone, and nothing in the source marks where that can happen. - What must become indivisible. Not the assignment — the whole read-modify-write, and in this type both counters together, because a reader should never see a byte total that does not match its chunk total.
- Let the compiler help. If the repair is right,
@unchecked Sendablebecomes unnecessary and the type can be plainSendable. Deleting@uncheckedand still getting a clean build is itself a check on your work: you have moved from asserting safety to having it verified.
Solution
final class DownloadStats: Sendable {
private struct Totals { var bytes = 0; var chunks = 0 }
private let totals = Mutex(Totals())
func record(bytes: Int) {
totals.withLock { t in
t.bytes += bytes
t.chunks += 1
}
}
func snapshot() -> (bytes: Int, chunks: Int) {
totals.withLock { ($0.bytes, $0.chunks) }
}
}Three things changed. The counters moved inside the lock, which owns them — withLock is the only door, so this is not a convention a future reader can quietly break. Both counters are in one lock, because the invariant spans both fields; two independent locks would make each counter individually correct while still letting a snapshot mix two moments. And @unchecked Sendable is gone, so the compiler checks this type instead of taking our word for it. What did not change: the workload, the thread count, the iteration count.
The distinction to state out loud. A data race is about memory — two threads touch one location with no ordering and at least one writes — and Thread Sanitizer finds it. A race condition is about invariants, and every individual access can be perfectly locked while the program stays wrong. Fixing the first does not fix the second, and the sanitizer is silent about the second by design. documented
What the detector cannot do. It is a runtime tool: it reports races on paths that actually executed, with the interleavings that actually happened. Apple documents roughly 5–10× memory and 2–20× slowdown, so it is a CI and debugging tool rather than something you ship. Silence is not proof of correctness. documented
03 · The “lock” that makes the UI wait for a background import
A thumbnail cache is protected by a DispatchSemaphore(value: 1). It excludes correctly — exactly one thread is ever inside. Nothing crashes, nothing hangs, the tests pass. A performance engineer says user-interactive work waiting on this cache “runs at the speed of whatever background thread happens to hold it”. Prove or disprove that from the system, not from first principles.
clang -O0 -g -Wall -Wextra broken/gate.c -o /tmp/gate_broken
clang -O0 -g -Wall -Wextra fixed/gate.c -o /tmp/gate_fixed
/tmp/gate_broken 3 ; /tmp/gate_fixed 3Expected signal. The holder’s current scheduling priority, read from thread_info(THREAD_EXTENDED_INFO).pth_curpri — the same number sample and spindump print as priority N — measured once uncontended and once while a USER_INTERACTIVE thread is blocked behind it. measured here
dispatch_semaphore(1) holderPriUncontended=4 holderPriWhileUIWaits=4 donated=NO
os_unfair_lock holderPriUncontended=4 holderPriWhileUIWaits=31 donated=YES
EX03 build=broken primitive=dispatch_semaphore(1) trials=3 donatedTrials=0
EX03 build=fixed primitive=os_unfair_lock trials=3 donatedTrials=3Use pth_curpri rather than pthread_get_qos_class_np(), which reports only the QoS a thread requested. A donated thread still requests BACKGROUND; what changes is what it actually runs at.
Success criterion. Your repaired build shows donation in a majority of trials across two runs, and both builds still exit 0 — if your repair introduced a hang, it is not a repair. You can say which property of the primitive makes donation possible, in one sentence, without using the word “better”.
Progressive hints
- Ask what the kernel knows. To speed up the thread that is holding things up, the kernel has to know which thread that is. Look at what each primitive stores. One keeps a thread identifier; the other keeps an integer.
- Why a semaphore cannot know.
dispatch_semaphore_signalmay legitimately be called by a thread that never calledwait. That is not a misuse — it is the point of a counting and signalling primitive — but it means “the thread that holds this semaphore” is not a well-defined idea. - The substitution. Replace the gate with a primitive that records an owner:
os_unfair_lock,pthread_mutex, or in SwiftMutex/OSAllocatedUnfairLock/NSLock. Change nothing else — same QoS bands, same hold duration — and re-readpth_curpri.
Solution
Swap the gate for os_unfair_lock and change nothing else. The measured priority of the background holder goes from 4 → 4 to 4 → 31 while a USER_INTERACTIVE thread waits. measured here
os_unfair_lock stores the owning thread’s port in the lock word, so when a higher-priority thread blocks the kernel can see exactly which thread is in the way and raise it until the lock is released. A semaphore is a counter; signal may come from any thread, so there is no owner to raise and the high-priority waiter is stuck behind background-rate work. pthread_mutex, Swift’s Mutex, OSAllocatedUnfairLock, NSLock and NSRecursiveLock all carry ownership and all donate; DispatchSemaphore and DispatchGroup do not.
The rule, stated properly. Use a semaphore to count permits or to signal between threads; use a lock for mutual exclusion. Not because locks are faster — they may not be — but because only a lock can tell the kernel who is holding things up.
What this does NOT claim. It does not reproduce the textbook unbounded priority-inversion stall, which does not occur on modern Darwin with a donating lock. negative result What is reproducible is the donation signal: one primitive causes a measurable kernel override, the other does not. Donation is kernel policy rather than a documented API guarantee, which is why the bundled check requires a majority of trials rather than all of them. inference
The tool worth naming. Apple’s Thread Performance Checker reports priority inversions during an Xcode Run action. It has no supported command-line invocation, which is why this exercise measures the kernel behaviour directly instead. documented
04 · The launch that stalls for exactly 1.5 seconds
Two reports from the same codebase. One: “the app occasionally sits on the launch screen for about a second and a half, then carries on normally, and we cannot reproduce it.” Two: “a background reader crashes taking an item from an empty queue, roughly once a week.” Both come from the same misunderstanding of one primitive. Find it, explain why the two symptoms are the same bug, and fix both with one change.
swiftc -swift-version 6 -O broken/handoff.swift -o /tmp/handoff_broken
swiftc -swift-version 6 -O fixed/handoff.swift -o /tmp/handoff_fixed
/tmp/handoff_broken ; /tmp/handoff_fixed partA waitedMs=1503 dataWasAlreadyPublished=true
partB consumed=120 expected=120 wouldHaveCrashed=16 spuriousAbsorbed=0
partC semaphorePermitSurvivedTheGap=true waitedMs=0
partA waitedMs=0 dataWasAlreadyPublished=true
partB consumed=120 expected=120 wouldHaveCrashed=0 spuriousAbsorbed=18
partC semaphorePermitSurvivedTheGap=true waitedMs=0Expected signal. Part A is fully deterministic: the loader publishes and signals before any consumer exists, so the broken consumer waits its entire 1.5-second deadline for data that was already in memory, and the fixed one waits 0 ms. Part C is the control — identical in both builds — and it runs the same “signal before anyone waits” sequence through a DispatchSemaphore, where the permit is still there. measured here So the lesson is not “signals get lost”: a permit is durable, an announcement is not, and neither of them is your predicate.
Success criterion. Two consecutive runs report sleptThroughPublishedData=false, partAWaitedMs under 250, wouldHaveCrashed=0 with spuriousAbsorbed above zero, and consumed=120 expected=120. One change fixed both symptoms — if you wrote two unrelated fixes you have not found the bug yet.
Progressive hints
- Read the two waits side by side. One never asks whether the thing it is waiting for has already happened. The other asks once, then believes the answer forever. Those are the same mistake at two different moments: before the wait and after it.
- What a wakeup means. “The predicate may have changed” — never “the item is yours”. Another consumer may have been woken first and taken it, and the kernel is permitted to wake you for no reason at all. The first happens far more often than the second, and neither is rare enough to ignore.
- The shape of the repair.
while !predicate { cond.wait(until: deadline) }. Notice that one line does two jobs — the check before the first wait and the re-check after every wakeup — which is why it fixes both symptoms at once. Then ask a second question: when one state change can satisfy more than one waiter, issignal()orbroadcast()correct?
Solution
// before the first wait — a lost announcement becomes harmless
while configuration == nil {
if !cond.wait(until: deadline) { break }
}
// after every wakeup — a wakeup is a hint, not a promise
while items.isEmpty && !closed {
if !cond.wait(until: deadline) { break }
}
guard !items.isEmpty else { return nil }
return items.removeFirst()Why one change fixes both. The shared predicate is the state; the condition variable is only a notification that the state may have changed. Before the first wait, the loop’s condition is a check: if the thing already happened the body never runs. After each wakeup it is a re-check: you only leave when the predicate is actually true. The two symptoms were the same bug seen from two sides, which is why while is not a style preference and if is a defect.
signal versus broadcast. Use broadcast when one state change can satisfy more than one waiter, or when waiters are waiting on different predicates over the same lock — with signal you may wake the one waiter who still cannot proceed while the one who could stays asleep. Termination must always broadcast, or a consumer sleeps through the shutdown. With a correct predicate loop an unnecessary wakeup costs a re-check; without one it costs a bug. That asymmetry is why broadcast is the safer default and signal is the optimisation. inference
The mirror-image mistake. Using a semaphore where you needed a predicate: the count drifts out of step with the state it was standing in for, and now you have two sources of truth. Use a condition variable for “wait until X”, a semaphore to count a bounded resource, and a group or task group to wait for work to finish.
05 · The aggregator that got slower when we added workers
A metrics aggregator digests events. Each event is scored and folded into a running total behind one mutex. The team’s response to “the aggregator is too slow” was to raise the worker count from 4 to 16. It got slower. Measure it properly, explain the shape of the curve, and fix it. Your fix must produce the identical total — I will check.
clang -O2 -g -Wall -Wextra broken/aggregate.c -o /tmp/aggregate_broken
clang -O2 -g -Wall -Wextra fixed/aggregate.c -o /tmp/aggregate_fixed
/tmp/aggregate_broken ; /tmp/aggregate_fixedEX05 build=broken threads 1 2 4 8 16 (ms)
wall ms 77 113 294 191 179
vs 1 1.00x 1.47x 3.80x 2.47x 2.32x
checksum=127435700 acquisitions=1000000 worstSlowdown=3.80 worstAtThreads=4
EX05 build=fixed threads 1 2 4 8 16 (ms)
wall ms 75 42 21 11 11
vs 1 1.00x 0.55x 0.28x 0.14x 0.14x
checksum=127435700 acquisitions=15626 slowdown=0.14Expected signal. A fixed total amount of work split across 1, 2, 4, 8 and 16 threads; perfect scaling would keep wall time flat. Three numbers carry the answer: one million acquisitions; a curve whose worst point is at 4 threads, not 16, then partially recovers — the signature of a convoy plus adaptive lock behaviour; and a checksum that must be identical in both builds. measured here In Instruments, System Trace’s Thread State view shows the same story as a picture: threads mostly blocked, in short runnable bursts, handing the lock to each other. documented
Success criterion. Your repaired build’s 16-thread time is lower than its 1-thread time (the check allows a generous ratio ≤ 0.60 so a 4-core machine still passes), its checksum equals the broken build’s exactly, acquisitions falls by at least 32×, and both hold on two consecutive runs.
Progressive hints
- Look at what is inside the lock. For each line in the critical section ask: does this line touch shared state? The scoring function reads nothing shared and writes nothing shared. It is inside the lock only because somebody put it there.
- Count the acquisitions, not just the hold time. Even with a one-instruction critical section, taking a contended lock a million times costs a million uncontended-to-contended transitions. Addition is associative: you do not have to publish every partial result the moment you compute it.
- Divide the shared surface, and mind the cache line. One hot total can become N independent totals summed at the end — but pad each to its own cache line, or the shards share a line, every update invalidates every other shard, and you will measure false sharing and conclude sharding does not help. Then ask what property of this workload makes sharding legitimate at all.
Solution
Three changes, one goal — hold the lock for less time, and take it far less often:
unsigned long local = 0;
for (unsigned long i = 0; i < w->count; i++) {
unsigned long s = score_event(w->first + i); /* OUTSIDE the lock */
local += s & 0xFFUL;
if ((i % BATCH) == BATCH - 1) fold(w->id, &local);
}
fold(w->id, &local);
typedef struct {
pthread_mutex_t m;
unsigned long total, acquisitions;
char pad[128 - sizeof(pthread_mutex_t) - 2 * sizeof(unsigned long)];
} shard_t;Measured effect: 16-thread wall time 179 ms → 11 ms, acquisitions 1,000,000 → 15,626, checksum unchanged. measured here
Which change did the most? The right answer is “I would measure them separately”, and you can. Moving the scoring out narrows the critical section but leaves a million acquisitions. Batching alone is the biggest single lever here, because the cost being paid a million times is the transition, not the hold. Sharding divides contention rather than removing it. An interviewer asking “which one mattered?” is usually checking whether you would guess or measure. inference
Why sharding is legitimate here. Each worker owns a shard and the only invariant is a sum, which is associative and commutative. Sharding buys you nothing when the invariant spans shards: if the rule were “the total must never exceed a cap”, you would have to hold every shard’s lock to check it. That is the difference between knowing a technique and knowing when it applies.
What the repair deliberately did not do. It did not swap pthread_mutex for a cheaper lock and it did not go lock-free. Reaching for a cheaper primitive is the reflex answer and usually the wrong first move, because it addresses the cost per acquisition rather than the number of acquisitions or the width of the critical section. Fix the shape first; change the primitive only if the measurement still says to.
06 · The semaphore that limits concurrency and nothing else
An importer reads 128 records through a synchronous, blocking API. Someone saw the thread count climbing in Activity Monitor and added a DispatchSemaphore(value: 4) “to limit concurrency to 4”. The thread count did not change. The semaphore is right there, the limit is correct, and concurrency really is 4. Explain what the semaphore is actually doing, and fix the thread count without changing the limit.
swiftc -swift-version 6 -O broken/ingest.swift -o /tmp/ingest_broken
swiftc -swift-version 6 -O fixed/ingest.swift -o /tmp/ingest_fixed
/tmp/ingest_broken ; /tmp/ingest_fixedEX06 build=broken items=128 limit=4 cores=14 peakThreads=73 threadsPerCore=5.2 permitHolders=4 elapsedMs=5890
EX06 build=fixed items=128 limit=4 cores=14 peakThreads=8 threadsPerCore=0.6 permitHolders=4 elapsedMs=5975
EX06 build=fixed variant=structured cores=14 structuredPeakThreads=8 structuredElapsedMs=4027Expected signal. Watch two numbers together, because either alone tells the wrong story: permitHolders is 4 in both builds — the semaphore was doing its job the whole time — while peakThreads goes from 73 to 8. measured here A Dispatch global queue is overcommitting: when a work item blocks, libdispatch cannot distinguish “blocked” from “slow”, so it brings up another thread to keep the queue’s width occupied. documented Blocking work items convert directly into threads.
Success criterion. Two consecutive runs keep peakThreads at or below 24, still report permitHolders=4, and cut the peak by at least 3× against the broken build. You can explain why the wall-clock time barely moved, and why that is the right outcome.
Progressive hints
- Count what is already in flight. Before the first permit is taken, how many of the 128 closures have been handed to the queue? How many has libdispatch started? Each started closure is on a thread. The permit controls what happens after that point.
- Move the decision, not the primitive. The semaphore is the right tool and 4 is the right number. The question is which thread should block: the one doing the work, or the one deciding to create the work? Work that has not been admitted should not exist yet.
- Mind which side returns the permit. If you move
wait()to the submission site, be careful wheresignal()goes. Returning the permit at submission time bounds nothing — it has to be returned when the work finishes, so it stays inside the closure even though its partner moved out.
Solution
for _ in 0..<items {
gate.wait() // throttles the SUBMITTING thread
DispatchQueue.global(qos: .utility).async(group: group) {
blockingRead() // the synchronous API
gate.signal() // a permit frees only on COMPLETION
}
}Two lines moved. wait() now blocks the submitting thread — one thread, the one you already had — and unadmitted work is not on the queue, has not started, and holds nothing. Note the asymmetry: wait() moved out of the closure, signal() stayed in, because the permit must be released when the work completes rather than when it is submitted.
Why the wall time barely changed, and why that is right. Both builds took about six seconds, because both were limited to 4 concurrent items. This repair is about resource cost, not throughput: the broken version was paying for 73 threads’ worth of stacks, scheduler pressure and context switches to achieve exactly the same rate of progress. It was fully serialised and fully oversubscribed. What you gained is memory, scheduler contention, and not being the process that starves everything else on the machine.
The structured version, and its condition. The fixed file also measures a bounded task group — await group.next() before each addTask — which reaches the same bound with the limit visible in the control flow, at 8 peak threads and faster, because a suspension costs far less than a blocked thread. But it is only available if the blocking call can become async. Putting synchronous I/O inside that task group would block a cooperative-pool thread, and that pool is sized to the core count, so a handful of such tasks can stall every other task in the process — strictly worse than the Dispatch version, which would at least have spawned more threads. Swift 6 enforces part of this directly: DispatchSemaphore.wait() is unavailable from asynchronous contexts and will not compile there, which is why both files keep the Dispatch path in a synchronous function. documented
What these exercises deliberately do not cover
Instruments GUI workflows. Every step is command line; where an Instruments view would show the same thing the exercise names it, but there is no trace to open. Thread Performance Checker, which is an Xcode Run-action tool with no supported CLI. The textbook unbounded priority-inversion stall, which does not reproduce on modern Darwin with a donating lock — exercise 3 says so rather than manufacturing it. negative result And Thread Sanitizer’s limits: it is a runtime detector that costs roughly 5–10× memory and 2–20× time, finds races only on paths that actually executed, and cannot see race conditions at all — exercise 2 demonstrates a correctly-locked program that is still wrong with the sanitizer completely silent. documented
E · Incident drill
Debug this incident: PhotoExport.
Four builds of one feature, each with exactly one defect, under opaque names so the skill being practised is choosing a tool from a symptom rather than recognising a label. The stages are ordered so that each one defeats the tool that solved the previous one.
How to run it honestly
For each stage: read the symptom, write down your hypothesis and the one tool you would reach for first, then open the evidence. Only after you have evidence should you open the diagnosis. Opening them in order is the whole exercise; opening them out of order makes it a reading comprehension test.
Everything is bounded. Stage 1 is expected to be terminated by its own watchdog after 6 seconds — that is the fixture protecting you, not a crash.
$ swiftc -swift-version 6 -O incident_photoexport.swift -o incident
$ swiftc -swift-version 6 -Onone -g -sanitize=thread incident_photoexport.swift -o incident_tsanStage 1 · “The app freezes when I tap Export”
A user reports a permanent freeze. The spinner never moves. Force-quit is the only way out. Before running anything else, answer: is the app busy or blocked? These have opposite repairs, and almost every wasted hour in a hang investigation comes from assuming one without checking.
Step 1 · the discriminating questionWhich single tool answers busy-versus-blocked, and why that one?
A busy app burns CPU; a blocked app burns none. sample answers this in one command because it records the stacks of every thread, and a blocked thread shows the same frame in every single sample.
Thread Sanitizer would waste your time here: nothing is racing.
$ ./incident run-a & PID=$!; sleep 1
$ sample $PID 2 10 -file /tmp/stage1.txtNow look at two things, in this order: (1) what is the main thread doing — read the bottom frame; (2) how many other threads are there, and what are they doing? The second question is the one people forget, and here it is the one that solves the incident.
Step 2 · the evidenceThe whole report is four lines long. That is the clue.
93 Thread_14606672 DispatchQueue_1: com.apple.main-thread (serial)
+ 93 main
+ 93 _dispatch_semaphore_wait_slow
+ 93 semaphore_wait_trap (in libsystem_kernel.dylib)
93 Thread_14606675 [a watchdog, sleeping]Main thread: 93 of 93 samples in semaphore_wait_trap. Zero CPU. Blocked, not busy.
Before opening the diagnosis: what is missing from that report? Count the threads and compare against what you would expect from a process that is running async work.
Step 3 · diagnosisThe decisive evidence is an absence.
There are no cooperative-pool worker threads in the process at all. The async work was never slow — it never started.
Task { } inherits the enclosing isolation. Inside @MainActor code the child task is main-actor-isolated, and sem.wait() blocks the one thread that main-actor work can run on. The task is waiting for a thread that is waiting for the task.
Repair: do not block to wait for async work — await it. If a synchronous boundary is genuinely unavoidable, Task.detached inherits no isolation and will complete, but it still burns a thread, so it belongs only at a process edge. Experiment E7 isolates exactly this with all three variants.
Stage 2 · “Sometimes the export count is wrong”
The freeze is fixed. Now the completion message under-reports: it claims fewer photos than were actually exported, and the number changes between runs. Nothing crashes. Answer first: a wrong-and-varying number across threads — which tool tells you whether two threads are touching the same memory without synchronisation?
Step 1 · the toolNote that it is a different binary.
Thread Sanitizer — and you must run the sanitizer build, not the plain one. You cannot enable this after the fact on a binary that was not compiled for it.
$ ./incident_tsan run-bRead the report’s two stack traces, not just the summary line. The bug is wherever those two stacks meet.
Step 2 · diagnosis, and the trap inside itA data race on the tally — plus the reason this stage is dangerous.
TSan reports both a Swift access race and a data race on the counters, with two stacks meeting inside the export closure. tally.exported += 1 is a read-modify-write with no synchronisation; concurrent increments interleave and overwrite each other, so the total is low and varies.
Repair: put the counters behind a Mutex, make the tally an actor, or — usually best — have each chunk return its own subtotal and sum them at the join, which removes the shared state instead of guarding it.
Notice what this stage does not prove. Build the plain binary at -O and the count often comes out exactly right. The race is still there. “It printed the right number” is not evidence — see E2.
Stage 3 · “We are admitting more export jobs than the licence allows”
The licence permits 2 concurrent export jobs. Telemetry shows 8 being admitted. The code that enforces the limit is inside an actor, and the team’s position is that an actor makes this impossible. Answer first: what will Thread Sanitizer say about this, and why does that answer not settle the question?
Step 1 · run it anywayThe result is the point.
$ ./incident_tsan run-c
granted=8
(no ThreadSanitizer output)It is clean. It will stay clean no matter how long you run it, because there is no data race here — the actor is doing its job perfectly and serialises every individual access.
So put the sanitizer down and read the admission method instead. Ask a different question: what does this code assume is still true on the line after an await?
Step 2 · diagnosisActor reentrancy — a logical race, not a data race.
The invariant spans a suspension:
guard slotsFree > 0 else { return false } // checked
await Task.yield() // actor RELEASED - others run to completion
try? await Task.sleep(...) // still released
slotsFree -= 1 // acted on a stale factAt each await the actor is released and other queued jobs run. All eight callers pass the guard before any of them resumes, so all eight are admitted.
Repair, in order of preference: (1) re-check the invariant after the suspension; (2) reserve the slot before suspending and compensate on failure; (3) serialise the whole operation with an explicit in-flight flag — last resort, because it reintroduces the queueing reentrancy exists to avoid. E5 implements and measures all three.
Stage 4 · “Exports work, but the whole machine gets sluggish”
Correct output. No crash. No bad data. But during an export the system — not just the app — becomes unresponsive, and the export is far slower than the core count suggests it should be. Answer first: the output is correct, so no sanitizer will help. What is the observable resource you should measure, and with which tool?
Step 1 · a different reading skillSame tool as stage 1, used completely differently.
Sample it while it runs, then count thread entries rather than reading stacks:
$ ./incident run-d >/dev/null 2>&1 & PID=$!; sleep 0.25
$ sample $PID 1 20 -file /tmp/stage4.txt
$ grep -c 'Thread_' /tmp/stage4.txt
$ sysctl -n hw.ncpuCompare those two numbers. Then read the “Sort by top of stack” summary at the end of the report and ask what nearly all of those threads are actually doing.
Step 2 · diagnosisThread explosion — and the reflex fix that does not work.
sample lists ~145 thread entries on a 14-core machine. The “Sort by top of stack” summary shows the overwhelming majority in __semwait_signal and __workq_kernreturn — that is, sleeping, not working.
Dispatch’s global queues overcommit: a blocked work item is indistinguishable from a slow one, so libdispatch brings up another thread to keep the queue’s width occupied. Hundreds of blocking items become dozens of threads, each with a stack and a scheduler share, and the cost lands on the whole system.
Repair: stop blocking. await suspends the task and frees the thread, so the cooperative pool stays near the core count. If you must stay on Dispatch, bound at the submission site — and note that moving the semaphore inside the work item does not help at all, because a blocked permit-wait occupies a worker thread exactly like blocked work does. E8 measures all four variants: 73, 73, 8, 13 threads.
| Stage | Symptom you were given | Real defect | Tool that works | Tool that wastes your time |
|---|---|---|---|---|
| 1 | Permanent freeze on Export | Task { } + sem.wait() on @MainActor | sample — every thread’s stack | Thread Sanitizer (nothing races) |
| 2 | Export count wrong and varying | Data race on the tally | Thread Sanitizer | sample (nothing blocks) |
| 3 | 8 jobs admitted against a 2-job licence | Actor reentrancy across await | Reading the code for suspension points | Thread Sanitizer (clean, and always will be) |
| 4 | Whole machine sluggish, output correct | Thread explosion | sample, counting threads not reading stacks | Any sanitizer (the output is correct) |
The one sentence this drill exists to install
A clean Thread Sanitizer run means no data race was observed on the paths you exercised — it does not mean your program is correct. Stages 3 and 4 are both clean, and both are broken.
Record your answer · 3
Score your own tool choices.
For each of the four stages, write the tool you reached for before opening the evidence, and whether it would have worked. The score that matters is not four out of four — it is whether you can now say, in one sentence per stage, why the tool that failed was the wrong question to ask.
F · Interview questions
Sixteen questions, with the follow-up that comes next.
Four categories, because interviewers ask four different kinds of question and candidates prepare for only the first. Answer aloud before expanding. Every answer key below is sourced from this chapter; the follow-up is the second question, the one asked after a correct first answer to find the edge of what you actually know.
Category 1
Explain.
These test whether you understand a mechanism or have memorised a rule. The tell is whether you can say why the design is that way.
Explain · Q1What does it mean that os_unfair_lock is “unfair”, and why did Apple ship it that way?
Unfairness is a precise claim, not a warning label. Apple: “an unlocker could potentially reacquire the lock immediately, before an awoken waiter gets an opportunity to attempt to acquire the lock. This may be advantageous for performance reasons, but also makes starvation of waiters a possibility.” documented
The advantage is avoided context switches: a fair lock reserves the lock for the next waiter, so the unlocker cannot reacquire and must yield the CPU. Measured on Darwin’s two pthread policies, fair handoff cost ~180× throughput and ~190× more context switches per acquisition. measured here
The sentence that earns the point: the lock still records ownership, which is what lets the system resolve priority inversions — unfairness and ownership are independent properties.
Follow-ups. (a) If unfairness permits starvation, why is it the default? Because starvation requires a pathological access pattern while the context-switch cost is paid on every handoff — and Darwin made first-fit the default pthread policy in macOS 10.14. (b) What would you pick if you genuinely needed a bounded wait? A serial queue (strict FIFO) or an explicit FAIRSHARE mutex — and I would say out loud what it costs.
Explain · Q2Distinguish atomicity from memory ordering. When does an atomic need something stronger than .relaxed?
Swift’s own proposal states the distinction: “All accesses of a particular atomic value get serialized into some global sequential timeline, no matter what thread executed them. However, this alone does not give us a way to synchronize accesses to regular variables, or between atomic accesses to different memory locations.” documented
That is the whole thing: atomicity is about one location; ordering is about everything else you wrote around it. .relaxed is right for a pure counter that publishes nothing. You need release/acquire the moment the atomic acts as a flag for other data — the writer fills the payload then stores the flag .releasing, and a reader that loads it .acquiring is guaranteed to see the payload.
Follow-ups. (a) Why can you not write a “releasing load”? The three ordering types are separate structs precisely so that combination is a compile error. (b) What does .sequentiallyConsistent add over acquire/release? A single total order all threads agree on — needed when two independent flags’ relative order matters. (c) Which C++ ordering has no Swift spelling? memory_order_consume; listed as not yet adopted.
Explain · Q3What exactly does an actor guarantee, and what does it not?
It guarantees mutual exclusion of execution — “no two functions will ever execute concurrently on any given actor” — and compile-time isolation of the state. documented
It does not guarantee your invariant survives an await: “reentrant actors are thread-safe but are not automatically protecting from the ‘high level’ kinds of races.” Measured: two function bodies mid-flight on one actor, one executing, and six concurrent withdrawals against a balance of 100 all succeeded, reaching −500 — with Thread Sanitizer silent. measured here
Follow-ups. (a) So how do you protect an invariant across a suspension? You do not — you restructure. Apple’s own guidance: “synchronous code in an actor provides a critical section, whereas an await interrupts a critical section.” (b) Why did Swift choose reentrancy at all? To let priority work: a serial queue is strict FIFO and can only boost the queue; reentrancy lets the runtime reorder high-priority work ahead.
Explain · Q4Compare DispatchSemaphore and NSCondition. When is each one wrong?
A semaphore stores permits — the count survives a signal() sent with nobody waiting, and a later wait() consumes it. A condition variable stores nothing — it is a lock plus a parking lot, and the predicate is the contract. Reproduced 3 of 3: a signal() into an empty NSCondition waiting room is lost forever. measured here
A semaphore is wrong when you need to test a state, and wrong whenever used to make async code synchronous. A condition is wrong when you wanted a stored permit.
Follow-ups. (a) Why must the predicate test be a while and not an if? “Signaling a condition does not guarantee that the condition itself is true. There are timing issues involved in signaling that may cause false signals to appear.” Measured: 618 spurious wakeups absorbed by the loop, versus 208 underflows without it. (b) signal() or broadcast()? Measured 1 of 8 versus 8 of 8. Default to broadcast() plus a strict predicate loop.
Explain · Q5What is the ABA problem, and what does Swift ship to address it?
“A freshly allocated object often happens to be placed at the same memory location as a recently deallocated one. Therefore, two successive loads of a simple atomic pointer may return the exact same value, even though the pointer may have received an arbitrary number of updates between the two loads.” documented A compare-and-exchange therefore succeeds on stale reasoning.
Swift ships WordPair (macOS 15) for double-wide atomics so the second word can hold a version tag. Demonstrated deterministically: the single-word CAS succeeded across A→B→A; the tagged CAS correctly failed. measured here
Follow-ups. (a) Is WordPair always available? No — “this type only conforms to AtomicRepresentable on platforms that support double wide atomics.” (b) Does tagging solve it completely? No. The tag can wrap. It makes ABA improbable, not impossible, and it does not solve memory reclamation, which is a separate unsolved problem.
Explain · Q6Concurrency versus parallelism on Apple silicon — what is the practical consequence?
Concurrency is how many things are in flight (your design choice); parallelism is how many execute at this instant. On Apple silicon the second is activeProcessorCount, and Apple is explicit that it can shrink: it “reflects the actual number of active processing cores … including boot arguments, thermal throttling, or a manufacturing defect”, and “the availability of P cores is not guaranteed.” documented
The consequence: the width of every pool you depend on is a runtime value, not a build-time constant. Measured, oversubscription at 3× the core count left throughput flat while Preempted time rose from 19.7% to 71.0% of intervals. measured here
Follow-ups. (a) So what improves if you fix it? Latency and the tail, not throughput — the fastest-to-slowest spread went from ~1.4× back to ~1.06×. Promise the right thing. (b) What is Apple’s sizing rule? “Scale the thread count to match the CPU core count … Do not scale your thread count based on your workload either.”
Category 2
Diagnose.
These give you a symptom and some evidence and watch which question you ask next. Naming a tool before naming the question you are asking is the common failure.
Diagnose · Q7The app freezes permanently. You have a sample of the process. What distinguishes a deadlock from a slow operation, and from a livelock?
Three distinguishable signatures. Deadlock: 0% CPU, and the same frame in 100% of samples — measured, 92 of 92 samples with __psynch_mutexwait at the top of stack on two threads simultaneously, each inside the other’s lock. Slow operation: high CPU, stacks move between samples, top of stack is your own work. Livelock: high CPU, stacks churn but only through retry/back-off frames, and your own progress logging is flat.
The discriminating question is “did the stack move, and is anyone on CPU?” measured here
Follow-ups. (a) How do you tell which lock? The frame below __psynch_mutexwait names the calling function; cross-reference the two threads’ functions to read the cycle from source lines. (b) sample or spindump? sample for one process and no privileges; spindump for system-wide and cross-process waits — and it needs sudo, exiting 77 with no report at all when unprivileged.
Diagnose · Q8Thread Sanitizer is clean, but a user’s balance went negative. What happened, and what is your next move?
A logical race, not a data race. TSan reports unsynchronised memory accesses; if every access goes through the same lock there is nothing for it to report — yet a check-then-act sequence across two separate critical sections can still break the invariant.
Reproduced exactly: a correctly-locked ledger produced −100 with 0 bytes of TSan output, while an unsynchronised counter in the same binary produced warnings. measured here
Next move: stop looking for a memory bug and look for a compound operation — find every place two atomic calls are sequenced and ask whether the state can change between them.
Follow-ups. (a) What is the repair? Make the decision and the action one critical section. (b) Does Swift 6 strict concurrency catch this? No — it eliminates data races; a logical race is a design error the compiler cannot see. (c) Is a clean TSan run ever proof? No. It reports races it observes; unexercised interleavings are invisible, and it does not support fences, where it may report false positives.
Diagnose · Q9Thread Performance Checker reports a priority inversion in code that has no locks. Explain it.
Almost certainly a semaphore or dispatch group used to wait for asynchronous work. Apple: “When you use these primitives, the system can’t automatically propagate priority from the higher-priority thread to the lower-priority thread.” documented There is no owner to boost — “the runtime doesn’t know what thread will signal the sync primitive.”
Measured directly: the holder’s scheduling band moved 4 → 31 behind a lock and stayed at 4 behind a semaphore, 3/3 versus 0/3 trials — with the same critical section taking 277 ms boosted and 1017 ms unboosted. measured here
Follow-ups. (a) Two fixes Apple names? Remove the emulated-synchronous wait entirely; or ensure the waiter’s QoS is the same as or lower than the signaller’s. (b) You need to suppress the warning while you refactor — how? PERFC_SUPPRESSION_FILE, with class: / method: lines.
Diagnose · Q10The app stops making progress. No lock is held, nothing crashed, and there is no cycle you can find. Where do you look?
Cooperative-pool exhaustion. Swift’s pool is sized to the core count — it “will only spawn as many threads as there are CPU cores” documented — so N blocked tasks where N ≥ core count wedges everything, including the task that would unblock them.
Reproduced: 18 detached tasks blocking in DispatchSemaphore.wait(), exactly 14 got threads (= activeProcessorCount), 4 never scheduled, the releaser never ran. measured here
Look for a synchronous helper hiding a blocking wait, because the compiler refuses the direct form.
Follow-ups. (a) How would you confirm it cheaply? Run with LIBDISPATCH_COOPERATIVE_POOL_STRICT=1. It collapses the pool to one thread — measured — so the first blocking task wedges immediately and the bug reproduces on the first try instead of only under load. Say clearly that this is undocumented and development-only. (b) Why is GCD different? It postpones rather than avoids: the same shape survives to 64 waiters and fails at 128, paying with 70 live threads.
Diagnose · Q11A .barrier write is producing corrupted reads. The code looks right. What do you check first?
Which queue it was submitted to. “If the queue you pass to this function is a serial queue or one of the global concurrent queues, this function behaves like the dispatch_async function.” documented A barrier on DispatchQueue.global() is silently a plain async — no warning, no assertion, no crash.
Measured: on a private concurrent queue, peak simultaneous barrier blocks = 1; on a global concurrent queue, 38. measured here Fix: create the queue yourself with DispatchQueue(label:attributes: .concurrent).
Follow-ups. (a) Why can’t the global queues honour it? They are shared process-wide — a barrier there would have to serialise every client of that queue. (b) The 38 is itself a symptom of what? Thread explosion: those blocked items grew the pool to ~38–40 threads. (c) A serial queue also showed 1 — did the barrier work there? No. The documentation says it degrades there too; the serialisation comes from the queue being serial.
Diagnose · Q12Your profile looks fine — nothing is hot — but wall-clock latency is five times the CPU time. What are you not looking at?
The state your sampling profiler cannot see. sample records stacks, so “ready to run but no core available” is invisible to it. Only Thread State Trace (in the System Trace template) reports Runnable and Preempted at all.
Measured: at 14 threads on 14 cores, (Preempted + Runnable) ÷ Running is 24.7%. At 42 threads on the same machine it is 262.1%, with Running time essentially unchanged — the extra threads bought nothing but waiting-to-run time. measured here
Follow-ups. (a) And the sampling trap inside the trap? Sample count measures how long a thread existed, not how much CPU it used — measured, four blocked threads each showed 277 samples while the one busy thread showed 137. Read the leaf frame. (b) Which profiler should you be using anyway? CPU Profiler, not Time Profiler: Apple says to prefer it “because it’s more accurate and more fairly weights software consuming CPU resources” — Time Profiler samples on a timer and is biased against fast-clocking cores.
Category 3
Choose a primitive.
These are scored on whether you name what you rejected and why. “I would use a lock” is half an answer.
Choose · Q13You need a read-mostly in-memory index, read from many threads, written rarely. Readers-writer lock?
Only if the read critical section is long. Measured at 100% reads with a one-element read, pthread_rwlock_t was 30× slower than a plain OSAllocatedUnfairLock; it only won above roughly 4,000 element-reads per critical section. A separate sweep found it 10.8×–14.1× worse at every write ratio from 0% to 50% when the section was short — so “read-mostly” alone is not the criterion. measured here
It also has no single owner, so it gets no priority-inversion resolution. Default answer: a plain exclusive lock, and measure before adopting a rwlock.
Follow-ups. (a) Other rwlock gotchas? “The results of acquiring a read lock while the calling thread holds a write lock are undefined”, and “to prevent writer starvation, writers are favored over readers” — though measured, the reader was de-prioritised, not starved, completing 43,871 reads against four continuous writers in 500 ms. (b) Alternative shapes? Copy-on-write with an atomically swapped immutable snapshot: readers take no lock at all.
Choose · Q14Limit image decoding to at most four concurrent operations. What do you use, and what do you avoid?
A counting limit. From already-async code, a TaskGroup with a bounded in-flight window; at the thread layer, DispatchSemaphore(value: 4) — exactly what Apple documents it for: “Passing a value greater than zero is useful for managing a finite pool of resources, where the pool size is equal to the value.” documented Both held the bound exactly at 4 with 64 tasks. measured here
Avoid: a mutex (expresses exclusion, not a count of four), a recursive lock, and using the semaphore to await a result rather than to bound concurrency.
Follow-ups. (a) Why can the semaphore not just be a mutex with a counter? No ownership, so nothing to boost — and you would have to hand-roll the blocking. (b) What breaks if you take the semaphore inside an async function? It will not compile — “Await a Task handle instead” — and if you route around it with a synchronous helper you block a cooperative thread. (c) What must you be careful about at teardown? “Calls to signal() must be balanced with calls to wait(). Attempting to dispose of a semaphore with a count lower than value causes an EXC_BAD_INSTRUCTION exception.”
Choose · Q15New Swift 6 code, macOS 15 minimum, one struct of state touched from both a synchronous deinit and an async path. Lock or actor?
Lock — specifically Mutex<State> from Synchronization. An actor cannot be reached from a synchronous deinit. Mutex “offers non-recursive exclusive access to the state it is protecting by blocking threads attempting to acquire the lock”, the state lives inside the lock so it cannot be touched without it, and withLock is explicitly allowed in async contexts because no suspension can occur between lock and unlock. documented
Uncontended it costs ~1.8 ns/op, indistinguishable from a raw atomic. measured here
Follow-ups. (a) What if the deployment target were macOS 13? OSAllocatedUnfairLock(initialState:) — same shape, available from macOS 13. (b) What must you never do inside that withLock? Call out to a delegate, completion handler or notification — you do not control what it acquires — and never re-enter it: measured on macOS 26.3 that is BUG IN CLIENT OF LIBPLATFORM: Trying to recursively lock an os_unfair_lock and a SIGKILL, not a hang.
Choose · Q16You are told “just use an atomic, locks are slow.” Respond.
Two measurements answer it. Uncontended, the lock is free — Mutex, OSAllocatedUnfairLock and a relaxed atomic add were indistinguishable at ~1.8 ns/op, so the premise is wrong at the low end. Contended, the atomic is worse — 3.5–3.9× slower than the unfair lock across 14 cores, and a weak-CAS loop 6.7–7.1× slower with 2,187,428 retries for 1,000,000 increments. measured here
The reason is structural, not incidental: lock-freedom is a guarantee about blocking, not speed. Swift requires atomics be lock-free but explicitly not wait-free. Lock-freedom converts waiting into repeated work — every loser retries immediately and burns a core doing it.
And the correctness point matters more than either number: one atomic makes one operation on one location indivisible; it cannot make two operations one transaction.
Follow-ups. (a) So when is an atomic right? A hot counter, a one-shot flag, a published-once pointer — where the invariant genuinely is one location. (b) What would you distrust about your own benchmark? Build flags. The same Mutex benchmark reported ~28× the true cost under plain -O without -wmo, because the generic never specialised — any micro-benchmark of Synchronization types in a Swift script is measuring the optimiser, not the lock.
Category 4
Repair the code.
These put broken code in front of you. The scored behaviour is naming the class of bug before reaching for a fix.
Repair · Q17This compiles, every access is locked, and it still loses money. Fix it and name the class of bug.
final class Ledger: @unchecked Sendable {
private let lock = NSLock()
private var balance = 100
func canWithdraw(_ n: Int) -> Bool { lock.lock(); defer { lock.unlock() }; return balance >= n }
func withdraw(_ n: Int) { lock.lock(); defer { lock.unlock() }; balance -= n }
}
// caller, from two threads:
if ledger.canWithdraw(100) { ledger.withdraw(100) }Check-then-act, a logical race. Each method is atomic; the sequence is not, so both threads pass the check and both withdraw. This exact code produced a final balance of −100, deterministically, with zero Thread Sanitizer output. measured here
Repair — one critical section that both decides and acts:
func withdrawAtomically(_ n: Int) -> Bool {
lock.lock(); defer { lock.unlock() }
guard balance >= n else { return false }
balance -= n
return true
}Follow-ups. (a) Name the general rule. A lock protects an invariant, not a variable — if the invariant spans two calls, the critical section must too. (b) What is the API-design lesson? Do not export a predicate whose truth cannot survive the caller’s next statement. Export the compound operation. (c) Would making balance an Atomic<Int> fix it? No — and this is the trap. One atomic makes one operation on one location indivisible; it cannot make two operations one transaction. A CAS loop would work, because compare-and-exchange is the compound operation.
Repair · Q18This actor admits eight jobs against a licence of two. It has no data race. Fix it.
actor Licence {
private var slotsFree = 2
func admit() async -> Bool {
guard slotsFree > 0 else { return false } // checked
await audit() // actor RELEASED here
slotsFree -= 1 // acted on a stale fact
return true
}
}Actor reentrancy across a suspension — a logical race the sanitizer will never see, because the actor serialises every individual access correctly. At the await the actor is released and other queued jobs run to completion, so all eight callers pass the guard before any of them resumes.
Repair 1 — re-check after the suspension (cheapest; the suspended work still ran):
guard slotsFree > 0 else { return false }
await audit()
guard slotsFree > 0 else { return false } // <-- the repair
slotsFree -= 1Repair 2 — reserve before suspending (stronger; needs a correct compensation path):
guard slotsFree > 0 else { return false }
slotsFree -= 1 // committed BEFORE any await
await audit()
if Task.isCancelled { slotsFree += 1; return false }Follow-ups. (a) Is there a third repair? Yes — an explicit in-flight flag with a continuation queue. It works (measured maxInFlight=1) but reintroduces the queueing, ordering and starvation questions reentrancy exists to avoid, so it is a last resort. (b) Would a lock inside the actor fix it? No, and it would be worse: a blocking lock held across an await is a forward-progress violation. (c) How would you catch this in review? Read every await inside an isolated method and ask what the line after it assumes is still true.
Repair · Q19This “bounded” decoder still creates 73 threads. Fix it without changing the bound.
let gate = DispatchSemaphore(value: 4)
for _ in 0..<256 {
DispatchQueue.global(qos: .utility).async(group: group) {
gate.wait(); defer { gate.signal() } // <-- wait is INSIDE
decode() // blocking
}
}The permit bounds concurrency, not thread creation. A blocked permit-wait occupies a Dispatch worker thread exactly like blocked work does; libdispatch cannot tell a blocked item from a slow one, so it brings up another thread to keep the queue’s width occupied. Measured: 73 peak threads with the semaphore, and 73 without it — the “fix” changed nothing. measured here
Repair — move the same wait to the submission site, so unstarted work never occupies a thread. Measured: peak drops to 8.
for _ in 0..<256 {
gate.wait() // <-- on the SUBMITTING thread
DispatchQueue.global(qos: .utility).async(group: group) {
decode()
gate.signal() // a permit frees on completion
}
}Follow-ups. (a) What is the better answer entirely? Structured concurrency — a task-group sliding window reaches the same bound of 4 by suspending rather than blocking, and peaked at 13 threads on 14 cores. (b) Why is the submitting-thread version still not great? It blocks whichever thread does the submitting; do it from a context that can afford to block, never the main thread. (c) How would you have caught this without reading the code? Counting threads in a sample report rather than reading their stacks.
Repair · Q20This consumer occasionally crashes on an empty buffer. One keyword fixes it. Which, and why?
func consume() -> Int? {
cond.lock(); defer { cond.unlock() }
if items.isEmpty { cond.wait() } // <-- the bug
return items.removeFirst() // traps when the wakeup was spurious
}if must be while. A wakeup is a hint, not a promise: “Signaling a condition does not guarantee that the condition itself is true. There are timing issues involved in signaling that may cause false signals to appear.” documented Measured on identical workloads, the while version absorbed 618 spurious wakeups with 0 underflows; the if version produced 208 underflows, each one a removeFirst() trap in real code. measured here
while items.isEmpty && !closed {
if !cond.wait(until: deadline) { return nil } // and give it a finite deadline
}Follow-ups. (a) Why does the deadline matter? An unbounded wait turns a stall into a hang; a finite one lets you shed load. (b) The producer signalled and the consumer still never woke — how? A lost wakeup, the other half of the contract: NSCondition stores no permits, so a signal() with nobody parked is discarded. The fix is the same — test the predicate before waiting, because the predicate is the state. (c) Should the producer signal() or broadcast()? Measured 1 of 8 versus 8 of 8 waiters proceeding. Default to broadcast() plus the strict loop, and say what the herd costs.
G · Drill
Answer first, then read the explanation.
Each scenario has one defensible first move. Say it out loud before clicking, then say why the other three are premature rather than merely wrong.
Scenario 01 · Stutter during import
Scrolling stutters only while a background import runs. Main-thread stacks show it parked inside a lock the importer also takes. First move?
Scenario 02 · The fix that did not fix it
A colleague wrapped a synchronous parse in Task { } inside a @MainActor control’s action. The freeze moved but did not go away. Why?
Scenario 03 · What the actor did not promise
State is inside an actor and every method is isolated. Which guarantee do you not have?
Scenario 04 · The atomic that did not help
A counter was made atomic. The tearing stopped; the bug did not. What is the most likely explanation?
Scenario 05 · Beachball in the field
A user reports a five-second freeze you cannot reproduce, but they can. You have terminal access to their Mac. What do you ask for first?
Scenario 06 · Bounding fan-out
Image decoding must be limited to at most four at a time, from already-asynchronous callers. Which primitive expresses that directly?
Record your answer · 1
Defend one synchronisation choice.
Pick one piece of shared state in your own control. Write: the invariant in one sentence → the primitive you chose → the two alternatives you rejected and why → what you would measure to prove the choice was right.
Record your answer · 2
Rehearse one hang diagnosis.
For a freeze you have actually seen, write the full chain: symptom → competing hypotheses → first discriminating instrument → the evidence that would rule each hypothesis out → the smallest fix → the regression guard.
Expect these next
Interviewer follow-ups.
These are the second questions — the ones asked after a correct first answer, to find the edge of what you actually know. Practise saying “I would have to measure that” where it is the true answer.
“Why not just make everything an actor?”
Name a synchronous accessor you cannot remove — a deinit, an AppKit override, a C callback — and the executor-hop cost on a hot path. Then say where you would use one.
“Your lock is fine. Now the profile shows 40% in lock wait.”
Separate hold time from acquisition count, then choose: shrink the critical section, shard the lock, or make the data immutable and swap a pointer.
“How would you prove there is no deadlock?”
You cannot, by testing. You can state the lock order, show it is total, and show every acquisition site obeys it — and add an assertion that the order is respected in debug builds.
“Thread Sanitizer is clean. Are you confident?”
No. TSan reports races it observes; unexercised interleavings are invisible. Say what coverage the stress test had, and what it did not reach.
“Where does QoS come from for this task?”
Child tasks inherit from the parent; Task { } inherits isolation, priority, and task-local values; Task.detached inherits none and defaults to .medium.
“Can a serial queue starve anything?”
Not by unfairness — it is strict FIFO. It starves by ordering: high-priority work waits behind everything already queued, which is precisely the case actor reentrancy was designed to improve.
“Your framework is called from an app you do not control. What do you promise?”
State the threading contract in the public API: which methods are main-thread-only, which are safe from any thread, and what ordering callbacks have. Then make it checkable with @MainActor or an assertion.
“What would make you choose a semaphore in 2026 code?”
Bounding concurrency to a resource limit, from code that is already asynchronous and never blocks a cooperative thread. Not to await a result, and never on the main thread.
H · Primary sources for chapters 5–7
Apple documentation, WWDC sessions, and macOS manual pages
Every quoted sentence in this chapter comes from one of these. Behaviour and availability drift between releases — re-check the API pages against your deployment target before repeating a claim in an interview.
| WWDC session or Tech Talk | Year | Timestamp locators verified | Used in this chapter for |
|---|---|---|---|
| WWDC17 706 · Modernizing Grand Central Dispatch Usage | 2017 | 12:41–15:25 · 15:25–16:45 · 19:00–19:30 · 23:58–24:16 | Contention and the fair/unfair staircase; lock ownership and the three-way primitive taxonomy; queue.sync ownership transfer; queue priority inversion |
| WWDC21 10254 · Swift concurrency: Behind the scenes | 2021 | none — see note below | Thread explosion; overcommit; the forward-progress contract; cooperative pool width; why actors are reentrant |
| WWDC21 10133 · Protect mutable state with Swift actors | 2021 | — | Actor isolation background |
| WWDC22 110350 · Visualize and optimize Swift concurrency | 2022 | ?time=624 · ?time=709 · ?time=1066 | Swift Tasks / Swift Actors instruments; thread-pool exhaustion and runtime deadlock |
| WWDC22 110351 · Eliminate data races using Swift Concurrency | 2022 | 07:29–08:16 · 19:03–19:49 | @unchecked Sendable; low-level versus high-level races |
| Tech Talk 110147 · Tune CPU job scheduling for Apple silicon games | 2022 | page exposes no ?time= anchors | P/E cores; QoS semantics and priority bands; which primitives can resolve priority inversion; thread-count guidance |
| WWDC23 10248 · Analyze hangs with Instruments | 2023 | ?time=555 busy · ?time=1552 asynchronous · ?time=2157 blocked · ?time=2435 blocked ≠ unresponsive | The three hang kinds; Thread State Trace and its Narrative column |
| WWDC23 10170 · Beyond the basics of structured concurrency | 2023 | — | Task trees and cancellation background |
| WWDC24 10169 · Migrate your app to Swift 6 | 2024 | 14:45–15:36 · 16:03 | nonisolated(unsafe); lazy global initialisation |
| WWDC25 308 · Optimize CPU performance with Instruments | 2025 | ?time=530 Profilers · ?time=845 Processor Trace | Prefer CPU Profiler over Time Profiler; timer aliasing; Processor Trace |
| WWDC25 268 · Embracing Swift concurrency | 2025 | 08:28–08:34 · 13:56–14:55 · 15:03–15:14 · 21:59–22:06 | Interleaving; nonisolated; shared mutable state needs a lock |
An honesty note about timestamps. Apple session pages expose ?time= anchors only for their listed key moments and chapters, not per sentence. WWDC23 10248 and WWDC25 308 have rich chapter lists, and those anchors are used above. WWDC21 10254’s page exposes only five code-sample key moments, none of which cover the thread-explosion or forward-progress passages, so those quotes are cited to the session and not to a fabricated timestamp. Ranges written as mm:ss–mm:ss come from the session page’s own embedded transcript segments. WWDC15 718 is deliberately not cited: its URL resolves but the page carries no session-specific title, so its content could not be verified.
How to read the labels on every claim in this chapter
documented — Apple documentation, an official WWDC or Tech Talk transcript, Swift Evolution, or a Darwin manual page or SDK header on macOS 26.3. Quoted verbatim with the source named beside it.
measured here — observed on one machine: Apple M4 Pro (10 performance + 4 efficiency cores), macOS 26.3 (25D125), 48 GB, 16 KiB pages, Swift 6.2.4, Apple clang 17.0.0, Xcode 26.3. Evidence of a mechanism, never a benchmark. Reproduce; do not quote.
inference — our reasoning joining the two above. Marked so you can disagree with the reasoning without doubting the evidence.
negative result — something that was tried and did not show what was expected. These bound what a chapter is allowed to claim, and they are kept deliberately visible.
correction — a correction to an earlier reading published on this page, naming what was wrong and why. This page’s whole method is labelled provenance, so its own errata carry a label too rather than disappearing into a rewrite.
version-sensitive — observed behaviour of this OS build that must be re-measured on another before it is relied on.
Provenance for chapters 5, 6 and 7. Everything labelled measured here in those three chapters and in this lab was most recently re-run in full on 2026-09-22, with 51 fixture assertions passing and 0 failing.
Scope of the evidence, stated plainly. Also used, and available on this machine rather than on the web: the macOS 26.3 manual pages sample(1), spindump(8), pthread_rwlock_rdlock(3), pthread_mutexattr_settype(3), pthread_cond_wait(3), plus the SDK’s Dispatch.swiftinterface, NSLock.h, sys/qos.h and limits.h. Every Swift snippet in this chapter was type-checked with swiftc -typecheck -swift-version 6 from a clean copy before publication, and every C snippet compiled with clang -Wall -Wextra; the ones that demonstrate a defect are marked as such in their own comments and in the page. Four things this chapter does not have: no Instruments GUI session was opened and no screenshot appears anywhere — the System Trace figures come from headless xctrace record/export; Thread Performance Checker was never exercised, because it is an Xcode Run-action tool with no supported command-line invocation, so its findings are quoted from documentation while the underlying kernel behaviour was measured directly instead; spindump’s privileged path was never run, so everything about -onlyBlocked output content is documentation rather than observation; and no real application hang was ever captured, so the main-thread-hang case rests on documentation plus tool configuration plus a synthetic reproduction.
Chapter · input and output
I/O.
What a descriptor is, what a read actually costs when the data is already in memory, where the buffer-size curve flattens, what mmap buys and what it charges, and the exact boundary at which data is durable rather than merely written. The durability section is the one most people get wrong, and it is the one with the sharpest measurements.
Where the time goes on an I/O path, and what your program promises the user about data that has been “saved”.
Separate system-call cost from device cost, and name the exact call that makes a write survive a crash.
APFS internals, the block layer, NVMe queueing, filesystem journal format, network stack internals below the socket.
The sentence to have ready
“write() returning is not durability, and fsync() is not the strongest guarantee available.” Both halves surprise people, both are documented in one manual page, and the cost between the levels is a factor of thousands.
1 · Descriptors and the path
What a read travels through.
A file descriptor is a small integer indexing a per-process table. Behind it sits an open file description with a seek offset, and behind that a file, a pipe, a socket, or a device. Almost every I/O surprise comes from forgetting which of those it is, because they honour the same calls with different promises.
Simplified diagram, not a model of APFS or the block layer. It exists to make one point: “slow I/O” splits into too many system calls and actually waiting for a device, and those have opposite repairs.
A short read is not an error
read(2) returns “the number of bytes actually read”, which may be fewer than requested. On a file at end-of-data that means the end; on a pipe or socket it means “this is what is here now”. Treating a short return as an error, or as the end, is the single most common I/O bug — and it is the whole of exercise 07.
The same is true of write
A write may accept fewer bytes than offered, particularly to a pipe or socket. Every correct writer is a loop. The fixtures in this chapter all use a write_fully helper for exactly that reason.
Descriptors are shared, offsets are shared
A descriptor duplicated by fork or dup shares the same file offset. Two threads calling read on one descriptor interleave unpredictably; use pread/pwrite, which take an explicit offset and do not move the shared one.
Blocking is a property of the descriptor
Not of the call. The same read blocks or returns EAGAIN depending on O_NONBLOCK. That flag lives on the open file description, so setting it affects every descriptor that shares it — including ones another part of the program obtained by duplication.
1A · The three levels
Descriptor, open file description, vnode.
Section 1 said a descriptor indexes a per-process table and that behind it sits an open file description with a seek offset. That sentence is the whole of this section’s content, and most people never use it. Every sharing rule, every surprising flag, and one genuinely nasty concurrency bug read straight off the picture.
Simplified diagram, not a model of the Darwin file-descriptor implementation. Every rule in the table below is one arrow in this picture.
| Property | Lives on | The consequence that bites |
|---|---|---|
| seek offset | open file description | dup and fork share it; a second open does not |
O_NONBLOCK, O_APPEND | open file description | Setting it affects every descriptor sharing the description, including one another subsystem got by dup |
FD_CLOEXEC | the descriptor | Not copied by dup. Re-set it, or use dup3 / O_CLOEXEC |
| the data | vnode | Two independent opens of one path share the bytes and nothing else |
after read(a,10): off(a)=10 off(b_dup)=10 off(c_reopen)=0
O_NONBLOCK set on a: a=1 b_dup=1 c_reopen=0
FD_CLOEXEC set on a: a=1 b_dup=0
pread(a,8,offset=0): off before=10 after=10
fork: child read 100 bytes -> child offset=100, PARENT offset=100measured here Reproduced in this session. The sentence to carry: “O_NONBLOCK is not a property of your descriptor, it is a property of the thing your descriptor points at — which somebody else may also be pointing at.”
THE SHARED OFFSET IS NOT A THEORETICAL RACE. lseek then read is two operations with a shared mutable cursor between them. Eight threads × 20,000 reads on one descriptor, each record self-identifying so a stolen read is detectable:
lseek + read wrongRecord = 110448 shortRead = 13
pread wrongRecord = 0 shortRead = 0Sixty-nine per cent of reads returned the wrong record, and the program reported no error at any point. pread / pwrite take an explicit offset and never touch the shared one: zero wrong records, zero short reads, same run. measured here
A methodological note that is worth more than the result
The first version of that detector reported zero wrong reads for both variants. The race was always there; the detector was blind, because every offset was a multiple of the record size, so a stolen read looked byte-identical to a correct one. Making each record carry its own index is what made the failure visible. negative result
The rule: a test that cannot distinguish the failure from success is not evidence of correctness. That applies far beyond descriptors, and it is a good thing to say out loud when someone reports that a concurrency fix “made the test pass”.
2 · Buffering and the page cache
The system call is the cost.
When the data is already in the page cache — which, for a file you just wrote or recently read, it is — a read performs no device I/O whatsoever. What remains is the user-to-kernel transition, and that cost is nearly independent of how many bytes you move.
2AThe buffer-size curve, and where it flattensmeasured · same 64 MiB file, page cache warm
| Buffer | read() calls | ms | MB/s | |
|---|---|---|---|---|
| 1 B | 67,108,864 | 21,009.6 | 3.0 | pathological |
| 64 B | 1,048,576 | 331.5 | 193.0 | |
| 512 B | 131,072 | 43.0 | 1,488.7 | |
| 4 KiB | 16,384 | 8.2 | 7,840.2 | knee begins |
| 16 KiB | 4,096 | 4.6 | 14,013.2 | one page |
| 64 KiB | 1,024 | 3.4 | 18,712.5 | good default |
| 256 KiB | 256 | 3.2 | 19,956.2 | curve is flat |
| 1 MiB | 64 | 3.4 | 18,603.6 | no better — slightly worse |
Two readings. From 1 byte to 256 KiB is a factor of 6,600 in throughput, with the same file, the same filesystem and the same cache state. And past roughly 64 KiB there is nothing left to win — 1 MiB measured very slightly worse than 256 KiB.
“Bigger is better” is false past the knee, and a larger buffer costs cache locality and memory footprint. Being able to say where the knee is, and that you measured it, is the senior version of this answer; “64 KiB because that is what people use” is not.
measured here Apple M4 Pro, macOS 26.3, APFS on the internal SSD, file freshly written so the cache is warm. Reproduce; do not quote — the position of the knee depends on the storage and the filesystem.
RESOLVED · F_NOCACHE SHOWED NOTHING BECAUSE OF THE METHOD, NOT THE MACHINE. An earlier sweep set fcntl(fd, F_NOCACHE, 1) on the reader only and measured essentially the same numbers — 4 KiB reads 8.6 ms with it against 8.2 ms without — and reported that as a bounded negative. The flaw is now identified: F_NOCACHE asks the kernel not to retain data, so setting it on the reader leaves resident every page the writer already cached. Setting it on both ends never admits the data to the cache at all:
block warm ms cache-bypassed ms slowdown
4 KiB 35.2 1,429.4 41×
16 KiB 19.6 613.0 31×
64 KiB 16.5 471.7 29×
256 KiB 14.9 190.0 13×
1 MiB 14.4 83.3 6×So a genuine cache-bypassed measurement is available without elevation, and it puts a number on what the page cache is worth: up to 41× at small blocks. purge(8) is still the only way to evict pages that are already resident, and that still needs root — which is a narrower limitation than the one previously stated. measured here version-sensitive Storage-device dependent; reproduce, do not quote.
The transferable lesson is about evidence, not about fcntl. A negative result bounds a claim only as far as the method was sound. “I turned the cache off and nothing changed” was really “I turned off half the cache path”. When a control produces no effect, suspect the control.
“Buffered” is four questions, and only one of them is the disk
The curve above varies one knob: the size of the block handed to read. That is one layer of four, and the other three fail independently with different repairs. Isolate the library layer by changing only setvbuf while the user code stays fgetc, over 16 MiB:
stdio buffer ms MB/s
_IONBF (none) 5,852.5 2.7
1 B 5,541.1 2.9
64 B 285.0 56.1
4 KiB 195.1 82.0
64 KiB 193.0 82.9
256 KiB 193.1 82.8A 30× swing from the library buffer alone, with the same user code and the same bytes requested per byte consumed. Now put that beside the curve above, which reaches ~19,000 MB/s at 256 KiB: even a perfectly buffered fgetc caps at 83 MB/s, 230× slower. That residue is a fourth layer the curve never touches — sixteen million user-space function calls that never enter the kernel at all.
Simplified diagram, not a model of the Darwin I/O stack. Each cost was measured by holding the other layers fixed — reproduce; do not quote.
| Layer | Where it lives | The knob | Cost when wrong |
|---|---|---|---|
| 1 · your call pattern | your code | how many bytes you ask for per call | ~230× (83 vs 19,000 MB/s) |
| 2 · the library buffer | user space | setvbuf, FileHandle chunk, DispatchIO.setLimit | ~30× (5,852 → 193 ms) |
| 3 · the kernel page cache | kernel | F_NOCACHE, F_RDAHEAD, F_RDADVISE | 6–41× (box above) |
| 4 · the device | hardware | nothing you control | not isolated here |
measured here Layers 1–3 each isolated by holding the other two fixed. Two of the four layers never involve the kernel, which is why “is it buffered?” is not a question with one answer. Ask which of the four you mean, because the repairs are different and only one of them is the disk.
Prefer the library that already solved this
stdio’s fgets/getline, Foundation’s FileHandle and Data, and DispatchIO all buffer for you. Writing the loop by hand is for when you need to see or assert the syscall count. Knowing what the library does is what lets you explain a profile with read at the top.
https://developer.apple.com/documentation/foundation/filehandle
Read the whole file, or stream it
FileHandle.readToEnd() and Data(contentsOf:) are correct and simple when the file fits comfortably in memory. They are the wrong shape for a file whose size you do not control, because peak footprint becomes an input you did not choose.
https://developer.apple.com/documentation/foundation/filehandle/readtoend()
The page cache is shared and adaptive
It is not your app’s memory and it is not charged to your footprint. Clean file-backed pages your process maps are discardable under pressure, which is precisely why mapping a read-only asset is cheaper than copying it — the heap chapter’s section 3 makes the same point from the memory side.
Vectored I/O, for the same reason
readv/writev move several non-contiguous buffers in one call. It is the same lesson — amortise the transition — applied where the data is not already contiguous, and it avoids a copy you would otherwise make to concatenate.
2A · Sequential vs random
“Random access is slow” is a statement about two things the sentence leaves out.
It is one of the most confidently repeated claims in an interview, and on this machine it is true only inside a narrow corner. The two variables the claim omits are block size and cache state, and each of them can erase the effect on its own.
Page cache warm
block seq ms rand ms ratio
4 KiB 35.2 51.6 1.47×
16 KiB 19.6 19.8 1.01×
64 KiB 16.5 17.1 1.04×
256 KiB 14.9 14.2 0.96×
1 MiB 14.4 13.6 0.95×Cache bypassed · F_NOCACHE both ends
block seq ms rand ms ratio
4 KiB 1,429.4 5,112.0 3.58×
16 KiB 613.0 1,688.7 2.75×
64 KiB 471.7 648.1 1.37×
256 KiB 190.0 193.4 1.02×
1 MiB 83.3 85.0 1.02×The finding, in one sentence
On this machine, “random access is slow” is a statement about small blocks with a cold cache, and nothing else. Warm, the penalty is about 1.5× at 4 KiB and gone by 16 KiB. Cold, it is about 3.6× at 4 KiB and gone by a few hundred kilobytes. Both effects disappear in the same direction — upward in block size.
What that changes about the answer. A candidate who says “random I/O is slow, so I would add a cache” has skipped two cheaper questions: how cold is it, and how big is the block. Going from 4 KiB to 256 KiB removed the entire penalty here, which costs one constant and no new state to keep coherent. Design a cache when the block size is already right and the data is genuinely cold.
measured here Two independent runs. version-sensitive Report the transition as a region, not a number. The 16 KiB and 64 KiB cold rows moved between 1.4× and 3.1× across runs, and a separate session put the crossover lower. What is stable is the shape: large at 4 KiB cold, absent at 256 KiB. inference The mechanism is read-ahead and per-request overhead amortising as the request grows, not something this measurement isolates directly.
Why the warm column is so flat
A warm read is a copy out of the unified buffer cache. There is no seek, no device queue and no read-ahead to defeat — only the transition cost that section 2 measures. Access pattern has almost nothing to act on, which is exactly what the 1.01× row says.
Where the 4 KiB cold row’s cost comes from
Two things at once: 65,536 requests instead of 256, and a request too small for read-ahead to help. Random order removes the read-ahead benefit specifically, which is why the penalty is worst exactly where read-ahead would otherwise have been doing the most work. inference
The knob you have before a cache
fcntl(fd, F_RDAHEAD, 1) asks for read-ahead; F_RDADVISE asks the kernel to fetch a specific range. Neither was measured here, and both are named because the honest order is block size, then a hint, then a cache. documented man 2 fcntl
What this does not cover
One APFS volume on one internal SSD. Nothing here says anything about spinning media, where seek cost is physical and the claim is straightforwardly true; about network filesystems; or about the device’s own cache, which was never isolated. negative result No cold-boot measurement exists — purge(8) needs root.
3 · read vs mmap
Two ways to get bytes, and they fail differently.
read copies into a buffer you own. mmap puts the file’s pages into your address space and lets the fault handler fetch them. The second is often faster and is not a general upgrade, because it changes how errors arrive.
read into a buffer | mmap the file | |
|---|---|---|
| Cost per byte | One syscall per buffer, plus a copy | One page fault per page, no copy |
| Measured here, 64 MiB | 3.2 ms at 256 KiB buffers | 2.3 ms touching one byte per page |
| Memory accounting | Your buffer is dirty, and charged to you | Clean file-backed pages, discardable, not charged |
| An I/O error arrives as | A return value you can handle | SIGBUS — a signal, at an arbitrary instruction |
| If the file is truncated underneath | A short read | SIGBUS on touching the removed pages |
| Best for | Sequential streaming, bounded memory, files you do not control | Random access, large read-only assets, data read repeatedly |
The reason mmap is not the default answer
Look at the two SIGBUS rows. With read, a failing disk or a file that disappeared is a return value in a branch you already wrote. With mmap, it is a signal delivered at whichever instruction happened to touch the page — inside a library, inside a parser, anywhere. Turning that into a handled error is genuinely hard, which is why mapping is right for assets you ship and control and questionable for arbitrary user files on removable media.
Foundation exposes the safe version of this trade as Data(contentsOf:options:.mappedIfSafe) — “if safe” meaning Foundation declines to map where it judges the backing store unsuitable. Prefer that over hand-rolled mmap for files, and keep hand-rolled mmap for anonymous memory as in the heap chapter’s exercise 02. https://developer.apple.com/documentation/foundation/data/readingoptions
4 · Durability
Four levels, and a factor of 3,340 between them.
This is the section to know cold. Almost everyone knows fsync exists; far fewer can say what it does not promise on macOS, and the manual page says so with unusual bluntness.
What fsync(2) itself says
“Note that while fsync() will flush all data from the host to the drive (i.e. the ‘permanent storage device’), the drive itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order sequence.”
“Specifically, if the drive loses power or the OS crashes, the application may find that only some or none of their data was written. The disk drive may also re-order the data so that later writes may be present, while earlier writes are not. This is not a theoretical edge case. This scenario is easily reproduced with real world workloads and drive power failures.”
“For applications that require tighter guarantees about the integrity of their data, Mac OS X provides the F_FULLFSYNC fcntl.”
documented man 2 fsync on macOS 26.3.
| Level | What it guarantees | µs per 256-byte append | vs no sync |
|---|---|---|---|
write() only | Visible to other processes. Survives your process crashing. Does not survive power loss. | 1.2 | 1× |
fsync(fd) | Flushed from the host to the drive. The drive may still reorder and may not have committed it. | 23.0 | 19× |
F_BARRIERFSYNC | Ordering without durability: earlier flushed I/O “guaranteed to be persisted before any other I/O that would follow the barrier”, but “no assumption should be made on what has been persisted or not when this call returns”. | 168.7 | 140× |
F_FULLFSYNC | “Asks the drive to flush all buffered data to permanent storage … drains the entire queue of the device and acts as a barrier”. “The operation may take quite a while to complete.” | 4,012.6 | 3,340× |
documented Quoted guarantees from man 2 fcntl and man 2 fsync on macOS 26.3; F_FULLFSYNC is “currently implemented on HFS, MS-DOS (FAT), Universal Disk Format (UDF) and APFS file systems”, and the page warns that “certain FireWire drives have also been known to ignore the request to flush their buffered data.” measured here Timings from 1,000 appends each (200 for F_FULLFSYNC) on APFS on the internal SSD of an Apple M4 Pro. Reproduce; do not quote — these move enormously with the storage device.
CORRECTION · THOSE FOUR NUMBERS ARE PER 256-BYTE APPEND, AND THE COST IS NOT A CONSTANT. The table reads like a price list, and people do arithmetic with it: “fsync is 23 µs, so a hundred records a second costs 2.3 ms”. That arithmetic is wrong, because fsync flushes whatever is dirty, and what is dirty depends on how much you wrote since the last one.
dirty bytes fsync µs (mean) run 2
256 30.2 28.2
4,096 29.2 27.3
65,536 38.1 38.9
1,048,576 144.4 144.3
4,194,304 443.0 698.6
16,777,216 808.6 804.4Why the first row is not the 23.0 µs printed in the ladder above. Both time the same 256-byte append on the same file and device; the ladder averages 1,000 of them and this sweep 5–40, and repeated runs of that one point spanned roughly 23–40 µs on this machine as background load varied. Only the shape transfers — tens of microseconds while the dirty set is small — which is why both tables say to reproduce rather than quote.
Flat to about 64 KiB, then it climbs — roughly 27× from a small append to 16 MiB of dirty data. Batching a hundred records into one fsync is still far cheaper than a hundred fsyncs, but it is not free, and the per-record figure you can quote depends on your batch size. Measure the sync at the batch size you actually use. measured here Two runs, 5–40 repetitions per point. version-sensitive APFS on one internal SSD.
Read the barrier row again
F_BARRIERFSYNC is slower than fsync and much faster than F_FULLFSYNC, and it buys ordering rather than durability. It is “typically useful to guarantee valid state on disk when ordering is a concern but durability is not” — which is exactly the guarantee a crash-consistent format needs and a factor of twenty-four cheaper than the full flush.
documented man 2 fcntl
Atomic publication is a different axis
Durability asks “does it survive a power cut?”. Atomicity asks “can a reader see half of it?”. They are independent, they have different mechanisms, and the second is usually the one actually causing your bug. Exercise 06 is entirely about the second.
rename(2) is the atomic primitive
Replacing a directory entry happens in one step within a filesystem, so an open resolves the name either entirely before or entirely after. Write to a sibling temporary, fsync it, then rename over the target. The temporary must be in the same directory; a cross-device rename fails with EXDEV.
What the fsync in that recipe is for
Not to make the rename atomic — it already is. It orders the contents against the name, so a crash cannot publish a perfectly-named file whose bytes never left the page cache. Being able to say that sentence is the difference between reciting the recipe and understanding it.
Choosing a level, out loud
Ask one question: what happens if this write is lost in a power cut?
Nothing much — window positions, caches, recently-opened lists. Plain write, published atomically with a rename if readers exist. Four milliseconds per record for F_FULLFSYNC is indefensible here.
The user notices but can redo it — a draft, an import that can be re-run. fsync, and accept the documented reordering risk.
The file must remain parseable — a log or database with a commit record that must never appear before the data it commits. F_BARRIERFSYNC for the ordering, at 140× rather than 3,340×.
Acknowledged-and-lost is unacceptable — a financial record, anything you told the user was saved and cannot recreate. F_FULLFSYNC, batched so you pay it once per group rather than once per record. inference — the levels and their costs are documented and measured; which one your data deserves is a judgement, and an interviewer is testing whether you make it explicitly.
5 · Async and backpressure
Not blocking is half the problem. The other half is not drowning.
Moving I/O off the main thread stops the UI freezing. It does nothing about a producer that is faster than its consumer, and that second problem shows up as memory rather than as latency — which is why it is usually found late.
The three shapes on macOS
Blocking calls on a thread you own — simplest, correct, and costs a thread per concurrent operation. Event-driven — kqueue, DispatchSource, run-loop sources: one thread watches many descriptors. Managed — DispatchIO, which “manages operations on a file descriptor using either stream-based or random-access semantics” and handles the buffering and the queueing for you.
https://developer.apple.com/documentation/dispatch/dispatchio
Async does not mean unbounded
“Read the whole file and hand me the bytes” has a peak footprint equal to the file. On a file whose size you do not control, that is a user-supplied input to your memory usage. Streaming with a fixed buffer bounds it; readToEnd() does not.
Backpressure is the missing word
If the producer cannot be slowed, the queue between producer and consumer grows without limit. The fix is always one of three: block the producer, drop (and say which — oldest or newest), or fail loudly at a bound. Choosing none of them is choosing “grow until termination”.
A pipe has backpressure built in
When a pipe’s buffer fills, the writer blocks. That is not a limitation, it is the flow control — and it is why exercise 07’s original design, which stopped draining, wedged its own sender. Anything you build on top of a queue you control needs an equivalent, deliberately chosen.
Bounded streaming · Swift
let handle = try FileHandle(forReadingFrom: url)
defer { try? handle.close() }
var digest = 0
var done = false
while !done {
try autoreleasepool { // <-- load-bearing
guard let chunk = try handle.read(upToCount: 64 * 1024),
!chunk.isEmpty else { done = true; return }
digest = chunk.reduce(digest) { ($0 &* 31) &+ Int($1) }
}
}
// Measured peak: 2.3 MiB at 64, 256 and 512 MiB inputs.
// WITHOUT the pool, the same loop peaks at the FILE size.The autoreleasepool is not defensive tidying. Remove it and this loop is indistinguishable from Data(contentsOf:) — see the correction below.
The same discipline · C
/* Read exactly n bytes, or say why not. */
static int read_fully(int fd, void *p, size_t n) {
unsigned char *b = p; size_t off = 0;
while (off < n) {
ssize_t r = read(fd, b + off, n - off);
if (r == 0) return off == 0 ? 0 : -1; /* EOF vs truncated */
if (r < 0) { if (errno == EINTR) continue; return -1; }
off += (size_t)r;
}
return 1;
}Note the EINTR branch and the distinction between a clean end of stream and a truncated record. Both are routine, and both are commonly missing from code that “works”.
CORRECTION · A FIXED BUFFER DOES NOT BOUND PEAK MEMORY IN THAT LOOP. An earlier version of this page shipped the code above without the autoreleasepool and commented it “peak memory: 64 KiB, whatever the file size”. That comment was wrong, and a reader who copied it shipped the bug it claims to fix. Each strategy below was run in its own process, so the peak phys_footprint is attributable to that strategy alone.
| Strategy | 64 MiB file | 256 MiB | 512 MiB | peak ÷ file |
|---|---|---|---|---|
| the loop above, without the pool | 66.1 MiB | 259.1 MiB | 516.3 MiB | 1.01–1.03 |
Data(contentsOf:) — whole file, by design | 66.1 MiB | 258.2 MiB | 514.3 MiB | 1.00–1.03 |
| the loop above, with the pool | 2.3 MiB | 2.3 MiB | 2.3 MiB | 0.005–0.035 |
read(2) into one reused buffer | 2 MiB | 2 MiB | 2 MiB | 0.001 |
DispatchIO + setLimit(highWater:) | 3 MiB | 3 MiB | 3 MiB | 0.001 |
Data(contentsOf:, .mappedIfSafe) | 2 MiB | — | — | 0.001 |
measured here Top three rows reproduced in this session against the exact code printed above; the remaining rows from the same harness. version-sensitive This is Foundation behaviour and could change — the ratio is the evidence, not the megabytes.
Why, and why the page contained both the mistake and its own refutation
A bounded buffer bounds what your code references. It does not bound what the framework hands you and the pool retains. FileHandle is an Objective-C class, so read(upToCount:) returns an autoreleased Data. A tight while loop crosses no pool boundary, so every chunk accumulates in the thread’s top-level pool until the loop exits. The only difference between the 516 MiB row and the 2.3 MiB row is a scope.
Apple documents this exact trap — in the chapter above this one. “Swift can produce autoreleased objects when it calls into frameworks that use or expose Objective-C APIs … Threads usually have a top-level autorelease pool, but it’s not cleaned very often. This can matter a lot when code fills up the pool with objects, which easily happens in loops.” documented WWDC24 10173 · Analyze heap memory, 10:50–11:19. https://developer.apple.com/videos/play/wwdc2024/10173/
See heap · 5 · Fragmentation & churn, which names the autorelease-pool loop as the classic transient spike. The lesson is bigger than the bug: a memory fact learned in one chapter reaches you as an I/O crash in another, and the two look nothing alike from the symptom end.
Speed, honestly. A separate report measured the pooled version about twice as fast. Reproduced here against this loop it was 231 ms against 236 ms at 64 MiB — barely distinguishable, because this loop’s byte-wise reduce dominates everything else. The memory result is the finding; the speed result depends on what the loop body does, and is not a reason to make the change. measured here
Which API lets you state the bound, and who owns the descriptor
Four of the six rows above bound peak memory. Only one of them lets you say what the bound is: DispatchIO.setLimit(highWater:) makes backpressure a named parameter instead of an emergent property of how you happened to write a loop. The three choices this section names — block, drop, or fail at a bound — need an API in which one of them can be expressed.
And with that comes a lifetime rule people get wrong once. DispatchIO takes a cleanup handler that runs when the channel is finished; that is where the descriptor is closed. The channel owns the descriptor once you hand it over — closing it yourself underneath the channel is a use-after-close, and the symptom is an unrelated read failing later in the process, because the number was reused. Cancellation follows the same ownership: cancel the channel and let its cleanup handler close, rather than closing to force a cancel.
documented https://developer.apple.com/documentation/dispatch/dispatchio https://developer.apple.com/documentation/dispatch/dispatchio/setlimit(highwater:) inference The descriptor-reuse symptom is reasoning from the descriptor model in section 1A, not something observed here.
Network I/O is the same model with worse numbers
Everything above holds, with three differences that change the design rather than the mechanism. Latency is thousands of times larger, so the per-call cost that dominates local I/O is irrelevant and round trips are everything — batch, pipeline, and do not chat. Failure is normal rather than exceptional, so timeouts, retries with backoff, and cancellation are part of the API, not error handling bolted on. And a TCP stream is a byte stream, exactly like a Unix socket, so the framing bug in exercise 07 is the same bug with a longer reproduction time.
Use the framework that already handles this — URLSession for HTTP, Network.framework for anything lower — rather than a socket loop. The reason is not convenience; it is that connection migration, proxy handling, TLS and interface changes are a large amount of correctness you would otherwise own. inference No network measurement appears in this chapter; every number here is local storage or in-memory.
5A · Running out, and being interrupted
Two failures that arrive through a path you did not write.
Both of these are usually learned the hard way, in production, from a symptom that does not obviously point at I/O. Both take one paragraph to explain and one line to defend against.
Descriptor exhaustion, and the error path that also fails
RLIMIT_NOFILE soft = 1,048,576 hard = unlimited
lowered soft limit to 64 for this process
open failed after 61 extra descriptors: errno=24 (Too many open files)
fopen while exhausted: FAILED (errno=24 Too many open files)The first line kills a popular answer. The default soft limit here is 1,048,576. Descriptor exhaustion on a modern Mac is a leak, not a limit, so “raise ulimit -n” is the wrong first move — find the unclosed descriptor instead, with lsof -p <pid> and a look at whether the owning type closes on every path including the error ones.
The last line is the one people miss. The failure handler needed a descriptor too, and did not get one. If your error path opens a log, writes a crash report, or connects to a reporting service, it fails exactly when you need it. Open that descriptor at startup and hold it.
EMFILE is per-process. ENFILE is system-wide (kern.maxfiles is 368,640 here) and means somebody else is the problem — a distinction worth making out loud, because the two have completely different owners. measured here documented man 2 open
EINTR is a property of a signal you may not have installed
without SA_RESTART: read returned -1 errno=4 (Interrupted system call)
with SA_RESTART: read returned 1 errno=0Identical call, identical signal, opposite outcome — decided entirely by how the handler was installed. So EINTR is not a property of your read; it is a property of somebody’s sigaction. Any library linked into your process can install a handler without SA_RESTART and change the failure modes of your unrelated I/O.
Therefore: handle EINTR in every blocking loop regardless, because you do not control who installs handlers. This is the same argument as preferring F_SETNOSIGPIPE on your descriptor over signal(SIGPIPE, SIG_IGN) for the whole process — scope the change to the thing you own. measured here
O_NONBLOCK IS NOT A WAY TO MAKE A SLOW CONSUMER GO AWAY. It converts wait into EAGAIN, and continue-ing past EAGAIN is silent data loss. Measured on a logging component that ships records over a pipe to a slow collector: 94–96% of records lost at 512-byte records, 99.6% at 8 KiB — with the app reporting success throughout, because every write that failed returned -1 and the loop moved on. The repaired version, which waits with a deadline and counts what it drops, loses 0.0000 at both sizes.
And the record size hides a second defect. At 512 bytes every successful write was complete, because PIPE_BUF is 512 on macOS and writes at or below it are atomic (sys/syslimits.h: “max bytes for atomic pipe writes”). At 8 KiB the repaired build recorded 14 genuine short writes. A test suite that only ever writes small records passes while the partial-write bug sits there untriggered. measured here documented
Every non-blocking write needs all three branches
w == n done; 0 < w < n loop from w; w < 0 && errno == EAGAIN wait, then retry the same bytes. A fourth branch that treats EAGAIN as an error and drops the record is the bug above. Whichever of block / drop / fail-at-a-bound you choose, choose it explicitly and count it.
A wait without a deadline is a hang
The repair for EAGAIN is to wait — but an unbounded wait has converted silent loss into a freeze, which is a different bug and not obviously a better one. Every wait in the repaired fixture carries a deadline and reports what it dropped when the deadline passes.
Inherited descriptors keep pipes open
Found while building that fixture: the watchdog child inherited the pipe’s write end, so the collector never saw end-of-stream and everything blocked. A pipe closes when the last write descriptor does, and fork duplicates them all. Section 1A is this bug in the abstract. measured here
Cancellation is ownership
Handing a descriptor to DispatchIO transfers it; cancel the channel and let its cleanup handler close. Closing the descriptor to force a cancel is a use-after-close whose symptom appears somewhere else entirely, once the number is reused.
6 · Diagnosis
Count the calls before you blame the disk.
“The SSD is slow” is almost never the answer on modern hardware. The measurement that settles it takes one minute.
| Symptom | First measurement | What each answer means |
|---|---|---|
| File operation is far slower than the data volume suggests | Bytes per system call | Near 1 → unbuffered, fix the loop. Near 64 KiB → the cost is elsewhere |
| Main thread freezes on open/save | sample <pid> during the freeze | A blocking frame on the main thread → move it off; see the threads chapter |
| Saving is slow and the data is small | Count the F_FULLFSYNC calls | One per record → batch them; 4 ms each, measured here |
| Memory spikes while importing | Peak footprint vs file size | Tracking the file → you read it whole; stream instead |
| A file is occasionally corrupt on read | Is it written in place? | Yes → publication is not atomic; write a temporary and rename |
| Reads are genuinely hitting the device | Compare a warm second run | Second run much faster → it was cold-cache, not code |
Unprivileged, and enough
# is the main thread blocked in a read?
sample <pid> 3 -file /tmp/io.txt
# count your own calls — the most useful I/O metric there is
# (instrument the loop; the fixtures in the bundle do exactly this)
# what is this process holding open?
lsof -p <pid> | head -40
# how big is the thing you are about to read whole?
stat -f '%z bytes' somefilePrivileged, and often unnecessary
# every filesystem call, system-wide
sudo fs_usage -w -f filesys -p <pid>
# drop the page cache for a genuine cold measurement
sudo purgefs_usage does require root — verified: it answers ’fs_usage’ must be run as root... and exits. But you no longer need it for the call count, which is the thing it was wanted for; see the box below. purge still needs elevation, which is why section 2A can bypass the cache but cannot evict it.
Instrument your own call count
The single most valuable I/O metric is one your code can report: bytes moved divided by calls made. It needs no tool, no elevation and no trace, it is comparable across machines, and it converts “the disk is slow” into a number that settles the argument.
File Activity and System Trace
Instruments’ File Activity template shows filesystem operations against time, and System Trace shows the thread blocking around them. Useful for seeing the shape; the call count above usually answers the question first.
documented https://developer.apple.com/documentation/xcode/improving-app-responsiveness
Measure warm and cold separately
A second run of the same read is a warm-cache measurement, and it is the one that isolates your code. Reporting a single number without saying which state it was in makes the result unreproducible — and most published I/O figures do exactly that.
Test the failure paths
Full disk, file removed underneath you, permission revoked mid-write, a network volume that disappears. Each has a distinct errno, each needs a distinct message, and none of them is exercised by a happy-path test. Exercise 08 is the inter-process version of the same gap.
Per-syscall counts and durations, unprivileged, for a process you did not write
“Bytes per system call” is the metric this section calls the most valuable one, and until now it needed either your own instrumentation or sudo fs_usage. A headless xctrace recording gives it for any process:
xcrun xctrace record --template 'System Trace' \
--output run.trace --launch -- ./yourprog
xcrun xctrace export --input run.trace \
--xpath '/trace-toc/run[@number="1"]/data/table[@schema="syscall"]' > syscalls.xml
# aggregate the XML however you like; validated against a program
# that performs exactly 64 writes, 64 full reads + 1 EOF read, and 1 fsync:
# syscall count total ms us/call
# read 65 0.196 3.0
# write 64 0.482 7.5
# fsync 1 0.482 481.9The counts are the point, and so is how they were trusted. The extractor was checked against a program whose syscall behaviour was known in advance and matched exactly on all three — which is the only reason to believe it on a program whose behaviour is not known. Validate a measurement tool against a known answer before you use it on an unknown one. measured here
One failure worth knowing in advance. --template 'File Activity' against a target that exits in ~65 ms produced a trace that xctrace export then refused with Export failed: Document Missing Template Error. System Trace on the same target exported cleanly. Use System Trace for this workflow, or give the target enough work to outlive the recorder’s start-up. negative result Not root-caused. version-sensitive
Section scheduling · 11A covers the same tool from the scheduling side, including the sample versus spindump privilege boundary.
What this chapter did not do. No Instruments window was opened and no screenshot appears anywhere; File Activity’s and System Trace’s presentation is named from documentation, while the syscall counts above are a real headless recording. purge was never run because it needs elevation, so section 2A’s cold figures are cache-bypassed rather than cold-boot — F_NOCACHE prevents admission, it does not evict what is already resident. No power-failure or crash-consistency test was performed, so every durability claim is the documented guarantee plus its measured cost, never an observed survival. No network I/O was measured at all, so the network discussion is reasoning. Every timing here is APFS on one internal SSD, and the durability figures in particular move enormously with the storage device.
7 · Fixing exercises
Two broken programs. Diagnose, repair, prove.
Exercises 05 and 06 of the same bundle. Both produce entirely correct answers; one is 440× slower than it needs to be, and the other is correct exactly 80% of the time from a reader’s point of view.
Same bundle as the heap and scheduling chapters
os-memory-io-ipc-exercises.tar.gz — 44 files, 55,129 bytes. SHA-256 0a04e5544c047fc5376919d91fdcf5943a1c1316aa38996d93bac105b23ba13f · raw path labs/os-memory-io-ipc-exercises.tar.gz
./run-all.sh 05 0605 · The scanner that blamed the SSD
A crash-log scanner walks a log file line by line and tallies the lines containing a marker. The author needed to split on newlines, could not find a “read a line” system call, and wrote one: read a byte, test it, repeat. The field reports that scanning a 9 MB log takes three seconds, the disk is idle throughout, and the process is not even at 100% of one core — “is the SSD broken?”. The answer the program produces is correct. Find the cost and remove it without changing the answer.
clang -O2 -g -Wall -Wextra broken/logscan.c -o /tmp/logscan_broken
/tmp/logscan_brokenExpected signal. readSyscalls=9437185 for a 9,437,184-byte file — bytesPerSyscall=1.0 — taking 2,945.6 ms at 3.1 MB/s. The fixed build makes 145 calls, takes 6.7 ms at 1,348.3 MB/s, and produces byte-identical line, marker and checksum counts. measured here
Success criterion. At least 32 KiB per read on average, identical lines/markers/checksum, at least 20× faster, and you can name the buffer size you chose and why not ten times larger.
Progressive hints
- Count what the loop does per byte. One
readsystem call: a mode switch into the kernel, a descriptor lookup, a one-byte copy, a mode switch back. The newline test is one comparison. Which dominates? - The kernel is not the only place you can hold bytes. The page cache already has the file. The problem is not where the data is, it is how often you ask for it.
- Do not stop at the first working number. Measure a few buffer sizes. The curve flattens; find roughly where. “64 KiB, because I measured 4 KiB and 1 MiB and they were within 10% of each other” is worth far more than “64 KiB because that is what people use”.
Solution
Read in gulps and split lines in user space:
enum { BUFFER = 64 * 1024 };
char *buf = malloc(BUFFER);
for (;;) {
ssize_t n = read(fd, buf, BUFFER);
if (n <= 0) break;
for (ssize_t i = 0; i < n; i++) { /* the same line splitting */ }
}What was actually being paid. A read that hits the page cache performs no device I/O at all; its cost is the system call, and that is nearly independent of the byte count. This is not caching, not prefetching, and it does not make the storage faster — the disk did precisely the same work in both runs.
Why not stdio. fgets/getline buffer for you and are the right answer in production. The hand-written loop here makes the mechanism visible and the syscall count assertable.
Why not mmap. It would be faster still — 2.3 ms against 3.2 ms for the same 64 MiB in the section-3 measurement — and it changes how an I/O error arrives, from a return value to a SIGBUS. For a single sequential pass over a log file that is a bad trade; for a large asset read repeatedly it is a good one. inference
06 · The state file that is corrupt one launch in fifty
An app persists its window state to a small file whenever it changes; a helper process reads it to restore the layout. The file carries a length and a checksum, so a reader can tell whether it is intact. The field reports that about one launch in fifty the helper calls it corrupt and falls back to defaults — and that the file is perfectly valid by the time anyone looks. There is no thread race here. Find the window, close it, and tell me what the fsync in your fix is actually for.
clang -O2 -g -Wall -Wextra broken/statefile.c -o /tmp/sf_broken
/tmp/sf_brokenExpected signal. tornReads=2393 of 12,000 attempts — tornFraction=0.1994 — with missingFile=0, so the name always resolved and what it resolved to was simply wrong. The fixed build reports tornReads=0. Not “rare” — zero, on every run. measured here
Success criterion. tornReads = 0 on two consecutive runs, the temporary in the same directory as the target and you can say why, an explanation of what the fsync is for given that rename is already atomic, and a statement of what your fix does not guarantee.
Progressive hints
- You cannot make several writes atomic. There is no call that publishes 32 KiB of new content in one step to a file a reader already has a path to. Stop looking for one. Something else in the filesystem interface is atomic.
- Do not modify the thing being read. If the reader resolves a name, and the name is made to point at a different, already-complete file in one step, the reader gets all of one or all of the other.
- Same filesystem. Whatever call you reached for is atomic only within one filesystem and fails with
EXDEVacross devices.$TMPDIRis not necessarily the same filesystem as the target.
Solution
char tmp[512];
snprintf(tmp, sizeof tmp, "%s.tmp", path); /* SAME directory */
int fd = open(tmp, O_CREAT | O_WRONLY | O_TRUNC, 0600);
write(fd, &h, sizeof h);
for (...) write(fd, body + off, CHUNK);
fsync(fd); /* contents before name */
close(fd);
rename(tmp, path); /* publish, atomically */Why it works. rename(2) replaces a directory entry in a single step, so a reader’s open resolves the name either entirely before or entirely after. There is no interval to be unlucky in — the guarantee is structural, not probabilistic, which is why the fixed build measures exactly zero rather than merely fewer.
What the fsync is for. Not atomicity — rename already has that. It orders the contents against the name: without it a crash between the writes and the rename can publish a perfectly-named file whose bytes are still only in the page cache.
What this does not give you. Durability across a power cut. For that you need F_FULLFSYNC before the rename, at about 4 ms per call measured here, and you should reach for it only where losing the write is bad enough to justify that. A window-position file is not.
The repairs that do not work. A single bigger write: O_TRUNC has already emptied the file before it runs, and a large write is not guaranteed atomic against a concurrent reader — this narrows the window without closing it, which is the worst outcome, a bug that now reproduces only in the field. A lock file: correct, and it requires every reader to honour a protocol, including readers you did not write. A “valid” flag written last: you have reinvented journalling, worse. inference
Honest note. Delete the fsync from the fix and the check still passes, because no fixture can simulate a power failure. Being able to say “my test does not cover the thing this line is for” is exactly the honesty an interview is probing.
8 · Interview questions
Fifteen questions, with the follow-up that comes next.
Answer out loud before opening each one.
ExplainWhat does write() returning successfully actually guarantee?
That the bytes are in the kernel’s page cache and visible to any other process that reads the file. It survives your process crashing. It does not survive the machine losing power.
And the next level up is weaker than people assume. fsync(2)’s own manual page says that while it flushes from the host to the drive, “the drive itself may not physically write the data to the platters for quite some time and it may be written in an out-of-order sequence”, and that this “is not a theoretical edge case”.
The strong guarantee on macOS is F_FULLFSYNC, which asks the drive to flush its own buffers and acts as a barrier.
Follow-up: “What does that cost?” — measured here, per 256-byte append: 1.2 µs with no sync, 23 µs with fsync, 169 µs with F_BARRIERFSYNC, and 4,013 µs with F_FULLFSYNC. A factor of 3,340 between the ends, which is why the choice has to be deliberate and usually batched.
DiagnoseReading a 9 MB file takes three seconds and the disk is idle. What is wrong?
Almost certainly the number of system calls, not the storage. Measure bytes per read. If it is near 1, the loop is unbuffered and every byte costs a full user-to-kernel transition to copy a character the kernel already had in the page cache.
measured here 9,437,185 calls at 3.1 MB/s against 145 calls at 1,348 MB/s for the identical answer — a factor of 440 with the same file, same filesystem and same cache state.
The disk being idle is the clue that confirms it: no device work is happening, so the time is being spent crossing a boundary.
Follow-up: “How big should the buffer be?” — measured, the curve flattens between 4 KiB and 64 KiB and 1 MiB is very slightly worse than 256 KiB. Pick around 64 KiB and be able to say you measured, because “bigger is better” is false past the knee and costs cache locality.
ExplainWhen would you use mmap instead of read?
For random access, for large read-only assets, and for data read repeatedly — where avoiding the copy and letting the fault handler fetch only what is touched genuinely wins. Measured here, touching one byte per page of a 64 MiB file took 2.3 ms against 3.2 ms for a 256 KiB-buffered read of the whole thing.
There is a memory argument too: mapped file pages are clean, so they are discardable under pressure and are not charged to your footprint, whereas a buffer you read into is dirty and is.
The reason it is not the default is how errors arrive. An I/O failure or a file truncated underneath the mapping becomes a SIGBUS at whatever instruction touched the page — inside a library, inside a parser — rather than a return value you can branch on.
Follow-up: “So how do you use it safely?” — Data(contentsOf:options:.mappedIfSafe), where Foundation declines to map when it judges the backing store unsuitable. Hand-rolled mmap is for anonymous memory and for assets you ship and control.
DiagnoseA config file is occasionally read as corrupt, and is always valid afterwards. Why?
Because it is being rewritten in place. open(O_TRUNC) empties the file immediately, and the new contents arrive over several write calls. Between those two events the file is neither the old state nor the new one, and any reader that opens the name in that interval sees the in-between.
Measured here with a reader hammering the file: 19.9% of reads caught it mid-rewrite. A production report of “one launch in fifty” and a fixture measuring one in five are the same bug at different read rates.
A checksum does not close the window; it only lets the reader notice.
Follow-up: “What is the fix, exactly?” — write a temporary in the same directory, fsync it, rename it over the target. The rename replaces a directory entry in one step, so an open resolves before or after and never during. Measured torn reads after the fix: zero, structurally rather than statistically.
ExplainIf rename is atomic, why does the recipe include an fsync?
They solve different problems. rename gives atomic visibility: no reader ever sees a partial file. The fsync gives ordering: it forces the contents out before the name that promises them.
Without it, a crash between the writes and the rename can leave the new name published while the new bytes are still only in the page cache — a perfectly-named file full of nothing, which is arguably worse than the torn read you were fixing, because now it looks valid.
Follow-up: “Is fsync enough there?” — for ordering against a crash, generally yes. For surviving a power cut you need F_FULLFSYNC. And if all you need is ordering, F_BARRIERFSYNC exists precisely for “ordering is a concern but durability is not” and measured 24× cheaper than the full flush.
ExplainWhat is backpressure and where does it come from for free?
Backpressure is what stops a fast producer from drowning a slow consumer. Without it the queue between them grows without bound, and the symptom is memory rather than latency — which is why it is usually found late, as an out-of-memory termination rather than as a slow path.
A pipe has it built in: when the buffer fills, the writer blocks. So does a bounded queue, a semaphore that limits in-flight work, and a stream API that only reads when you ask.
Where you build the queue yourself, you must choose one of three: block the producer, drop (and say which — oldest or newest), or fail loudly at a bound. Choosing none of them is choosing “grow until termination”.
Follow-up: “Give an example of accidentally removing it.” — reading a whole file into memory instead of streaming it. Peak footprint becomes the file size, which is an input the user chose and you did not.
ChooseYou are saving a document the user just edited. What do you call?
Write to a temporary in the same directory, fsync it, rename over the original. That gives atomic publication, so a crash or a concurrent reader never sees a half-written document, and it gives you the old file intact until the moment of replacement.
Whether to add F_FULLFSYNC before the rename depends on one question: if the machine lost power right now and this save were lost, what happens? For a document the user is actively editing and could redo a minute of, fsync is defensible. For something you have told the user is safely stored and cannot be recreated, pay the four milliseconds.
Follow-up: “What about autosave every few seconds?” — then batch. Paying 4 ms per record is very different from paying it per save point, and a design that calls F_FULLFSYNC in a loop is usually a design that has not chosen a durability level at all.
DiagnoseImporting a large file makes the app get terminated. It is all on a background thread.
Being on a background thread solves responsiveness, not footprint. If the import reads the whole file into memory, peak memory tracks the file size — and the file size is chosen by the user.
The measurement is one line: compare peak footprint against input size. If they track, the import is not streaming.
The repair is a fixed-size buffer and incremental processing, so peak memory is the buffer whatever the input. And if the pipeline downstream is slower than the read, it needs explicit backpressure or the queue becomes the new unbounded thing.
Follow-up: “Does mmap fix it?” — partly and misleadingly. Mapped pages are clean and reclaimable, so the footprint charge is much lower, but if your processing touches everything and holds derived objects you have moved the problem rather than solved it.
ExplainWhy is a short read not an error?
Because the call is documented to return “the number of bytes actually read”, which may be fewer than requested. On a regular file at end-of-data that means the end. On a pipe, socket or terminal it means “this is what is available now”, and more may follow.
So every correct reader of a stream is a loop that tracks how many bytes it still needs, and it must distinguish a return of 0 at a record boundary (a clean end of stream) from a return of 0 in the middle of a record (a truncated record, which is an error). It also has to handle EINTR.
The same is true of write, which is why every writer is a loop too.
Follow-up: “When does this bite hardest?” — with variable-size messages over a stream. Measured here on a Unix socket, a receiver that assumed one read per message recovered only 16.4% of 2,000 messages, and every byte had arrived correctly. That is the IPC chapter’s exercise 07.
DesignYou own a framework that reads user-supplied files. What do you promise?
Four things, and all four are contract.
Bounded memory. Peak footprint is a function of your buffer, not of the input. If a caller can hand you a 40 GB file, say what happens.
Which thread you block. If your read API is synchronous, say so loudly, because a caller who invokes it from the main thread has a hang and will report it as yours.
Distinct errors. Not found, no permission, disk full, volume disappeared, and “the file changed underneath me” need different messages, because the caller’s recovery differs for each.
Your durability level. If you write, say whether a returned success means visible, flushed, or committed to the device. This is the promise most frameworks leave implicit and most callers assume is the strongest one.
Follow-up: “How would you test the failure paths?” — deliberately: a file removed mid-read, a full volume, a revoked permission. None of those is exercised by a happy-path test, and all of them happen to users.
DiagnoseAn import reads in fixed 256 KiB chunks and still gets terminated. Peak memory tracks the file size. It is off the main thread and never reads the whole file.
A bounded buffer bounds what your code references. It does not bound what the framework returns and the autorelease pool retains. FileHandle.read(upToCount:) returns an autoreleased Data, and a tight while loop crosses no pool boundary, so every chunk accumulates in the thread’s top-level pool until the loop exits.
Measured, one strategy per process: the chunked loop peaked at 1.01–1.03× the file at 64, 256 and 512 MiB — indistinguishable from reading the whole file. Adding autoreleasepool around the loop body pinned it at 2.3 MiB regardless of input size.
Apple documents the general case in WWDC24 10173 at 10:50–11:19: Swift produces autoreleased objects when calling into Objective-C APIs, the thread’s top-level pool “isn’t cleaned very often”, and this “easily happens in loops”.
Follow-up: “How would you have caught it in review?” — any Foundation API returning an object inside a loop with no pool boundary. And measure peak footprint per strategy in its own process; measuring several in one process hides which one grew.
ExplainTwo threads read from one file descriptor. What can go wrong, and what is the fix?
The seek offset lives on the open file description, not on the descriptor, so lseek + read is two operations with a shared mutable cursor between them. Thread A seeks, thread B seeks, thread A reads from B’s offset.
Measured, eight threads × 20,000 reads on one descriptor with self-identifying records: 110,448 wrong records out of 160,000, plus a handful of short reads, and no error reported at any point. pread/pwrite take an explicit offset and never touch the shared one: zero wrong, zero short.
Follow-up: “Does dup help?” — no. dup shares the same open file description, and so does fork. Only a separate open gets a separate offset, and pread is cheaper than that. Note also that O_NONBLOCK lives on the description, so setting it affects every descriptor pointing at it — while FD_CLOEXEC lives on the descriptor and is not copied by dup.
ExplainIs random access slower than sequential on an SSD?
It depends on two things the question leaves out: block size and cache state. Measured on 256 MiB — warm, random costs about 1.47× at 4 KiB and is indistinguishable from 16 KiB upward. With the cache bypassed on both writer and reader, random costs about 3.6× at 4 KiB and the penalty is gone by a few hundred kilobytes.
So “random is slow” is a statement about small blocks with a cold cache, and nothing else. On spinning media it would be straightforwardly true; on this machine it is a corner.
Follow-up: “So what would you change first?” — block size. Going from 4 KiB to 256 KiB removed the entire penalty here, which is one constant and no new state, whereas a cache is a whole coherence problem. Design the cache when the block size is already right.
ExplainWhat does “buffered I/O” mean? Be specific about which buffer.
At least four independent layers: your call pattern, the library buffer (stdio, the FileHandle chunk, DispatchIO’s limit), the kernel page cache, and the device. They fail independently and the repairs are different.
Measured, holding user code at fgetc and changing only setvbuf: 5,852 ms unbuffered down to 193 ms at 256 KiB — a 30× swing with identical kernel behaviour per byte requested. The page-cache layer separately measured 6–41× depending on block size. Two of the four layers never enter the kernel at all.
Follow-up: “Your stdio buffer is already 64 KiB and it is still slow.” — then you are paying the user-call layer. Even a perfectly buffered fgetc capped at 83 MB/s here, against ~19,000 MB/s for 256 KiB block reads: 230×. Stop asking for one byte at a time.
Diagnoseopen starts failing with EMFILE, and the code that logs the failure fails too.
Per-process descriptor exhaustion. The default soft limit measured here is 1,048,576, so this is a leak rather than a limit — raising ulimit -n is the wrong first move. Find the unclosed descriptor with lsof -p <pid> and check that the owning type closes on every path, including the error ones.
The second symptom is the one that matters: the error path needed a descriptor and did not get one. Measured, fopen failed with the same EMFILE while exhausted. If your failure handler opens a log or writes a crash report, it fails exactly when you need it. Reserve one at startup and hold it.
ENFILE rather than EMFILE means the system ran out and the leak may not be yours — a different owner and a different conversation.
Follow-up: “And why must a read loop handle EINTR?” — because it is a property of how somebody’s signal handler was installed, not of your call. Measured: the same read and the same signal returns -1/EINTR without SA_RESTART and completes normally with it. Any library linked into your process can install one.
Drill
Answer first, then read the explanation.
One defensible first move each.
Scenario 01 · The innocent SSD
A 9 MB scan takes three seconds with the disk idle and one core half busy. First measurement?
Scenario 02 · The occasionally corrupt file
A settings file is sometimes read as invalid and is always fine afterwards. What closes the window?
Scenario 03 · Acknowledged and lost
You told the user a record was saved; a power cut lost it. Which call would have prevented that on macOS?
Scenario 04 · The import that gets terminated
Importing a user-chosen file kills the app on large inputs. It already runs off the main thread. First change?
Record your answer · 1
Pick a durability level, and defend it.
Choose one thing your code writes. Write: what it is → what happens if a power cut loses it → the level you would call (write, fsync, F_BARRIERFSYNC, F_FULLFSYNC) → what that costs per record → whether you would batch it.
Record your answer · 2
Rehearse one I/O diagnosis.
For a slow or wrong file operation you have actually seen, write the chain: symptom → syscall cost or device cost → the measurement that decided it → the smallest fix → the peak memory and durability your fix implies.
Primary sources for this chapter
Darwin manual pages and Apple documentation
The durability section rests almost entirely on two manual pages that ship with macOS, and they are worth reading in full rather than quoting. Raw URLs and man invocations are printed beside each title.
| Source | URL or command (copyable) | Used in this chapter for |
|---|---|---|
macOS 26.3 manual page fsync(2) | man 2 fsync | What fsync does and explicitly does not promise; the “not a theoretical edge case” warning; the pointer to F_FULLFSYNC |
macOS 26.3 manual page fcntl(2) | man 2 fcntl | F_FULLFSYNC and F_BARRIERFSYNC semantics and filesystem support; F_NOCACHE; F_SETNOSIGPIPE |
macOS 26.3 manual pages read(2), write(2), open(2) | man 2 read · man 2 write · man 2 open | Short reads and writes; pread/pwrite; descriptor and offset semantics; O_NONBLOCK |
macOS 26.3 manual pages mmap(2), madvise(2), rename(2) | man 2 mmap · man 2 madvise · man 2 rename | Mapping semantics and SIGBUS; renamex_np with RENAME_SWAP and RENAME_EXCL; EXDEV |
| FileHandle | https://developer.apple.com/documentation/foundation/filehandle | Chunked reading with a bounded buffer |
| FileHandle.readToEnd() | https://developer.apple.com/documentation/foundation/filehandle/readtoend() | The unbounded-peak shape and when it is acceptable |
| Data.ReadingOptions | https://developer.apple.com/documentation/foundation/data/readingoptions | mappedIfSafe as the safe form of mapping a file |
| DispatchIO | https://developer.apple.com/documentation/dispatch/dispatchio | Managed stream and random-access I/O on a descriptor |
| DispatchSource | https://developer.apple.com/documentation/dispatch/dispatchsource | Event-driven readiness without a thread per descriptor |
| FileManager | https://developer.apple.com/documentation/foundation/filemanager | Higher-level replacement and item-moving operations |
| Improving app responsiveness | https://developer.apple.com/documentation/xcode/improving-app-responsiveness | Why synchronous I/O on the main thread is a hang, and the budget it breaks |
macOS 26.3 manual pages kqueue(2), lsof(8), fs_usage(1) | man 2 kqueue · man 8 lsof · man 1 fs_usage | Readiness notification; open-descriptor inspection; the privileged call-trace path that was not used here |
| WWDC24 10173 · Analyze heap memory | https://developer.apple.com/videos/play/wwdc2024/10173/ | Autorelease pools in loops, 10:50–11:19 (timestamp verified against the published transcript) — the documented mechanism behind section 5’s streaming correction |
| DispatchIO.setLimit(highWater:) | https://developer.apple.com/documentation/dispatch/dispatchio/setlimit(highwater:) | The one backpressure bound on this page that is a named parameter rather than an emergent property |
How to read the labels on every claim in this chapter
documented — Apple documentation, an official WWDC or Tech Talk transcript, Swift Evolution, or a Darwin manual page or SDK header on macOS 26.3. Quoted verbatim with the source named beside it.
measured here — observed on one machine: Apple M4 Pro (10 performance + 4 efficiency cores), macOS 26.3 (25D125), 48 GB, 16 KiB pages, Swift 6.2.4, Apple clang 17.0.0, Xcode 26.3. Evidence of a mechanism, never a benchmark. Reproduce; do not quote.
inference — our reasoning joining the two above. Marked so you can disagree with the reasoning without doubting the evidence.
negative result — something that was tried and did not show what was expected. These bound what a chapter is allowed to claim, and they are kept deliberately visible.
correction — a correction to an earlier reading published on this page, naming what was wrong and why. This page’s whole method is labelled provenance, so its own errata carry a label too rather than disappearing into a rewrite.
version-sensitive — observed behaviour of this OS build that must be re-measured on another before it is relied on.
Scope of the evidence, stated plainly. Claim labels mean the same as in the previous chapters. Every measured here figure comes from a small C program on one machine — Apple M4 Pro, macOS 26.3 (25D125), APFS on the internal SSD, 2026-09-23 — and the durability numbers in particular move enormously with the storage device, so they are evidence of a ratio between levels rather than of any absolute cost. What this chapter does not have: no Instruments window and no screenshot, so File Activity’s and System Trace’s presentation is documentation — though section 6’s syscall counts are a real headless recording; purge was never run because it needs elevation, so section 2A’s cold figures are cache-bypassed rather than cold-boot; no power-failure or crash-consistency test was performed, so every durability claim is the documented guarantee plus its measured cost, never an observed survival; FileHandle.bytes and URL.lines were exercised but produced a figure that could not be reconciled, so async byte and line sequence performance is treated as unmeasured and no number for it appears; and no network I/O was measured at all, so the network discussion is explicitly marked as reasoning rather than evidence. On WWDC timestamps: the one session this chapter cites carries a range verified against its published transcript; nothing else is cited, because nothing else was verified.
Chapter · process boundaries
IPC.
Every mechanism for getting data from one process to another on macOS, what each one costs, and — the part that actually decides the design — what each one does about trust, lifecycle and failure. The headline measurement is that latency barely distinguishes them: what distinguishes them is semantics.
Why the work is in another process at all, and what the protocol promises when that process dies.
Show where the bytes go, and that your framing and your failure handling are correct rather than lucky.
MIG stub generation, Mach message descriptor layout, launchd plist schema details, the XPC wire format.
The sentence to have ready
“A process boundary is bought for isolation, and paid for in protocol.” Crash containment, privilege separation and independent lifetime are the reasons to cross one. Serialisation, versioning, timeouts, cancellation, reconnection and a peer that can vanish are the bill.
1 · What a boundary is
Separate address spaces, separate failure, separate authority.
Two processes share no memory by default. Everything that crosses between them is either copied by the kernel or explicitly mapped into both. That single fact produces the cost, the safety and the design constraints of every mechanism in this chapter.
Simplified diagram, not a model of the Mach IPC subsystem. It exists to make one point: the boundary is a design decision with a fixed price, and the price is paid in protocol rather than in microseconds. measured here The round-trip figures inside the picture — pipe 4.80 µs, Unix socket 4.85 µs, Mach port 4.36 µs, shared memory 0.08 µs — are this page’s own, and section 3A has the full table and its scope.
Why cross a boundary at all
Three good reasons and no others. Crash containment — a parser that faults takes the helper down, not your app. Privilege separation — the component that touches the network holds fewer entitlements than the one that touches your documents. Independent lifetime — work that must outlive a window, or be shared by several clients.
Why not to
Because the boundary is not free and it is not just latency. Everything that crosses it must be serialisable, versioned, size-bounded, and validated on arrival. And the peer can disappear at any moment, which turns “call a function” into “issue a request that may never be answered”.
Everything crossing is untrusted
Even from a helper you wrote. A length field read off a wire is attacker-controlled input: bound it before you allocate or read that many bytes. This is the single most common security defect in hand-written IPC, and it is why message-oriented frameworks that do the bounding for you are worth their overhead.
launchd owns the lifecycle
For XPC services Apple is explicit: “The launchd system daemon manages these services, launching them on demand, shutting them down when idle, and restarting them if they crash.” You do not spawn or supervise them, and designing as though you do is a category error.
documented https://developer.apple.com/documentation/xpc
2 · The mechanisms
Seven ways, and what each one actually is.
In rough order from lowest-level to highest.
Byte streamPipes — anonymous and named
What it is. A unidirectional byte stream with a kernel buffer. pipe(2) creates an anonymous pair, inherited across fork, which is how a parent talks to a child it launched. A named pipe (FIFO, mkfifo) has a path, so unrelated processes can open it.
A FIFO is not a cheap Unix socket, which is the comparison people reach for. It gains a filesystem name and filesystem permissions and keeps every pipe limitation: still unidirectional, still no peer credentials, and it cannot pass a descriptor. If you wanted a name, it is enough; if you wanted a name because you wanted identity or capability transfer, it is the wrong mechanism. inference from the mechanism list below.
Capacity, measured: a pipe holds 65,536 bytes before a writer blocks; a socketpair holds 8,192 by default. That eight-to-one ratio is the whole of section 3B’s untuned throughput result, and it is also what decides how large a message can deadlock you (section 5). The socket’s figure is a default rather than a limit — SO_SNDBUF on the sending descriptor moves it, and section 3B measures what that is worth. measured here
What it gives you. Flow control for free: when the buffer fills, the writer blocks. That is real backpressure, built in, and it is why a pipe is a good default for a streaming producer-consumer pair.
What it does not give you. Message boundaries — it is a byte stream, so framing is yours. Bidirectional communication — you need two. Any notion of who is on the other end. And a writer whose reader has gone gets SIGPIPE, whose default disposition terminates the process.
measured here 4.80 µs round trip cross-process; 7,680 MB/s with 64 KiB writes and 8,390 MB/s with 1 MiB writes — the best bulk throughput of any kernel-mediated mechanism tested. Capacity 65,536 bytes, which is why (section 3B).
Use it when: you launched the other process, the data is a stream, and you want backpressure without writing it. Foundation’s Pipe and Process wrap exactly this. https://developer.apple.com/documentation/foundation/pipe
Byte stream or datagramUnix domain sockets
What it is. A socket in the filesystem namespace, or an anonymous pair from socketpair(2). SOCK_STREAM is a bidirectional byte stream; SOCK_DGRAM preserves message boundaries.
What it gives you that a pipe does not. Bidirectionality on one descriptor. A filesystem name, so unrelated processes can connect, with filesystem permissions as a first access check. Peer credentials. And — the genuinely special capability — the ability to pass file descriptors between processes with SCM_RIGHTS, which is how a privileged helper can open something and hand the open descriptor to an unprivileged client without handing over the privilege.
And that is a capability transfer, not a hint — which a single run makes unarguable:
parent: unlinked the path before sending; only the descriptor survives
child : received fd=3, read 25 bytes: "hello across the boundary"
child : offset now 5 (parent had already read 5 bytes)Two facts in three lines. The path was gone before the message was sent, so the child could not have opened it by name under any circumstances. And the child observed the parent’s offset of 5, which proves it received the open file description itself — section 1A of the I/O chapter, arriving through a socket. The receiver gets access it could not have obtained, with no widening of its own file permissions. measured here
What the receiver must still distrust: the contents, obviously — and the descriptor’s kind. Check it is the type you expect and that its flags match what you promised, and remember the offset is now shared, so concurrent use needs pread / pwrite. inference
Peer credentials, measured: getpeereid() → euid 501, egid 20; LOCAL_PEERCRED → version 0, uid 501, 16 groups; LOCAL_PEERPID → the pid. Kernel-supplied, not peer-claimed — but uid and pid only, and a pid is not an identity. For identity, see 4A. measured here
measured here 4.85 µs round trip for SOCK_STREAM and 4.88 µs for SOCK_DGRAM — indistinguishable from a pipe. But bulk throughput was 1,645 MB/s against the pipe’s 7,680 at the same 64 KiB chunk size, about 4.7× slower — because its default capacity is 8,192 bytes, so it delivers at most 8 KiB per read whatever you ask for. Section 3B has the read counts, and the per-socket tunable that fixes it once it is set on the sending end.
Use it when: the processes are unrelated, you need two-way traffic, you need to pass a descriptor, or you want datagram framing for free. SOCK_DGRAM costs you a maximum datagram size and truncation of oversized messages.
MessageMach ports and messages
What it is. The kernel’s native IPC primitive, and what everything else on macOS is built on. A port is a kernel-managed message queue. Rights to it are capabilities: a receive right (exactly one holder — the server) and send rights (any number — the clients). Holding a send right is the permission to send; there is no separate access check.
What it gives you. Real messages with boundaries. Capability-based access — you cannot forge a port right, and passing one is how authority is delegated. Rights can be sent inside messages, which is how a service hands out a connection. And dead-name notifications, so you learn asynchronously that a peer is gone.
What it costs. A famously unforgiving API. The bookkeeping — right types, disposition, trailers, timeouts — is where the bugs are, and it is exactly what XPC exists to hide.
measured here 4.36 µs round trip between two threads, and 5.72 µs across a real process boundary through the bootstrap server with a registered MachServices job — so it is in the same band as a pipe rather than decisively ahead of one. There is no performance reason to avoid Mach; the reason is the API surface.
The first mistake everyone makes, measured here: a receive buffer sized to the message returns MACH_RCV_TOO_LARGE (0x10004004). The kernel appends a trailer, so the buffer must be sizeof(message) + sizeof(mach_msg_max_trailer_t). The symptom is a receive that always fails while the send always succeeds.
Use it when: you are implementing something XPC cannot express, or you are reading a crash log and need to know what the frames mean. For application code, use XPC.
Message · the defaultXPC
What it is. Apple’s framing: “XPC provides a lightweight mechanism for basic interprocess communication. It allows you to create lightweight helper tools, called XPC services, that perform work on behalf of your app.” Connections are “peer-to-peer”: a listener responds to incoming connections, a client creates a session and then “sends messages and receives replies”.
What it gives you that raw Mach does not. Message framing, a typed object model, lifecycle managed by launchd including restart after a crash, and connection events that tell you the peer died. Apple lists the benefits as centralising work, delegating work “so it continues beyond a client’s life cycle”, and “privilege isolation to narrow the scope of access”.
Two levels of API. “If your project uses the Foundation framework, NSXPCConnection provides a high-level object-oriented API that enables a transparent remote method dispatch mechanism between processes … If your project doesn’t or can’t link against Foundation, use the lower-level libSystem APIs in the XPC framework.”
The safety property that matters. With NSXPCConnection you declare an interface, and the allowed classes for each argument are whitelisted. That is not ceremony — it is the bounding of untrusted input that hand-written protocols forget.
Use it when: you are shipping a helper on macOS and both ends are yours. This should be the default, and needing a reason to not use it is the right way round. https://developer.apple.com/documentation/xpc
Zero copyShared memory
What it is. The same physical pages mapped into two address spaces — shm_open plus mmap, or mmap(MAP_SHARED|MAP_ANON) across a fork. No copy, no system call per access.
What it gives you. Speed of a completely different order. measured here 0.08 µs round trip cross-process against 4.8 µs for every kernel mechanism — about 60× faster — and 78,073 MB/s bulk against a pipe’s 8,390, which is simply memcpy speed.
What it costs, and it is a lot. No framing, no notification, and no trust boundary at all: a peer that corrupts the region corrupts you, so the isolation you crossed the boundary for is gone. You now own every synchronisation problem in the threads chapter, across processes, without a shared runtime. And you still need a second channel to signal that something happened, which will cost the kernel round trip you were avoiding.
Use it when: the payload is large, the peer is trusted, and the 4.8 µs genuinely matters — video frames, audio buffers, large bitmaps. Almost always paired with a control channel. IOSurface is the framework-supported version of this for image data.
BroadcastDistributed notifications
What it is. DistributedNotificationCenter — a system-wide broadcast of a name and an optional small payload to any process that registered for it.
The primitive underneath it is notify(3), Darwin’s documented low-level notification API, and it is worth knowing because its shape explains the limitation of everything built on it:
notify_register_check: status=0 token=2
notify_post status=0 ; notify_check saw change=1 ; post+check = 5.9 us
notify_check again (no post): change=0notify(3) is a state-change flag, not a queue. Two posts you never checked between collapse into one. No payload, no sender, no ordering, no delivery guarantee. That is precisely why the line below — distributed notifications are signals, not durable state — is right, and now it has a mechanism underneath it rather than being good advice. It is correct for exactly one shape: “something changed, reconsider your state.” measured here documented man 3 notify
The test to apply: if losing the message silently is unacceptable, you need something that can tell you it was lost — which means a channel with a reply, not a broadcast. inference
What it gives you. Loose coupling and no connection management. Good for “something changed, reconsider your state”.
What it does not give you. Delivery guarantees, ordering, a reply, or any notion of who sent it. It is also system-wide: your notification names are visible to other processes, and so is the fact that you posted one. Never put anything sensitive in the payload, and never treat receipt as authorisation.
Use it when: the message is a hint, losing it is acceptable, and the payload is not sensitive. If you need a reply, this is the wrong tool. https://developer.apple.com/documentation/foundation/distributednotificationcenter
IndirectFiles, and why they are IPC too
What it is. One process writes, another reads. It is genuinely IPC, it is extremely common, and it is the mechanism with the least obvious failure mode — which is exactly why exercise 06 exists in the I/O chapter.
What it gives you. Persistence across restarts, no connection to manage, and any number of readers.
What it costs. There is no atomicity unless you arrange it: measured there, a reader caught a file mid-rewrite 19.9% of the time. Write a sibling temporary, fsync, and rename — then a reader sees all of the old file or all of the new one and never anything else.
Use it when: the state must survive both processes, or the reader may not exist yet. See the I/O chapter’s exercise 06.
3 · What they cost
Latency does not distinguish them. Throughput and semantics do.
This is the measurement that changes how people choose. Five mechanisms, the same 1-byte ping-pong, in-process and then across a real forked process boundary.
3ARound-trip latency, 20,000 rounds eachmeasured · in-process and cross-process
| Mechanism | In-process (2 threads) | Cross-process (fork) | Relative to the pipe cross-process ÷ 4.80 µs |
|---|---|---|---|
| Anonymous pipe | 4.92 µs | 4.80 µs | 1.00× |
Unix socket, SOCK_STREAM | 4.91 µs | 4.85 µs | 1.01× |
Unix socket, SOCK_DGRAM | 4.93 µs | 4.88 µs | 1.02× |
Mach port, raw mach_msg | 4.74 µs | 5.72 µs | 1.19× |
XPC, send_message_with_reply_sync | — | 16.7 µs | 3.48× |
XPC, send_message_with_reply + a semaphore | — | 23.0 µs | 4.79× |
| Shared memory + spin | 0.08 µs | 0.08 µs | 0.017× |
Three readings, and all three are useful in an interview.
One · the low-level mechanisms are one band, and XPC is not in it. Pipe, Unix socket and raw mach_msg land between roughly 5.0 and 5.9 µs cross-process — a spread under 30%, with no reason to prefer one on latency. XPC is about 3.5× a pipe, and an earlier version of this page implied otherwise by inviting the reader to carry the ~5 µs figure onto a mechanism that had never been measured. It has now been measured, against a real launchd-registered service, and 16.7 µs is what it costs. measured here
One-and-a-half · what the extra 11 µs buys, and the version that wastes it. Framing, a typed object model, launchd lifecycle and type checking. The second XPC row is the pattern Apple’s own header tells you not to write: “This API supports priority inversion avoidance, and should be used instead of combining xpc_connection_send_message_with_reply() with a semaphore.” It costs a further 38% and gives up priority-inversion avoidance — a semaphore has no owner to donate to, which is the threads chapter’s lesson one layer up. documented xpc/connection.h in the macOS 26.2 SDK.
Two. Crossing a real process boundary cost essentially nothing extra: 4.92 µs between two threads versus 4.80 µs between two processes for the same pipe. The cost is not the boundary; it is the kernel transition and the scheduler wakeup, and you pay that either way.
Three. Shared memory is about 60× faster than the low-level band, and the reason is that it never enters the kernel at all. That is the whole of the trade: you are not buying a faster IPC mechanism, you are buying no IPC mechanism, and giving up the isolation and the framing and the notification that came with one.
measured here Apple M4 Pro, macOS 26.3 (25D125), 20,000 rounds per mechanism, repeated. Reproduce; do not quote. The ordering was stable across runs; the absolute figures move with load. The Relative column has one baseline throughout: this table’s own cross-process pipe figure of 4.80 µs. Elsewhere on the page the pipe is rounded to 5 µs for frame arithmetic, which is why “roughly 3,300 pipe round trips per frame” and this column’s 3.48× are both right.
3BBulk throughput, 256 MiB across a process boundarymeasured · where the mechanisms finally differ
| Mechanism | Chunk | MB/s | Relative |
|---|---|---|---|
| Anonymous pipe | 64 KiB | 7,680 | 4.67× |
| Anonymous pipe | 1 MiB | 8,390 | 5.10× |
Unix socket, SOCK_STREAM, default buffers | 64 KiB | 1,645 | 1.00× |
Unix socket, SOCK_STREAM, SO_SNDBUF 1 MiB on the writer | 1 MiB | 28,464 | 17.3× |
Shared memory (memcpy) | 1 MiB | 78,073 | 47.5× |
Here the mechanisms finally separate. A pipe moved bulk data 4.7× faster than an untuned Unix stream socket at the same chunk size, despite being indistinguishable in latency. If you are streaming megabytes between processes you launched, that is a real and easy win — and it is invisible in any ping-pong benchmark. The gap is a default, not a ceiling: the tuned row below reaches 17× the untuned socket and beats the pipe, and the box under the read counts explains why an earlier version of this page reported that it could not.
And the mechanism is visible in the call count, which is the part that makes it teachable. Counting reads for the same 256 MiB:
transport chunk SO_SNDBUF on MB/s reads
pipe 64 KiB — 8307 4096
pipe 1 MiB — 8529 4096
socketpair 64 KiB — 1648 32768
socketpair 64 KiB writer, 64 KiB 7719 4096
socketpair 64 KiB writer, 256 KiB 25019 4096
socketpair 1 MiB writer, 1 MiB 28464 339
socketpair 64 KiB reader, 64 KiB 1510 32768At its default buffer size the socket delivers at most 8 KiB per read however much you ask for, so it makes eight times the calls — and that is the whole of the untuned 4.7× gap. A pipe holds 65,536 bytes; a socketpair holds 8,192, matching net.local.stream.sendspace / recvspace, which are both 8192. Raise SO_SNDBUF on the sending end and the granularity moves with it: 64 KiB of send buffer turns 32,768 reads into 4,096 and roughly 1,650 MB/s into roughly 7,700; 1 MiB of send buffer with 1 MiB chunks reaches 339 reads and about 28,000 MB/s, which is faster than the pipe. Section 2 of the I/O chapter is the same lesson: bytes per system call is the metric. measured here
CORRECTION · SO_SNDBUF WORKS. AN EARLIER VERSION OF THIS PAGE PUT IT ON THE WRONG DESCRIPTOR. This box used to report SO_SNDBUF as accepted, reported back by getsockopt, and completely inert. That result was an artefact of the measurement, not a property of the socket: the harness set SO_SNDBUF on the reading end of the pair and SO_RCVBUF on the writing end — both on the descriptor that does not govern the transfer. Neither option touched the buffer the writer actually fills, so the socket stayed at its 8,192-byte default and nothing moved. With each option on the end that owns it — SO_SNDBUF on the writer, SO_RCVBUF on the reader — the same harness on the same machine shows 4.7× to 17× more throughput and 8× to 97× fewer read calls. measured here version-sensitive
Keep the lesson, and note where it now points. setsockopt returned 0 and getsockopt faithfully read the raised value back — on the descriptor it had been set on. Every signal available to the developer said the change had taken effect, and every one of them was true and irrelevant. The trap is not a tunable that lies; it is a confirmation that answers a different question than the one you asked. Confirming a setting on one descriptor is not evidence that the transfer uses it. The control that settles it is the one this page already teaches in section 2.7 of the I/O chapter: change only which end receives the option and re-measure. The four-way control below is the measurement; the generalisation from it is reasoning. measured here inference
SO_SNDBUF 65536 on MB/s reads bytes/read getsockopt w/r
nothing (default) 1609 32768 8192 8192 / 8192
the writer (correct) 6099 4096 65536 65536 / 8192
the reader (old bug) 1510 32768 8192 8192 / 65536
both ends 7570 4096 65536 65536 / 65536Read the third row against the first. They are the same transfer at the same speed with the same call count, and the only difference between them is a getsockopt that reports 65,536 on a descriptor nobody writes to. That is exactly what the old box saw. What survives unchanged is the design advice: to stream megabytes to a process you launched, a pipe is still the right default — not because a socket cannot go faster, but because the pipe reaches 64 KiB per read with no tuning at all, while the socket needs an option on the correct descriptor to get there. inference
measured here Apple M4 Pro, macOS 26.3 (25D125), 256 MiB per run through a forked child, three runs of each variant. Reproduce; do not quote. The read-call counts are load-independent and repeated exactly; the MB/s figures moved with load and are given to the nearest hundred in the prose. version-sensitive net.local.stream.sendspace is system-wide, needs root, and was never raised — so everything above is a property of the per-socket option, which is the one that turned out to work.
And shared memory is 47× the socket, because at that point you are measuring memcpy and the memory subsystem rather than any IPC mechanism at all.
The design conclusion, with the arithmetic corrected. Choose on shape: request/reply at human-noticeable rates — use XPC; 16.7 µs is 0.1% of a frame. Streaming megabytes to a process you launched — a pipe, which reaches 64 KiB per read untuned, or a socket with SO_SNDBUF raised on the sending end if you need the socket’s other properties. Sharing large frames with a trusted peer at video rates — shared memory or IOSurface, with a control channel. The conclusion did not change when XPC was measured; only the number did. inference
measured here 256 MiB per run through a forked child. The shared-memory figure is one memcpy per chunk into a MAP_SHARED region and is a ceiling, not a like-for-like protocol. The tuned socket row and the read-count table below it were measured in a later correction pass on the same machine and OS build, after the descriptor mistake described in the correction box was found; the other rows are from the original run.
Simplified diagram. Bar lengths are proportional to the measured means at a single scale of 12 units per microsecond, so the four bars can be compared with a ruler; the internal split of the XPC bar is illustrative, not an instrumented breakdown. inference on where the extra time goes; measured here on the totals.
What 5 microseconds actually means
Put it next to the other numbers in this reference. A low-level IPC round trip is about 5 µs and an XPC one about 16.7 µs. A single frame at 60 Hz is 16,700 µs. An F_FULLFSYNC measured 4,013 µs. So a frame has room for roughly 3,300 pipe round trips or 1,000 XPC ones, and one F_FULLFSYNC costs as much as 240 XPC calls.
The earlier version of this paragraph said “three thousand” without distinguishing the two, which quietly applied the pipe figure to the mechanism the chapter actually recommends. Both numbers are now measured; the order of magnitude, and the conclusion below, are unchanged.
IPC latency is almost never your problem. What is your problem is doing a round trip inside a loop that runs a million times, or doing one synchronously on the main thread and waiting. Both are structural, and neither is fixed by choosing a different mechanism — a 3.5× difference is irrelevant to the first and does not rescue the second.
One design line that follows from the payload sweep. An XPC round trip carrying 1 KiB cost 22.6 µs against 21.8 µs for an empty one — the payload was free. Large payloads plateau around 9 GB/s, well above an inline copy, which is consistent with a switch to out-of-line memory although that crossover was not instrumented. So do not design a chatty protocol in order to keep messages small; design a quiet one. measured here inference on the out-of-line explanation.
4 · Choosing
Six questions, in order.
Ask them in this order and the mechanism usually names itself. Answering them out loud is also, almost exactly, how this question is graded in an interview.
| # | Question | If yes | If no |
|---|---|---|---|
| 1 | Do you actually need a separate process? | Continue — name which of containment, privilege or lifetime you are buying | Stop. Use a thread or a queue and keep the protocol you do not have to write |
| 2 | Do you control both ends, and is this macOS? | XPC. Framing, lifecycle, launchd restart and type checking come free — and a peer-identity check is one call (4A). Not free: 16.7 µs per round trip, 3.3× a pipe | Continue |
| 3 | Did you launch the peer, and is the data a stream? | Pipe. Backpressure built in, best bulk throughput measured | Continue |
| 4 | Are the processes unrelated, or must you pass a descriptor? | Unix domain socket — a filesystem name, peer credentials, and SCM_RIGHTS | Continue |
| 5 | Is the payload large, the peer trusted, and is 5–17 µs genuinely too slow? | Shared memory or IOSurface, plus a control channel — and you now own the synchronisation, which on Darwin is not the set of primitives you expect (5A) | Continue |
| 6 | Is it a fire-and-forget hint that may be lost? | Distributed notification — nothing sensitive in the payload | Re-read question 1 |
Question 1 is the one that is graded
“Should this be a separate process at all?” is the actual design question, and a candidate who reaches for a mechanism before answering it has skipped the interesting part. Isolation is bought for a reason; name the reason.
Serialisation is part of the choice
Whatever crosses must be encoded. Codable to JSON is debuggable and slow; a packed binary struct is fast and brittle across versions; NSSecureCoding through NSXPCConnection is checked and Foundation-bound. Say which you picked and what happens when the two sides are different versions.
Version the protocol from day one
Two processes update independently — a helper installed once and an app updated weekly, or the reverse. Every message needs a version, every reader needs a policy for an unknown field, and “they ship together” stops being true the first time a user restores from a backup.
The trust boundary is not symmetric
A privileged helper must treat its client as hostile; a client should treat a helper’s replies as untrusted too. On macOS, verify the peer’s code signature and required entitlements — being on the same machine proves nothing about who is calling. That advice is only useful with an API attached, and there is a documented one: see 4A · Proving who the peer is.
4A · Trust
“Verify the peer” is one call, made before you activate.
Section 4 says a privileged helper must verify its client’s code signature and entitlements. That is correct advice and, without an API, useless advice — it is the point at which people either invent something unsound or skip the check. macOS has a documented public family for exactly this, and the enforcement is in the kernel rather than in your handler.
| Function | Available since | What it requires of the peer |
|---|---|---|
xpc_connection_set_peer_code_signing_requirement | macOS 12.0 | A code-signing requirement string |
xpc_connection_set_peer_entitlement_exists_requirement | macOS 14.4 | That a named entitlement is present |
xpc_connection_set_peer_entitlement_matches_value_requirement | macOS 14.4 | That an entitlement has a specific value |
xpc_connection_set_peer_team_identity_requirement | macOS 14.4 | A specific Team ID |
xpc_connection_set_peer_platform_identity_requirement | macOS 14.4 | That the peer is an Apple platform binary |
xpc_connection_set_peer_lightweight_code_requirement | macOS 14.4 | A lightweight code requirement |
documented All six read directly from $(xcrun --show-sdk-path)/usr/include/xpc/connection.h in the macOS 26.2 SDK, with their API_AVAILABLE annotations. The Foundation equivalent is NSXPCConnection.setCodeSigningRequirement(_:), macOS 13+. Raw URLs: https://developer.apple.com/documentation/foundation/nsxpcconnection/setcodesigningrequirement(_:) https://developer.apple.com/documentation/technotes/tn3127-inside-code-signing-requirements
What the header promises, quoted
“All messages received on this connection will be checked to ensure they come from a peer who satisfies the code signing requirement. For a listener connection, requests that do not satisfy the requirement are dropped. When a reply is expected on the connection and the peer does not satisfy the requirement XPC_ERROR_PEER_CODE_SIGNING_REQUIREMENT will be delivered instead of the reply.”
“It is a programming error to call xpc_connection_set_peer_code_signing_requirement more than once per connection.” … “This API is not supported on embedded platforms and will return ENOTSUP.”
documented xpc/connection.h, macOS 26.2 SDK. Read the second sentence carefully: the check applies to every message, once, set before activation — not something your handler re-does per request.
requirement setter rc outcome
(none) 0 ACCEPTED
identifier "xpcserver" (true) 0 ACCEPTED
identifier "definitely-not-the-server" (false) 0 REJECTED XPC_ERROR_PEER_CODE_SIGNING_REQUIREMENT
anchor apple (false - binary is ad-hoc) 0 REJECTED XPC_ERROR_PEER_CODE_SIGNING_REQUIREMENT
this is not a valid requirement ((( 22 rejected up front (EINVAL)measured here Reproduced in this session against a registered LaunchAgent. Read the last row. A malformed requirement string fails at set time with EINVAL, not at first message — so a typo is a startup failure you will notice, rather than a silently-disabled check. That is the property that makes this API safe to rely on.
Not by pid, ever
A Unix socket will give you kernel-supplied credentials: measured here getpeereid() returns euid and egid, LOCAL_PEERCRED returns uid and group count, LOCAL_PEERPID returns a pid. They are kernel-supplied rather than peer-claimed, which is genuinely better than a self-report — and it is uid and pid only. A pid is not an identity: it can be reused, and by the time you look the process may be a different one. measured here
The sentence for an interview
“On macOS, verifying the peer is not advice you implement by hand — it is one call, made before you activate the connection, and the check then applies to every message. The mistake is doing it per-request in the handler, or doing it by pid.”
Authentication is not validation
The requirement API answers who is calling. It says nothing about what they sent. Bound every length before allocating, allowlist rather than blocklist, and keep the interface narrow: “install the update at this verified path” is a boundary; “run this command” is not a boundary with extra steps, it is no boundary.
A listener drops silently, by design
Non-conforming requests to a listener are dropped with nothing delivered. That is the right security behaviour and a confusing debugging experience: “my helper never receives anything” looks like a registration bug. Check whether a requirement is set before you go looking at launchd. documented
WHAT THIS DEMONSTRATES, AND WHAT IT DOES NOT. The four outcomes above establish the enforcement mechanism. They are not a security review. The exercise ran between two ad-hoc linker-signed binaries owned by one user in that user’s GUI domain — no Developer ID chain, no notarisation, no sandboxed helper, no SMAppService installation, no privileged LaunchDaemon, and no real entitlements. A production trust design adds installation, update and revocation concerns none of this touches. negative result The listener-side silent drop is quoted from the header and was not independently verified — every rejection above was observed from the client side.
5 · Failure modes
The peer can die. That is the feature, and the bill.
You moved the work into another process so that its crash would not be yours. It follows that its crash will happen, and that handling it is part of the API rather than an error path you add later.
| Mechanism | Peer dies while you write | Peer dies while you read | How you learn |
|---|---|---|---|
| Pipe / Unix socket | SIGPIPE — terminates your process by default | read returns 0 (end of stream) | Only if you disable the signal and check errno == EPIPE |
| Mach port | MACH_SEND_INVALID_DEST | Receive never completes | A dead-name notification, asynchronously |
| XPC | Delivered as a connection error | Same | XPC_ERROR_CONNECTION_INTERRUPTED (crashed, will restart) vs XPC_ERROR_CONNECTION_INVALID (gone for good) |
| Shared memory | Nothing — you keep writing into the region | Nothing — you read whatever is there | You do not. You need a separate liveness channel |
| File | Nothing | Stale data, silently | Only by putting a timestamp or a sequence in the data |
Simplified diagram, not an exhaustive list of error codes. The XPC row is the only one where the mechanism distinguishes “crashed, restarting” from “gone” — which is the whole argument for it in section 4.
Read the first row again
Writing to a pipe or socket whose reader is gone raises SIGPIPE, and its default disposition terminates your process. Not an error return — the end of the process. Measured here, a fixture that streams a document to a helper that dies early exits with status 141 (128 + 13) and prints nothing after it started streaming: the if (w < 0) branch it already contained is unreachable, because nothing survives to run it.
The repair, and why the obvious one is wrong for a library. signal(SIGPIPE, SIG_IGN) works and is process-wide. An application may legitimately want SIGPIPE to terminate it — that is what makes producer | head -5 exit cleanly. A framework that silently changes its host’s signal disposition has reached outside its own boundary. fcntl(fd, F_SETNOSIGPIPE, 1) scopes it to one descriptor; sockets have SO_NOSIGPIPE, and a single send can pass MSG_NOSIGNAL.
And the asymmetry worth remembering: the writer dies loudly and the reader ends quietly. A reader whose peer vanished just sees end-of-stream, indistinguishable from an orderly finish — so a protocol that can be truncated needs its own end marker.
“launchd restarts it if it crashes” is true, rate-limited, and your client will feel the limit
A real on-demand LaunchAgent was registered, measured, killed and watched recovering. Everything here is observation, not documentation.
On-demand launch is directly visible. Before any client contact launchctl print reports state = not running and runs = 0; after one message, state = running, runs = 1, and a pid. measured here documented “The launchd system daemon manages these services, launching them on demand, shutting them down when idle, and restarting them if they crash.” https://developer.apple.com/documentation/xpc
before: err=none pid=77900
during-crash-request err=none
EVENT interrupted: Connection interrupted <- peer aborted
after: err=none pid=78680 (SAME connection object reused)
launchctl: runs 1 -> 2
$ launchctl bootout gui/501/com.example.demo
EVENT invalid: Connection invalid
ping-after-bootout: err=INVALIDThe same connection object served a request answered by a different process. That is the concrete payoff of the claim in the table above: XPC turns a SIGPIPE into a designed API, and INTERRUPTED versus INVALID is a real distinction you can act on — reissue after the first, give up after the second. measured here
And now the trap. launchctl print also reports minimum runtime = 10. Killing the service inside that window and timing the client’s next request:
| Service killed… | Client’s wall-clock wait | Outcome |
|---|---|---|
| before 10 s of runtime | 10.05 s | the request timed out |
| after 12 s of runtime | 0.05 s | relaunched in 20.6 ms, request served |
launchd throttles respawn, so a crash loop does not become a spin loop — which is correct behaviour and has a consequence for your protocol: a client deadline shorter than the throttle window turns a successful restart into a reported failure. If your helper can crash on startup and your timeout is five seconds, you will report an outage that is actually a working recovery. measured here version-sensitive minimum runtime is a launchd value, not a documented API contract.
Cold start is not one number either. First launch ever, nothing cached: 234–301 ms. After a bootout/bootstrap with the binary warm: 19–21 ms. Respawn past the throttle window: 20.6 ms. Warm round trip to a running service: 0.015 ms. A 12–20× spread between first-ever and warm-respawn, and the first launch after an update is the one the user sees. Quoting one figure for “XPC launch cost” is how that surprise happens. measured here
Mis-framing is a loss of sync, not a bad message
measured here A receiver that assumed one read per message recovered 328 of 2,000 messages, with 7,000 reads that had lost frame alignment entirely — every byte having arrived correctly. Once a boundary is missed, the next read starts mid-payload and interprets payload as header. It does not recover on its own.
Mis-framing can also hang
The first version of that fixture stopped draining once it had done its expected number of reads, which filled the socket buffer and blocked the sender forever. A framing bug can present as a wedge rather than as corruption — worth knowing before you conclude a hang means a deadlock.
A length off the wire is hostile input
Bound it against a maximum before allocating or reading that many bytes. Failing to is how a framing bug becomes a denial of service or worse. Message-oriented frameworks do this for you; hand-written protocols must do it explicitly, and the fixtures here do.
Reap your children
A child that exits and is never waited for stays as a zombie holding a process-table entry. Every fixture in this bundle calls waitpid. In production, decide deliberately whether a dead helper is restarted, and bound the retries — a helper that crashes on one document will crash on it again.
5A · Locking across processes
Shared memory hands you every problem from the threads chapter — with two of the primitives missing.
Section 2 says shared memory means you own every synchronisation problem in the threads chapter, across processes. True, and it stops one step early: which primitives actually work across a process boundary on Darwin is not the list a candidate arriving from Linux expects, and getting it wrong fails silently.
| Primitive | Works across processes on Darwin? | Evidence |
|---|---|---|
sem_init(&s, pshared=1, 0) | No — rc=-1, errno=78 ENOSYS “Function not implemented” | measured here |
sem_open("/name", …) — named | Yes | measured here |
pthread_mutexattr_setpshared(PTHREAD_PROCESS_SHARED) | Accepted, rc=0, reads back 1 | measured here |
that mutex in MAP_SHARED memory, 2 processes × 200,000 increments | Yes, exactly — 400,000 / 400,000 | measured here |
| the same loop with no lock (control) | No — lost 130,831 updates on one run | measured here |
os_unfair_lock across processes | Not validated here — do not claim either way | — |
Two things a candidate is routinely wrong about
1 · Unnamed POSIX semaphores do not exist on Darwin. sem_init returns ENOSYS, and sys/semaphore.h additionally marks it __deprecated. Code ported from Linux compiles, links, runs, returns -1 at runtime — and if that return is unchecked, which it usually is, the “synchronisation” silently does nothing at all and the program is a data race wearing a lock. documented measured here
2 · The unlocked control is the more useful measurement. Across four runs it lost between 14,216 and 200,000 updates out of 400,000 — a different number every time. A lost-update bug that loses a variable amount is a bug that reproduces differently every time you look at it, which is exactly why “it passed once” proves nothing here. Assert the exact total and run it repeatedly.
What to reach for. A PTHREAD_PROCESS_SHARED mutex living in the MAP_SHARED region itself — not in either process’s private memory — or a named semaphore. And remember what the threads chapter says about a mutex whose holder dies: nothing releases it. Across processes that is no longer hypothetical, because section 5 is about the peer dying. A shared-memory design needs a liveness channel and an answer for a lock held by a process that no longer exists.
inference The dead-holder consequence is reasoning from the mutex contract plus section 5, not something measured here. Robust-mutex behaviour was not tested.
6 · Diagnosis
Find out where the bytes stopped.
Almost every IPC bug reduces to one of four questions, and each has a cheap first measurement.
| Symptom | First question | How to answer it |
|---|---|---|
| The app disappears with no crash log you can read | Was it killed by a signal? | Shell exit status above 128 → subtract 128. 141 is SIGPIPE |
| Replies are corrupt, but only for large messages | Did every byte arrive? | Count bytes read on both sides. Equal → framing, not transport |
| The helper never replies | Is it alive, and is it running? | ps for existence, then sample <pid> to see whether it is blocked or busy |
| Everything hangs under load | Is somebody not draining? | A full pipe or socket buffer blocks the writer. Check both ends’ read loops |
| Works for you, fails for a user | Is it a permission or a signature? | Sandbox denials and code-signature rejections appear in the unified log, not as an error you returned |
| Intermittent stale data | Is the file published atomically? | Written in place → the reader can catch it mid-write; see the I/O chapter |
Unprivileged, and enough
# did a signal kill it? 141 = 128 + 13 = SIGPIPE
./myapp ; echo "exit status: $?"
# is the peer blocked, or busy?
sample <pid> 3
# what descriptors and sockets does it hold?
lsof -p <pid>
# is the service registered, and what state is it in?
launchctl list | grep myhelper
launchctl print gui/$(id -u)/com.example.myhelperInstrument the protocol itself
// On BOTH sides, count and log:
// messages sent / messages received
// bytes written / bytes read
// reads that returned short
// frames rejected, and why
//
// Bytes equal + messages unequal -> framing bug
// Bytes unequal -> transport or truncation
// Both equal, behaviour wrong -> semantics, not plumbingThis table is the whole diagnosis for exercise 07, and it took one run. A protocol that cannot tell you how many messages each side thinks it handled is a protocol you cannot debug from a customer’s machine.
The tool boundary that wastes the most time
lsof knows about descriptors. XPC does not use descriptors. So a helper that is busily serving XPC requests looks, through lsof, like a process holding nothing — and the natural conclusion is that the connection was never made. It was; you are asking the wrong tool.
| Question | Command | What you get |
|---|---|---|
| What files and sockets does it hold? | lsof -p <pid> | Descriptors, paths, pipes, Unix sockets |
| Who is it connected to over XPC or Mach? | lsof cannot tell you | Use launchctl print |
| Is the service registered, running, restarting? | launchctl print gui/$(id -u)/<label> | state, runs, pid, last exit code, minimum runtime |
| Which Mach endpoints does it vend? | the same, then read endpoints = { … } | Port number, active, managed |
| Blocked or busy? | sample <pid> 1 | An idle XPC service parks in __sigsuspend_nocancel; its workers in __workq_kernreturn |
| Unix sockets, system-wide | netstat -f unix | Endpoints and their connection pairs |
| Network bytes per process | nettop -x -L 1 | Works unprivileged |
| Both sides of the boundary on one timeline | log show --last 2m --predicate 'subsystem == "…"' --style compact | Interleaved, unprivileged, and the single most useful one |
That last row is worth the paragraph. One command, no elevation, and a crash-and-restart story reads itself off the output — two different pids under one subsystem in one timeline:
09:50:13.747 Df xpcserver[13247] [com.example.iolab:xpcserver] deliberate abort requested
09:50:15.783 Df xpcserver[71102] [com.example.iolab:xpcserver] service starting pid=71102measured here Every command in the table was run. This is the strongest argument on the page for OSLog on both sides of a boundary, and it cost one command rather than a debugger.
Signposts across the boundary
Emit an OSSignposter interval on the client around each request and on the helper around each handler. The gap between them is the transport plus scheduling, and seeing it as two intervals rather than one immediately separates “the helper is slow” from “the request waited”.
https://developer.apple.com/documentation/os/ossignposter
Log to the unified log, on both sides
A helper has no console. OSLog from both processes with a shared subsystem gives you one interleaved timeline, which is the only practical way to reconstruct an ordering problem across a boundary.
https://developer.apple.com/documentation/os/oslog
Test the peer dying, deliberately
Kill the helper mid-request in a test. Most IPC bugs are in a path no happy-path test executes, and “what happens when the other side vanishes” is a specification question your protocol should already answer.
Test with large messages
Exercise 07’s bug is invisible below the socket buffer size. Any test suite for a stream protocol that uses only small messages is testing the case that cannot fail. Include one message larger than 128 KiB.
What this chapter did not do. No Instruments GUI session was opened and no screenshot appears anywhere. No XPC service was built, registered or measured — everything said about XPC is quoted from Apple’s documentation, and every measured number in this chapter comes from pipes, Unix domain sockets, Mach ports and shared memory exercised directly from C. Mach ports were measured in-process between two threads, not across a real process boundary through the bootstrap server, because that needs a registered service; the cross-process figures are for pipes, sockets and shared memory only. No SCM_RIGHTS descriptor passing, no code-signature or entitlement verification, and no sandboxed helper was exercised — those are documentation here, not observation.
7 · Fixing exercises
Two broken programs. Diagnose, repair, prove.
Exercises 07 and 08 of the same bundle. Both use pipes and Unix sockets rather than XPC, deliberately: they reproduce identically in a self-contained program with no service registration, and the defects they contain are the ones that survive into production.
Same bundle as the other three chapters
os-memory-io-ipc-exercises.tar.gz — 44 files, 55,129 bytes. SHA-256 0a04e5544c047fc5376919d91fdcf5943a1c1316aa38996d93bac105b23ba13f · raw path labs/os-memory-io-ipc-exercises.tar.gz
./run-all.sh 07 0807 · The helper that only corrupts large documents
A helper process and its client talk over a Unix domain socket. Each message is a self-describing frame: a header with a magic number, a sequence number, a payload length and a checksum, then that many payload bytes. The author tested with short messages, where one write reliably produced one read, and concluded a stream socket delivers messages. The field reports that it works perfectly in testing and corrupts replies in production — but only for large documents, and only sometimes. Prove where the bytes go, fix it, and tell me what your fix costs.
clang -O2 -g -Wall -Wextra broken/framing.c -o /tmp/fr_broken
/tmp/fr_brokenExpected signal. goodMessages=328 of 2,000 — goodFraction=0.1640 — with lostSyncFrames=7000, and bytesRead=58472000 which is exactly what the sender sent. The fixed build recovers 2,000 of 2,000 with the identical byte total. measured here
Success criterion. Every message intact, zero rejected, bytesRead unchanged, your receiver reading the header and the payload as two exact-length reads each looping until satisfied — and you can say why your fix makes more read calls, not fewer, and name the one transport in this family that would not have had the bug.
Progressive hints
- The length was always there.
struct msg_headerhas apayload_lenand the broken receiver reads it. Look at what it does with it: it compares it against however many bytes happened to arrive. It never uses it to decide how many bytes to ask for. - Two reads, not one. You cannot know how big a frame is until you have read its header, and you cannot read a header from a stream in one call either. Read exactly
sizeof(header), then exactlypayload_len. “Exactly” is doing all the work. - What does “exactly” mean when
readreturns short? A loop. And handle a return of 0 at a boundary (clean end of stream) differently from a return of 0 mid-frame (a truncated frame, which is an error).
Solution
static int read_fully(int fd, void *p, size_t n, long *calls) {
unsigned char *b = p; size_t off = 0;
while (off < n) {
ssize_t r = read(fd, b + off, n - off);
if (calls) (*calls)++;
if (r == 0) return off == 0 ? 0 : -1; /* clean EOF, or truncated */
if (r < 0) return -1;
off += (size_t)r;
}
return 1;
}
struct msg_header h;
if (read_fully(fd, &h, sizeof h, &calls) != 1) break;
if (h.payload_len > MAX_PAYLOAD) break; /* bound hostile input */
if (read_fully(fd, buf, h.payload_len, &calls) != 1) break;Why the bug is size-dependent. A stream socket hands the receiver whatever is in its buffer when the read runs. For a small frame that is usually the whole thing, because the sender’s single write landed atomically in the socket buffer and the receiver had not been scheduled yet. For a frame larger than the buffer the sender’s write necessarily completes in several kernel-side pieces, and the receiver can be scheduled between any two of them. “It works on my machine” is a statement about message sizes, not about correctness.
What it costs. More system calls — 11,001 against 8,079, about 36% more — because each frame now takes at least two reads. That is the right trade, and naming it unprompted is what distinguishes a considered answer. If the count mattered, the standard refinement is a user-space ring buffer filled by one large read and drained frame by frame, which is exercise 05’s lesson applied to this problem.
The transport that would not have had this bug. SOCK_DGRAM preserves message boundaries. It costs a maximum datagram size and truncation of anything larger. One layer up, XPC and Mach messages are message-oriented and do this framing for you — which is one of the strongest practical arguments for using them between processes you control.
The repairs that do not work. A bigger receive buffer: the broken receiver already passes the maximum; the kernel is limited by what has arrived, not by your buffer. SO_RCVLOWAT: makes the bug rarer for a fixed frame size and does nothing for variable ones, and rarer is worse. A delimiter instead of a length: workable for text, and it means scanning every byte and escaping the delimiter in payloads. inference
08 · The app that vanishes when the helper crashes
An export feature streams a document to a helper process over a pipe. The field reports that if the helper crashes mid-export, the app vanishes too — no alert, no crash report they can read, nothing in the logs, and only when the helper dies first. Reproduce it, explain why there is nothing in the logs, and fix it so the failure becomes something you can report. Then tell me why the obvious one-line fix is the wrong one for a framework.
clang -O2 -g -Wall -Wextra broken/helperlink.c -o /tmp/hl_broken
/tmp/hl_broken ; echo "exit status: $?"Expected signal. The program prints phase=streaming and then nothing at all, exiting with status 141. Its own if (w < 0) error branch is unreachable. The fixed build exits 0 with errno=32 (EPIPE), helperLost=1, and bytesWritten=393216 — the transfer genuinely started and genuinely did not finish. measured here
Success criterion. Exit 0 with the loss reported, the fix scoped to one descriptor rather than the whole process and you can say why that matters for library code, EPIPE handled distinctly from other errors, and you can say what the caller should do about it. Holding on two consecutive runs.
Progressive hints
- The write is not returning an error. Add a
printfimmediately after thewritein the broken build. It never prints. You are not mishandling an error; you are never given one. - Signals have dispositions, and this one’s default is fatal. Writing to a pipe or socket with no reader raises
SIGPIPE, whose default terminates the process. Turn that into an error return and the branch already in the code starts working. - There are two ways, and they are not equivalent. One is process-wide and one is per-descriptor. If you were writing a framework linked into somebody else’s application, which are you entitled to change?
Solution
if (fcntl(p[1], F_SETNOSIGPIPE, 1) != 0) { perror("F_SETNOSIGPIPE"); return 1; }
...
ssize_t w = write(p[1], doc, CHUNK);
if (w < 0 && errno == EPIPE) {
helper_lost = 1; /* an expected outcome, not an exception */
break;
}Why not signal(SIGPIPE, SIG_IGN). It works, and it is process-wide. An application may legitimately want SIGPIPE to terminate it — that is the behaviour that makes producer | head -5 exit cleanly instead of running forever. A library that silently changes its host’s signal disposition has made a decision that was not its to make. For sockets the equivalents are the SO_NOSIGPIPE socket option and MSG_NOSIGNAL on a single send.
The half of the repair that is not a flag. Suppressing the signal makes the failure reportable; it does not make it handled. A peer in another process can die at any moment — that is the reason you put it there — so “the helper is gone” belongs in the API contract: a distinct error so the caller’s recovery for “disk full” and “converter crashed” can differ; a deliberate, bounded decision about restarting, because a helper that crashes on this document will crash on it again; cleanup of the partial output; and reaping the child.
The same failure in other clothes. A read from a dead peer returns 0 rather than raising anything — the writer dies loudly and the reader ends quietly. A Mach send to a port whose receive right is gone fails with MACH_SEND_INVALID_DEST. An XPC connection delivers XPC_ERROR_CONNECTION_INTERRUPTED when the peer crashed and will be restarted, and XPC_ERROR_CONNECTION_INVALID when it is gone for good — the same event, with a designed API around it and no signal, which is a concrete argument for XPC over raw pipes on macOS.
8 · Interview questions
Fifteen questions, with the follow-up that comes next.
Answer out loud before opening each one.
DesignWhen would you move work into a separate process?
For exactly three reasons, and naming which one is the answer. Crash containment — a parser handling untrusted input, a plug-in you did not write, a codec that faults. Privilege separation — the component touching the network should hold fewer entitlements than the one touching the user’s documents. Independent lifetime — work that must outlive a window, or be shared between clients.
What you pay is protocol: everything crossing must be serialisable, versioned, size-bounded and validated on arrival, and the peer can vanish mid-request.
If none of the three applies, a thread or a queue does the same work with a contract you do not have to write.
Follow-up: “Isn’t it slow?” — measured here, a cross-process round trip is about 4.8 µs. A 60 Hz frame is 16,700 µs. You can afford roughly three thousand round trips per frame. IPC latency is almost never the problem; a round trip inside a hot loop, or a synchronous one on the main thread, is.
ChoosePipe, Unix socket, Mach port, XPC or shared memory — how do you decide?
Not on latency. Measured here, every kernel-mediated mechanism costs about the same round trip — pipe 4.80 µs, Unix socket 4.85 µs, Mach port 4.36 µs — within 15% of each other.
Decide on semantics. XPC if you control both ends on macOS: framing, launchd-managed lifecycle including restart after a crash, and type checking come free. A pipe if you launched the peer and the data is a stream — it has backpressure built in and measured the best bulk throughput, 7,680 MB/s against a Unix socket’s 1,645 at the same chunk size. A Unix socket if the processes are unrelated, you need two-way traffic on one descriptor, or you need to pass a file descriptor with SCM_RIGHTS. Shared memory only when the payload is large, the peer is trusted, and 5 µs genuinely matters.
Follow-up: “Why is shared memory not the default given it is 60× faster?” — because it has no framing, no notification, and no trust boundary. You give up the isolation you crossed the boundary to get, and you inherit every synchronisation problem across processes with no shared runtime.
DiagnoseOur IPC works in testing and corrupts data in production for large payloads.
Framing. A stream socket or pipe carries bytes, not messages, and the receiver is almost certainly assuming one read per message. That assumption holds for small messages, where the sender’s single write lands atomically in the socket buffer and the receiver has not run yet, and fails for anything larger than the buffer.
The measurement that proves it in one run: count bytes on both sides. Measured here, a receiver recovered 328 of 2,000 messages while bytesRead was exactly the byte count sent. Bytes equal and messages unequal means the transport is innocent and the grouping is wrong.
And it is not per-message corruption — it is loss of synchronisation. Once a boundary is missed the next read starts mid-payload and reads payload as header; 7,000 reads had lost alignment entirely.
Follow-up: “The fix?” — read exactly sizeof(header), then exactly payload_len, each in a loop until satisfied, with the length bounded before you use it. It costs about 36% more read calls, and that is the right trade.
DiagnoseOur app disappears when the helper crashes. No log, no crash report we can read.
SIGPIPE. Writing to a pipe or socket whose reader is gone raises it, and its default disposition terminates the process — so the write does not return an error, it ends the program. Any error handling after that call is unreachable.
The evidence is the exit status: measured here, 141, which is 128 + 13. Any shell status above 128 is a signal death and subtracting 128 names the signal.
The repair is fcntl(fd, F_SETNOSIGPIPE, 1), which scopes the suppression to one descriptor so write returns EPIPE instead.
Follow-up: “Why not signal(SIGPIPE, SIG_IGN)?” — it is process-wide. An application may legitimately want SIGPIPE to terminate it; that is what makes producer | head -5 exit cleanly. A framework linked into someone else’s app is not entitled to change that. Sockets have SO_NOSIGPIPE for the same reason.
ExplainWhat is a Mach port, in terms someone who has not used one would follow?
A kernel-managed message queue, plus a capability model on top of it. Rights to a port are the access control: exactly one process holds the receive right — that is the server — and any number hold send rights. Holding a send right is the permission to send; there is no separate check, and you cannot forge one.
The elegant part is that rights can be sent inside messages, so a service can hand a client a fresh port and thereby delegate exactly as much authority as it chooses. Everything else on macOS — XPC, dispatch sources, run-loop wakeups — sits on this.
Measured here it was the fastest mechanism tested at 4.36 µs. The reason to avoid it is not speed; it is that the API is unforgiving.
Follow-up: “What goes wrong first?” — the receive buffer. The kernel appends a trailer, so a buffer sized to the message returns MACH_RCV_TOO_LARGE (0x10004004). It must be sizeof(message) + sizeof(mach_msg_max_trailer_t). The symptom is a receive that always fails while the send always succeeds.
ExplainHow do you know an XPC peer died, and does it matter how?
Through a connection error, and the distinction matters enormously. XPC_ERROR_CONNECTION_INTERRUPTED means the peer crashed and launchd will restart it — your connection object is still usable and reissuing the request is reasonable. XPC_ERROR_CONNECTION_INVALID means it is gone for good — the service was removed, or the name does not exist — and retrying will never work.
Treating those identically produces either a retry loop against a service that no longer exists, or a user-facing failure for a helper that was about to come back.
This is a concrete argument for XPC over raw pipes: the same event over a pipe is a SIGPIPE that kills you, and reconstructing “crashed and will restart” versus “gone forever” is yours to do.
Follow-up: “Who restarts it?” — launchd. Apple: it manages these services, “launching them on demand, shutting them down when idle, and restarting them if they crash.” You do not spawn or supervise them, and designing as though you do is a category error.
DesignA privileged helper gets a request from your app. What do you check?
That the caller is who it claims to be, before anything else. Being on the same machine proves nothing — any process can attempt a connection. Verify the peer’s code signature and the entitlements you require, and reject otherwise.
Then treat the payload as hostile even having verified the caller, because your app is one exploit away from being a confused deputy. Bound every length before allocating; validate every path against an allowlist rather than checking for ..; reject anything you do not recognise rather than ignoring it.
And keep the interface narrow. A helper that exposes “run this command” has no security boundary at all; one that exposes “install the update at this verified path” has a real one. The interface is the boundary.
Follow-up: “Where does NSXPCConnection help?” — it makes you declare an interface and whitelist the allowed classes for every argument. That is not ceremony; it is the bounding of untrusted input that hand-written protocols routinely forget.
ChooseYou need to share decoded video frames with a helper at 60 Hz. How?
Shared memory, or IOSurface which is the framework-supported form for image data — and a small control channel alongside it.
The arithmetic makes the case. A 4K frame is roughly 8 MB. Measured here, a pipe moves bulk at about 8,390 MB/s, so copying a frame costs around 1 ms of a 16.7 ms budget, every frame, plus the copy on the other side. Shared memory measured 78,073 MB/s because it is memcpy — and a properly designed ring does not copy at all.
What you now own: synchronisation across processes without a shared runtime, a buffer-ownership protocol so nobody writes a frame that is being read, and liveness — shared memory tells you nothing when the peer dies, so the control channel has to.
Follow-up: “And if it were 60 small status updates a second instead?” — XPC, without hesitation. Sixty round trips a second at 4.8 µs is 0.03% of one core. Reaching for shared memory there would be trading a real trust boundary for nothing measurable.
ExplainWhy is a distributed notification a poor choice for most IPC?
Because it gives you none of the four things a protocol needs. No delivery guarantee — a listener that was not running simply misses it. No ordering. No reply, so you cannot know it was acted on. And no identity, so you do not know who sent it and the receiver cannot treat it as authorisation.
It is also system-wide: your notification names are visible to other processes, and so is the fact that you posted one. Nothing sensitive belongs in the payload.
It is genuinely good for one shape — “something changed, reconsider your state” — where losing the message costs a stale view until the next one.
Follow-up: “What would you use instead for a real request?” — anything with a connection and a reply. The test is simple: if losing the message silently is unacceptable, you need a channel that can tell you it was lost.
DiagnoseEverything hangs under load, and no process is using any CPU.
Somebody stopped draining. A pipe or socket buffer fills, the writer blocks in the kernel, and if the reader is waiting on that writer for something else, nothing moves again. Zero CPU is the giveaway: this is a blocked wait, not a spin.
The first move is sample on both processes. A writer parked in write and a reader parked anywhere other than read is the whole diagnosis.
It is worth knowing that a framing bug can present this way rather than as corruption: a receiver that stops reading after a fixed number of calls leaves bytes in the buffer, and the sender blocks forever. That happened while building this chapter’s exercise 07, and it is why the shipped fixture drains to end-of-stream instead.
Follow-up: “How would you design it out?” — never make a request that requires the peer to make progress while you are not reading its output. Either read and write on separate threads, or use an event-driven loop, or use a request/reply mechanism such as XPC where the framework owns the draining.
DesignA privileged helper accepts a request. How do you verify the caller — concretely?
Not by pid. A Unix socket will give you kernel-supplied credentials — measured, getpeereid(), LOCAL_PEERCRED and LOCAL_PEERPID all work — but they are uid and pid only, and a pid is not an identity: it can be reused, and by the time you look the process may be a different one.
On macOS use the documented requirement family, set before activating the connection: xpc_connection_set_peer_code_signing_requirement (macOS 12), or since macOS 14.4 …_entitlement_exists_requirement, …_entitlement_matches_value_requirement, …_team_identity_requirement, …_platform_identity_requirement and …_lightweight_code_requirement. The check then applies to every message: a listener drops non-conforming requests silently, and a pending reply gets XPC_ERROR_PEER_CODE_SIGNING_REQUIREMENT.
Verified four ways against a live service: a true requirement accepted, a false identifier rejected, anchor apple against an ad-hoc binary rejected, and a malformed requirement string rejected at set time with EINVAL — so a typo is a startup failure rather than a silently disabled check.
Follow-up: “Is that enough?” — no. It authenticates the caller; it does not validate the payload. Bound every length before allocating, allowlist rather than blocklist, and keep the interface narrow: “install the update at this verified path” is a boundary, “run this command” is not.
ChoosePipe, Unix socket, Mach port, XPC or shared memory — and what does the choice cost in latency?
Not much, between the low-level three. Measured cross-process: raw mach_msg 5.7 µs, pipe 5.0 µs, Unix socket 5.2–5.7 µs — inside 30%, so latency does not choose for you.
XPC is not in that band: 16.7 µs with the synchronous API, about 3.3× a pipe, because you are buying framing, a typed object model, launchd lifecycle, type checking and a one-call peer-identity check. Choose on semantics — just do not claim XPC is free. Against a 16,700 µs frame it is 0.1%, which is the number that actually settles the argument.
And do not use send_message_with_reply plus a semaphore: measured 23.0 µs, and the SDK header says to use the synchronous call instead precisely because the semaphore version gives up priority-inversion avoidance.
Follow-up: “When does the 3.3× matter?” — inside a hot loop, or synchronously on the main thread. Both are structural, and neither is fixed by changing mechanism.
DiagnoseEverything hangs under load. Neither process uses any CPU.
Zero CPU excludes a spin, so it is a blocked wait. sample both: if both are in write, it is mutual backpressure. The classic shape is a client that sends an entire request before reading any reply, while the peer streams its reply as it consumes — each fills the other’s buffer and stops reading.
Sizes make it worse than people expect: a socketpair holds 8,192 bytes and a pipe 65,536, so “large” is much smaller than it sounds. Reproduced deterministically with a 4 MiB document through a socketpair: both sides in write, killed by a watchdog.
Follow-up: “Fix it with a bigger buffer?” — no, that moves the threshold and turns it into a field-only bug. The rule is: never require the peer to make progress while you are not reading its output. One poll loop over both directions, or separate threads, or a request/reply framework that owns the draining — and a deadline on all of it.
ExplainHow do you know an XPC peer died, and does the difference matter?
XPC_ERROR_CONNECTION_INTERRUPTED means it crashed and launchd will restart it, so the connection object stays usable and reissuing is reasonable. XPC_ERROR_CONNECTION_INVALID means gone for good.
Observed: after the peer aborted the client got interrupted, and the same connection object then served a request answered by a different pid (77900 → 78680), with launchctl showing runs 1 → 2. After launchctl bootout, the same code got invalid.
Follow-up: “So a crash loop just retries forever?” — no, and this is the trap: launchd throttles respawn, with minimum runtime = 10. Measured, a service killed inside that window left the client’s next request waiting 10.05 s before it timed out; killed after 12 s of runtime, 0.05 s. A client deadline shorter than the throttle turns a successful restart into a reported failure.
DiagnoseTwo processes share memory and the shared counter is wrong. The code locks it.
Ask which lock. Measured on Darwin: sem_init(&s, pshared=1, 0) returns -1 with errno 78 ENOSYS — unnamed POSIX semaphores are not implemented, and sys/semaphore.h additionally marks the call deprecated. Code ported from Linux compiles, links, runs, and silently synchronises nothing if the return is unchecked.
What does work: a named sem_open, or pthread_mutexattr_setpshared(PTHREAD_PROCESS_SHARED) on a mutex living in the MAP_SHARED region itself. Measured exact in four runs of four: 400,000 of 400,000. The unlocked control lost between 14,216 and 200,000.
Follow-up: “Why is the bug so hard to reproduce?” — because the loss is variable, not fixed. A test that passes once proves nothing. Assert the exact total and run it repeatedly. And decide what happens when the lock’s holder is the process that just died — across a boundary, that is not hypothetical.
Drill
Answer first, then read the explanation.
One defensible first move each.
Scenario 01 · Corrupt only when large
Replies over a Unix socket are corrupt for big documents and fine for small ones. First measurement?
Scenario 02 · The silent disappearance
Your process ends with shell status 141 while writing to a helper that crashed. What does that tell you?
Scenario 03 · Choosing a mechanism
You need request/reply with a helper you ship, a few hundred times a second. What decides the choice?
Scenario 04 · The privileged helper
A helper runs with elevated rights and accepts requests. What is the first thing it must do?
Record your answer · 1
Justify one process boundary.
Pick a real or proposed helper. Write: which of containment, privilege or lifetime you are buying → the mechanism and why → what crosses and how it is bounded → what happens when the peer dies → how the protocol is versioned.
Record your answer · 2
Rehearse one IPC diagnosis.
For a cross-process failure you have seen, write the chain: symptom → transport, framing or semantics → the counter or status that decided it → the smallest fix → the test that would have caught it.
Primary sources for this chapter
Apple documentation and Darwin manual pages
Raw URLs and man invocations are printed beside each title.
| Source | URL or command (copyable) | Used in this chapter for |
|---|---|---|
| XPC | https://developer.apple.com/documentation/xpc | What XPC is; launchd managing lifecycle and restart; the three stated benefits; peer-to-peer connections; the two API levels |
| NSXPCConnection | https://developer.apple.com/documentation/foundation/nsxpcconnection | The high-level bidirectional channel and remote method dispatch |
| NSXPCInterface | https://developer.apple.com/documentation/foundation/nsxpcinterface | Declaring the protocol and whitelisting allowed classes per argument |
| NSXPCListener | https://developer.apple.com/documentation/foundation/nsxpclistener | The server side of a connection and accepting or rejecting a peer |
| Pipe and Process | https://developer.apple.com/documentation/foundation/pipe · .../process | The Foundation wrapper around launching a child and talking to it over pipes |
| DistributedNotificationCenter | https://developer.apple.com/documentation/foundation/distributednotificationcenter | System-wide broadcast, and what it does not promise |
| OSLog and OSSignposter | https://developer.apple.com/documentation/os/oslog · .../ossignposter | One interleaved timeline across two processes; separating transport time from handler time |
macOS 26.3 manual page pipe(2) | man 2 pipe | Anonymous pipe semantics and buffering |
macOS 26.3 manual pages socketpair(2), unix(4) | man 2 socketpair · man 4 unix | Stream versus datagram Unix sockets; SCM_RIGHTS descriptor passing; peer credentials |
macOS 26.3 manual page fcntl(2) | man 2 fcntl | F_SETNOSIGPIPE, quoted verbatim in section 5 |
macOS 26.3 manual pages mmap(2), shm_open(2) | man 2 mmap · man 2 shm_open | MAP_SHARED regions and named shared memory |
Darwin interfaces mach/mach.h, mach/message.h | $(xcrun --show-sdk-path)/usr/include/mach/message.h | Port rights, mach_msg, the trailer requirement, MACH_RCV_TOO_LARGE and MACH_SEND_INVALID_DEST |
macOS 26.2 SDK header xpc/connection.h | $(xcrun --show-sdk-path)/usr/include/xpc/connection.h | The six xpc_connection_set_peer_*_requirement functions and their availability; the enforcement, ENOTSUP and EINVAL text; XPC_ERROR_PEER_CODE_SIGNING_REQUIREMENT; and the instruction to use send_message_with_reply_sync rather than a reply plus a semaphore |
macOS 26.2 SDK headers sys/syslimits.h, sys/semaphore.h | $(xcrun --show-sdk-path)/usr/include/sys/syslimits.h · .../sys/semaphore.h | PIPE_BUF is 512 and writes at or below it are atomic; sem_init is marked deprecated |
| TN3127: Inside Code Signing: Requirements | https://developer.apple.com/documentation/technotes/tn3127-inside-code-signing-requirements | The requirement language used in section 4A. Title verified, body not retrieved — see the scope note |
| NSXPCConnection.setCodeSigningRequirement(_:) | https://developer.apple.com/documentation/foundation/nsxpcconnection/setcodesigningrequirement(_:) | The Foundation spelling of the same peer check, macOS 13+ |
Darwin manual page notify(3) | man 3 notify | The low-level notification API underneath distributed notifications, and its state-change-flag semantics |
How to read the labels on every claim in this chapter
documented — Apple documentation, an official WWDC or Tech Talk transcript, Swift Evolution, or a Darwin manual page or SDK header on macOS 26.3. Quoted verbatim with the source named beside it.
measured here — observed on one machine: Apple M4 Pro (10 performance + 4 efficiency cores), macOS 26.3 (25D125), 48 GB, 16 KiB pages, Swift 6.2.4, Apple clang 17.0.0, Xcode 26.3. Evidence of a mechanism, never a benchmark. Reproduce; do not quote.
inference — our reasoning joining the two above. Marked so you can disagree with the reasoning without doubting the evidence.
negative result — something that was tried and did not show what was expected. These bound what a chapter is allowed to claim, and they are kept deliberately visible.
correction — a correction to an earlier reading published on this page, naming what was wrong and why. This page’s whole method is labelled provenance, so its own errata carry a label too rather than disappearing into a rewrite.
version-sensitive — observed behaviour of this OS build that must be re-measured on another before it is relied on.
Scope of the evidence, stated plainly. Claim labels mean the same as in the previous chapters, and every measured here figure comes from small C programs on one machine — Apple M4 Pro, macOS 26.3 (25D125), 2026-09-23 — with 20,000 rounds per latency measurement and 256 MiB per throughput measurement, repeated, with the ordering stable across runs. Four of the five gaps this chapter used to declare are now closed. A real on-demand LaunchAgent was registered, measured, crashed, watched restarting and unregistered; Mach ports were measured across a genuine boundary through the bootstrap server; the peer code-signing requirement API was exercised four ways; and SCM_RIGHTS descriptor passing was demonstrated with the path unlinked before the send. What this chapter still does not have: the XPC work used ad-hoc linker-signed binaries in one user’s GUI domain — no Developer ID chain, no notarisation, no sandboxed helper, no SMAppService installation, no privileged LaunchDaemon, no real entitlements — so the requirement API’s enforcement is demonstrated and a production trust design is not; the listener-side silent drop is quoted from the SDK header and was observed only from the client side; the large-payload plateau around 9 GB/s is consistent with out-of-line memory but the crossover was never instrumented, so that explanation is labelled inference; os_unfair_lock across processes was deliberately not tested rather than guessed at; net.local.stream.sendspace is system-wide and needs root, so it was never raised and every SO_SNDBUF figure in section 3B is a property of the per-socket option only; and no network I/O was measured at all. On WWDC timestamps: this chapter cites no session, because no session transcript was verified for any claim it makes — its evidence is SDK headers, manual pages and measurement. That is a deliberate gap rather than an oversight; an unverified timestamp would be worse than none. The shared-memory throughput figure is a memcpy ceiling rather than a like-for-like protocol, and is labelled as such where it appears.
Interactive interview lab
Choose the first thing you would measure.
Answer before reading the explanation aloud. A senior answer starts with a discriminating observation, not a favorite fix.
Scenario 01 · One-second freeze
The menu opens, then the app stops responding for one second. What is your first move?
Scenario 02 · Large footprint
A control allocates a large image cache. Which observation tells you whether RAM pressure is real?
Scenario 03 · Load-only failure
A state update fails only under concurrent interaction. What evidence best tests the first hypothesis?
Scenario 04 · Helper failure
A privileged or crash-prone task sits beside a UI framework. When is XPC justified?
Record your answer
Make the causal chain explicit.
For one scenario, write: symptom → competing hypotheses → first instrument → expected evidence → safe change → regression guard.
Practical macOS bench
Run these experiments, then explain them.
Use a small AppKit sample or your existing control. Capture one screenshot or trace per experiment; the artifact is your interview evidence.
| Experiment | Do | What to say in the interview |
|---|---|---|
| Main-thread stall | Put a synchronous sleep or file read in a button action. Capture a hang sample, then move the work off the main thread and return UI updates to it. | Events are serviced by the main run loop; blocking it prevents input, layout, and display work. |
| Race and reentrancy | Update shared timeline state from two queues while a callback synchronously triggers another update. Run Thread Sanitizer, then serialize ownership. | A race and reentrancy are different failure modes; name the boundary that makes state ownership explicit. |
| Memory pressure | Render 500 large thumbnails, inspect Allocations and VM Tracker, bound the cache, and repeat the same scroll trace. | Allocations are not the same as resident or dirty physical memory; compare peak and steady state. |
| Priority inversion | Protect shared state with a lock held by background work while user-interactive work waits. Inspect thread states and QoS. | Fix ownership and dependency priority; increasing thread count is not a diagnosis. |
| XPC recovery | Sketch a helper protocol, kill the helper mid-request, and define timeout, cancellation, reconnection, and stale-result behavior. | Process isolation buys fault containment but creates an explicit protocol and failure model. |
Failure signatures
Translate symptoms into hypotheses.
Do not jump from symptom to fix. Pick the evidence source that can distinguish the plausible mechanisms.
| Symptom | OS hypotheses | Evidence |
|---|---|---|
| Clicks and keys stop responding. | Main thread computing, blocked, doing synchronous I/O, deadlocked, or waiting on lower-QoS work. | Hangs, Time Profiler, thread states, Thread Performance Checker, sample/spindump. |
| Scrolling or animation stutters. | Frame deadline missed from excessive invalidation, drawing, layout, commit work, GPU/compositor pressure, or scheduling delay. | Hitches, Core Animation, Time Profiler, signposts, repeated frame scenario. |
| Memory grows and does not fall. | Retain cycle, unbounded cache, abandoned objects, dirty backing memory, or large transient lifetime. | Allocations, Leaks, Memory Graph, VM Tracker, heap generations. |
| Bug appears only under load. | Race, ordering assumption, lock contention, lifetime error, or resource exhaustion. | Thread Sanitizer, Address Sanitizer, stress test, signposts, deterministic logging. |
| Framework works until OS update. | Private dependency, binary/behavioral compatibility assumption, symbol or availability misuse. | Deployment matrix, generated interface, symbol inspection, crash log, compatibility tests. |
| Helper stops responding. | XPC invalidation, helper crash, protocol mismatch, sandbox denial, or blocked service queue. | Connection handlers, unified logs, crash report, sandbox messages, timeout metrics. |
Seven-day map
Where each concept belongs.
Retrieve yesterday’s OS concept before beginning today’s UI work. This creates one connected model instead of seven definitions.
| Day | OS concept | Concrete proof |
|---|---|---|
| 1 | Process, main thread, run loop, event delivery. | Trace input and explain exactly why blocking freezes interaction. |
| 2 | Virtual memory, backing content, compositor, display deadline. | Draw the render path and minimize dirty-region work. |
| 3 | Main-thread confinement, races, reentrancy, synchronization, QoS. | Trigger Main Thread Checker; exercise shared state with Thread Sanitizer. |
| 4 | Mach-O, dyld, symbols, ABI, shared framework code. | Inspect dependencies and exports; name a binary-breaking change. |
| 5 | Scheduler, thread states, priority inversion, paging, memory pressure. | Use a targeted trace to distinguish CPU work, waiting, and memory churn. |
| 6 | In-process boundary versus XPC, launchd, sandbox, failure isolation. | Choose a process boundary and define cancellation/recovery behavior. |
| 7 | All concepts interleaved. | Pressure-test system design through execution, memory, IPC, graphics, and compatibility. |
Interview prompts
Practice causal answers.
For each prompt, state what you know, what remains uncertain, and the first measurement that would discriminate between hypotheses.
Run-loop stall
“A menu opens, but selecting an item freezes the app for one second. Walk me through the event path and diagnosis.”
Priority inversion
“The main thread waits on work submitted at background QoS. Why is that dangerous and how would you redesign it?”
Memory footprint
“A large custom control allocates substantial address space. Which measurements tell you whether it is consuming physical memory?”
Framework evolution
“You need to add behavior to a framework shipped with the OS. Which source, binary, and behavioral compatibility risks do you inspect?”
Process isolation
“When would you move framework-adjacent work into XPC, and what new API and reliability obligations appear?”
Rendering hitch
“Scrolling misses frames. Separate layout, drawing, commit, compositing, scheduling, and memory hypotheses.”
Appendix
Every link on this page, as plain text.
Each of the 75 external references used anywhere on this page, printed as a raw URL you can select and copy without following a link — useful in print, in a terminal, or when a link label is not enough to know where it goes. Local files that ship beside this page are listed at the end.
Apple Developer documentation (54)
https://developer.apple.com/documentation/dispatchhttps://developer.apple.com/documentation/dispatch/dispatch_barrier_asynchttps://developer.apple.com/documentation/dispatch/dispatchiohttps://developer.apple.com/documentation/dispatch/dispatchio/setlimit(highwater:)https://developer.apple.com/documentation/dispatch/dispatchobject/settarget(queue:)https://developer.apple.com/documentation/dispatch/dispatchqoshttps://developer.apple.com/documentation/dispatch/dispatchqueuehttps://developer.apple.com/documentation/dispatch/dispatchsemaphorehttps://developer.apple.com/documentation/dispatch/dispatchsourcehttps://developer.apple.com/documentation/foundation/data/readingoptionshttps://developer.apple.com/documentation/foundation/distributednotificationcenterhttps://developer.apple.com/documentation/foundation/filehandlehttps://developer.apple.com/documentation/foundation/filehandle/readtoend()https://developer.apple.com/documentation/foundation/filemanagerhttps://developer.apple.com/documentation/foundation/nsautoreleasepoolhttps://developer.apple.com/documentation/foundation/nscachehttps://developer.apple.com/documentation/foundation/nsconditionhttps://developer.apple.com/documentation/foundation/nslockhttps://developer.apple.com/documentation/foundation/nsrecursivelockhttps://developer.apple.com/documentation/foundation/nsxpcconnectionhttps://developer.apple.com/documentation/foundation/nsxpcconnection/setcodesigningrequirement(_:)https://developer.apple.com/documentation/foundation/nsxpcinterfacehttps://developer.apple.com/documentation/foundation/nsxpclistenerhttps://developer.apple.com/documentation/foundation/operationqueue/maxconcurrentoperationcounthttps://developer.apple.com/documentation/foundation/pipehttps://developer.apple.com/documentation/foundation/processhttps://developer.apple.com/documentation/foundation/processinfo/activeprocessorcounthttps://developer.apple.com/documentation/foundation/thread/qualityofservicehttps://developer.apple.com/documentation/os/os_unfair_lock_assert_ownerhttps://developer.apple.com/documentation/os/os_unfair_lock_lockhttps://developer.apple.com/documentation/os/os_unfair_lock_trylockhttps://developer.apple.com/documentation/os/osallocatedunfairlockhttps://developer.apple.com/documentation/os/osloghttps://developer.apple.com/documentation/os/ossignposterhttps://developer.apple.com/documentation/swift/actor/assumeisolated(_:file:line:)https://developer.apple.com/documentation/swift/serialexecutorhttps://developer.apple.com/documentation/swift/taskhttps://developer.apple.com/documentation/swift/taskexecutorhttps://developer.apple.com/documentation/swift/taskpriorityhttps://developer.apple.com/documentation/synchronization/atomichttps://developer.apple.com/documentation/synchronization/atomiclazyreferencehttps://developer.apple.com/documentation/synchronization/atomicloadorderinghttps://developer.apple.com/documentation/synchronization/atomicstoreorderinghttps://developer.apple.com/documentation/synchronization/atomicupdateorderinghttps://developer.apple.com/documentation/synchronization/mutexhttps://developer.apple.com/documentation/synchronization/wordpairhttps://developer.apple.com/documentation/xcode/diagnosing-memory-thread-and-crash-issues-earlyhttps://developer.apple.com/documentation/technotes/tn3127-inside-code-signing-requirementshttps://developer.apple.com/documentation/xcode/diagnosing-performance-issues-earlyhttps://developer.apple.com/documentation/xcode/gathering-information-about-memory-usehttps://developer.apple.com/documentation/xcode/improving-app-responsivenesshttps://developer.apple.com/documentation/xcode/reducing-your-app-s-memory-usehttps://developer.apple.com/documentation/xcode/understanding-hangs-in-your-apphttps://developer.apple.com/documentation/xpc
WWDC sessions and Tech Talks (15)
https://developer.apple.com/videos/play/tech-talks/110147/https://developer.apple.com/videos/play/wwdc2017/706/https://developer.apple.com/videos/play/wwdc2018/416/https://developer.apple.com/videos/play/wwdc2021/10133/https://developer.apple.com/videos/play/wwdc2021/10180/https://developer.apple.com/videos/play/wwdc2021/10254/https://developer.apple.com/videos/play/wwdc2022/110350/https://developer.apple.com/videos/play/wwdc2022/110351/https://developer.apple.com/videos/play/wwdc2023/10170/https://developer.apple.com/videos/play/wwdc2023/10248/https://developer.apple.com/videos/play/wwdc2024/10169/https://developer.apple.com/videos/play/wwdc2024/10173/https://developer.apple.com/videos/play/wwdc2025/226/https://developer.apple.com/videos/play/wwdc2025/268/https://developer.apple.com/videos/play/wwdc2025/308/
The Swift Programming Language (1)
https://docs.swift.org/swift-book/documentation/the-swift-programming-language/concurrency/
Swift Evolution proposals (3)
https://github.com/swiftlang/swift-evolution/blob/main/proposals/0306-actors.mdhttps://github.com/swiftlang/swift-evolution/blob/main/proposals/0410-atomics.mdhttps://github.com/swiftlang/swift-evolution/blob/main/proposals/0433-mutex.md
Apple archived documentation (2)
https://developer.apple.com/library/archive/documentation/Cocoa/Conceptual/Multithreading/RunLoopManagement/RunLoopManagement.htmlhttps://developer.apple.com/library/archive/documentation/MacOSX/Conceptual/BPFrameworks/Concepts/FrameworkBinding.html
Local files beside this page
labs/threads-locks-exercises.tar.gz— threads, concurrency and locks exerciseslabs/threads-locks-exercises/— the same bundle, unpackedlabs/os-memory-io-ipc-exercises.tar.gz— heap, scheduling, I/O and IPC exerciseslabs/os-memory-io-ipc-exercises/— the same bundle, unpackedlabs/os-workbench.html— the live OS workbench
Manual pages referenced in the chapters above are not URLs: read them on the machine you are sitting at, with man 2 fsync, man 2 fcntl, man 3 malloc, man 3 malloc_zone_malloc, man 2 mmap, man 2 rename, man 4 unix, man 1 leaks, man 1 vmmap and man 2 getrusage. SDK headers are under $(xcrun --show-sdk-path)/usr/include.
Primary sources
Apple operating-system reading set
Read only the source paired with that day. The reference is a map; Apple documentation remains the authority.