# Exercise 04 — The meter that judders while the mean stays perfect
**Failure mode:** undeclared quality of service — background work competing in the foreground band
**Language:** C, pthreads · **Runtime:** about 8 seconds · **Difficulty:** the one that survives profiling
---
## The prompt
> A media app refreshes a level meter on a latency-sensitive thread: a small, fixed unit of
> work that must finish before the next frame is due. A library re-indexes the user's
> collection on a pool of CPU-bound threads.
>
> QA reports: "the meter judders while a re-index runs. But we profiled it — the meter's
> **average** frame cost is completely normal, nothing is blocked, and there is no lock
> anywhere near it."
>
> No thread ever enters the blocked state. Find the defect, fix it, and tell me exactly
> what your fix costs.
Answer before you run anything: **if nothing is blocked and the average is fine, what is
left that could make a thread late?**
---
## What you are given
| Path | What it is |
| --- | --- |
| `broken/heartbeat.c` | The starting point. The pool declares no QoS. |
| `fixed/heartbeat.c` | One correct repair. |
| `solution.patch` | The diff between the two, applies with `patch -p1`. |
| `check.sh` | Asserts the tail inflation in both builds. |
The fixture measures itself **twice** — once with the pool idle to establish this machine's
floor, and once with the pool running — so every figure it reports is a ratio against its
own baseline rather than a number from somebody else's laptop.
---
## Reproduce the measurement
```bash
cd exercises/04-qos-latency
clang -O2 -g -Wall -Wextra -pthread broken/heartbeat.c -o /tmp/hb_broken
clang -O2 -g -Wall -Wextra -pthread fixed/heartbeat.c -o /tmp/hb_fixed
/tmp/hb_broken
/tmp/hb_fixed
```
Observed on an Apple M4 Pro (10P + 4E), macOS 26.3 (25D125), 56 pool threads on 14 cores:
```
broken (unspecified QoS) fixed (background QoS)
idleP99Us 511 537
loadedP50Us 471 443
loadedP99Us 40,610 1,431
p50Inflation 1.02 1.04 <- the mean lies
p99Inflation 79.47 2.66 <- the whole bug
maxInflation 100.32 5.37
hogBatches 35,540 1,554 <- what it costs
```
**Reproduce these; do not quote them.**
---
## The evidence to collect
1. **`p50Inflation = 1.02`.** The median work unit is 2% slower under load. A profiler
reporting means, an average frame time, or a "total time in function" figure sees
nothing at all. This is why the team's profiling exonerated the meter.
2. **`p99Inflation = 79.47`.** One work unit in a hundred takes eighty times as long. At
60 Hz that is dropped frames roughly once a second — precisely "juddering".
3. **Nothing is blocked.** Read this alongside exercise 03: a thread can be late without
ever waiting for anything. It was **runnable and not running**, which is a third state
that many mental models collapse into "running". The kernel distinguishes them
(`TH_STATE_RUNNING` versus a thread simply not being on a core), and so should your
vocabulary.
4. **`hogBatches` falls from 35,540 to 1,554.** The repair is a *trade*, not a free win, and
the fixture prints the price so you cannot report the win without it. The re-index now
takes roughly twenty times as long in wall-clock terms while the user is doing something
else. That is usually correct — and it is a decision, not an optimisation.
5. **The band ladder is visible if you look.** Re-running the probe with the pool at
`QOS_CLASS_UTILITY` instead lands between the two: p99 inflation around 3x rather than
79x or 2.7x. QoS is not a two-position switch.
6. **In Instruments**, **System Trace** shows this directly: the meter thread's lane has
gaps where it is neither running nor blocked, and the same interval shows 56 other
threads running. That gap — runnable, not running, not waiting — is the picture of this
defect.
---
## Success criteria
- [ ] The broken build still blows out the tail — you have not edited it.
- [ ] Your fixed build's `p99Inflation` is at most **8.0**.
- [ ] Your fixed build's `p50Inflation` is still around 1, and you can explain why that is
*expected* rather than a sign the fix did nothing.
- [ ] You set the class **on each pool thread**, not on the thread that created them, and
you can say why that distinction exists.
- [ ] You can state what the fix costs in background throughput, with the number.
Run `./check.sh` to have all of that checked for you.
---
## Hints
Hint 1 — count the threads
`hogThreads=56` on a 14-core machine. Every one of them is CPU-bound and none of them has
said anything about how important it is. What does the scheduler have to go on?
Hint 2 — the meter already asked
`main` calls `pthread_set_qos_class_self_np(QOS_CLASS_USER_INTERACTIVE, 0)` before it
measures anything. Asking for the top class did not help. Why would it? A priority is only
meaningful relative to what else is in the queue.
Hint 3 — QoS is a property of a thread, not of a process
Setting a class affects the calling thread only. A thread created by a thread that set a
class does not inherit it through `pthread_create`. Where is the earliest point in each
pool thread's life at which it could declare itself?
---
## Solution
Reveal the solution
### The repair
```c
static void *hog(void *unused) {
(void)unused;
pthread_set_qos_class_self_np(QOS_CLASS_BACKGROUND, 0); /* first act */
...
}
```
One line, on each pool thread, as its first act.
### What that line actually does
It is **not** a speed dial and it does not slow the pool's instructions down. It is a
**placement** decision: it tells the scheduler which band this work belongs in, and on
Apple silicon that influences both which cores it is eligible for and how it is ordered
against other runnable work. The interactive thread stops taking turns against 56 peers.
The five ordinary classes, from `sys/qos.h`, are `USER_INTERACTIVE` (0x21),
`USER_INITIATED` (0x19), `DEFAULT` (0x15), `UTILITY` (0x11) and `BACKGROUND` (0x09). Apple
documents `BACKGROUND` as work "not initiated by the user and that the user may be unaware
of the results", to be run "in the most energy and thermally-efficient manner" — which is
exactly a library re-index, and exactly why the pool was in the wrong band.
### What the fix costs, stated plainly
`hogBatches` drops by roughly 20x. The re-index is now much slower whenever the user is
doing something else. If the re-index has a deadline of its own — a sync that must complete
before the app can be used — then demoting it is the wrong repair and the right repair is
to make the re-index smaller or interruptible. **Say which one you are choosing and why.**
### Why the median was useless
The interactive thread usually got a core promptly; the median never moved. Every defect of
this shape lives in the tail, and tail latency is not visible in any mean, any total, or
any "time spent in function" column. When a user says "judder", "hitch", "stutter" or
"occasionally", the first thing to fix is the *measurement*: percentiles, or a histogram,
or a worst-case, before anything else.
### The repairs that do not work
- **Raising the meter to `USER_INTERACTIVE`.** Already done in both builds. It is necessary
and not sufficient.
- **Fewer pool threads.** Helps, and hides the lesson: with 14 threads instead of 56 the
tail is smaller but the pool still competes in the same band. The defect is the band, not
the count. (Apple's own guidance is to size pools to the core count and explicitly *not*
to scale them with workload.)
- **`setpriority(2)` / `nice`.** Process-wide and coarse; it does not express what this
work is *for*, and on a system that schedules by QoS band it is the wrong vocabulary.
---
## Going further
- Change `QOS_CLASS_BACKGROUND` to `QOS_CLASS_UTILITY` and re-measure. You should land
between the two published figures. Being able to say "utility was not enough, and here is
the number" is a much stronger answer than "I set it to background".
- Reduce `HOG_MULT` from 4 to 1 and re-measure the broken build. The tail shrinks. Explain
why that does not mean the defect is fixed.