ET-SoC-1 · experiment plan · rung 9 of the observability ladder
The P3 cannot resolve a shire, but it can show which package the watts warm: the part of the card's power that no meter sees. The workloads are under exact software control, so lock-in averaging turns a 35 mK camera into a 0.2 mK one. That is enough to tell compute heat, DRAM heat and mesh heat apart.
The back is one surface with one emissivity (solder mask, 0.90–0.95), no tape needed, and shows the SoC and all four DRAM footprints in a single frame.
But the regulators and the DRAM chips are on the component side, so start with the whole card from the component side at 250 mm. In these open frames the back probably faces the CPU cooler 40–70 mm away; check before buying an arm.
A pipeline stage hands its output through DRAM, the next shire's scratchpad, or its own: 4.34, 4.42 and 5.30 W while moving 48, 593 and 1,484 GB/s.
If heat follows the data, the three images differ. It needs no new code, about 9 minutes of card time, and it shows in the phone app.
The P3 has no tripod thread, so it needs a cradle; Studio 45's laser cutter makes one in 20 minutes.
Raw 16-bit frames come through an open Linux driver (not macOS), so a Pi 5 on the tailnet records them without touching the shared lab hosts.
That is why a camera that cannot see a shire can still test the theory. Pick a load to see which parts the repo's measurements predict will warm.
+27.1 W · fp32 matmul, random data, 1,024 minions
The die rises 5.2 °C in 7 s. The regulator's three minion phases should lose about 5 W more (2.2 → 7.3 W, from the repo's fitted 19.6% loss share). The DRAM should stay flat. The same instructions on zeros add only 2.0 W.
Tested by E-T4 · E-T10
| Load | Power | Where the heat should go | Tested by |
|---|---|---|---|
| Idle | 31.8 W at 73 °C (aifoundry2) | Leakage is most of it: 23 of 36 W at 80 °C. 15.1 W reaches no metered rail — DDR, PCIe, IO and regulator losses. Which bodies sit above board ambient is the first thing the camera ranks. | E-T1 |
| Compute | +27.1 W · fp32 matmul, random data, 1,024 minions | The die rises 5.2 °C in 7 s. The regulator's three minion phases should lose about 5 W more (2.2 → 7.3 W, from the repo's fitted 19.6% loss share). The DRAM should stay flat. The same instructions on zeros add only 2.0 W. | E-T4 · E-T10 |
| DRAM stream | +9.8 W · tensor loads from DRAM · 6.3 W on no metered rail | The board's power tree budgets all four DRAM chips at about 3.2 W. So at least 2–3 W of the 6.3 W must be elsewhere: in the die's own DDR controllers (VDD_DDR, up to 4.4 W, under the heatsink at the die's memory edges) and in conversion losses. One 30 s burst on day one shows the split. | E-T2b · E-T3 |
| Scratchpad stream | +10.5 W · tensor loads from the shire's own scratchpad | 67% of the extra power lands on the SRAM rail, so the SRAM regulator should light up and the DRAM packages should not. It burns more power than the DRAM stream and puts none of it in the packages. | E-T3 · E-T11 |
| Mesh | +16.5 W · scratchpad reads 6 hops away · 49% on the mesh rail | The single NoC phase should take +2.3 W of conversion loss while each minion phase takes +0.09 W, a 25× contrast. A matmul does the reverse. Which physical phase is the NoC one is not yet known. | E-T5 · E-T15 |
| Target | Distance | What it shows | Emissivity prep | |
|---|---|---|---|---|
| 1 | Whole card, component side | 250 mm | Every package in one frame, ranked against board ambient: the census | none on mask or mould; tape metal |
| 2 | U15 minion + NoC regulator | 100–130 mm | The largest unmetered body: 2.2 → 7.3 W of loss; mesh vs matmul flips which phase heats | polyimide on the module top |
| 3 | Back of the PCB, over the package | 150–250 mm | The permanent lock-in view: one emissivity, every footprint | none (solder mask ε 0.90–0.95) |
| 4 | DDR / IO / Maxion buck band | 130 mm | The VDD_DDR buck's ~12% loss should answer a DRAM burst and not a matmul | polyimide if metal-topped |
| 5 | The four DRAM packages | 120–150 mm | The watts no meter sees — but budgeted ≈ 3.2 W for all four | none (black mould) |
| 6 | Heatsink base + fin tips | 150–250 mm | A physical owner for the 60–400 s thermal stages | polyimide patch — bare aluminium is a mirror |
| 7 | Whole rig: card, cooler, neighbour | 400 mm | Inlet air, exhaust plume, the airflow changes | matte-black ambient tab |
What fits in one frame (40° × 30° lens)
Rules for every view
Predicted signal ranges (bars) against the noise floor at the planned recording length (tick). The ranges come from the fitted thermal chain, measured workload watts and the board's power tree. Where the bracket is wide, the experiment exists to narrow it.
In the usable band, heat blurs 6–20 mm across the board: DRAM pairs separate, shires don't
Noise falls as 1/√time: 30 minutes of frames buys 0.23 mK per pixel
Per quadrature, per pixel. The shutter fires every ~90 s and can't be disabled, so every fit carries an offset and a slope per shutter segment. Plan with 4× this until E-T8 measures the real curve: half a pixel of mount wobble on a steep image gradient costs 70–170 mK per frame.
| Frequency | 0.02 Hz | 0.05 | 0.1 | 0.25 | 0.9 |
|---|---|---|---|---|---|
| Amplitude reaching the back, |sech((1+i)L/μ)| | 0.91 | 0.66 | 0.39 | 0.14 | 0.01 |
| Phase lag | 30° | 63° | 96° | 151° | — |
One-dimensional, back face treated as adiabatic for the AC wave; real transmission sits between this and e−L/μ. E-T9 measures it.
Tests the core theory: computation and memory transfers put heat in different places.
About 34 attended hours in six phases, plus 13.5 unattended
The whole-degree sensor at 10 Hz still averages to about 3 mK in 30 minutes. Random data on 1,024 minions, 2 s on and 22 s off (2.5 W average, inside the 3 W budget), swings the die 692 mK. That gives an absolute anchor for every later camera number.
Run predict_heat.py for each planned burst and timestamp the output before the session. The repo already does this for the flip model, and it catches power-match errors before any card time is spent.
Which bodies sit above board ambient at idle? That ranking is the first proportional split of the 15.1 W that no rail meters.
Of the 6.3 W that a DRAM stream puts on no metered rail, how much warms the four chips, how much the die's memory edges, how much the regulators?
timeout 10 enercat_host --pattern tload --operands random --harts 1 --slice-bytes 256K --seconds 8 --budget 9, repeated for 30 s; image at 250 mm, then 150 mm.A pipeline hands data to the next stage through DRAM, the next shire's scratchpad, or its own: 4.34, 4.42 and 5.30 W while moving 48, 593 and 1,484 GB/s. Does the heat follow the data?
tools/ettelem/run_onchip_power.sh /tmp/ir-relay-N 30 — one out-dir per repeat (it truncates its telemetry), change its timeout 40 to timeout 10, and stop any other sampler first: the script starts its own.dram warms the DRAM side and barely the die. scp warms the die and the SRAM regulator. hop warms the die and the NoC phase. The die-side differences are only 0.15–0.4 °C, marginal in the app; the package side is the discriminator.Random operands cost 9.84 W, zeros 7.09 W, for identical bytes: 1.46 W of the gap is off-rail. The scratchpad stream costs 10.51 W, two-thirds on the SRAM rail. Which packages move?
--operands random then zeros. The scratchpad control needs a prefill first — enercat_host --pattern tstore --operands <o> --slice-bytes 32K --scp — because enercat skips the operand fill for --scp.die_mv.ddr) should read about 5.6 and 3.9 mV — the one on-card sensor that answers to DRAM traffic alone.Random fp32 data adds 27.1 W, zeros 2.0 W, at the same instruction rate. Negative zeros are not gated and cost like ones (+10.6 W). Does the regulator follow the data?
timeout 10 sparsity_host --test fma --type fp32 --pattern none --values randn --shires 0xffffffff --per-shire 32 --seconds 7 --budget 9 --seed 1, then --values zeros, then --values file: tiles from make_tiles.py negzero. Start from 80 °C by the strict protocol.A mesh load should heat one regulator phase by +2.3 W and each minion phase by +0.09 W; a matmul does the reverse. The crossover names the phase and tests the repo's fitted 19.6% and 28.6% loss coefficients.
wire/hop6/random line verbatim: --pattern tload_pat --operands random --hop-distance 6 --slice-bytes 32K --stride 1K --access-bytes 1K --region 32K, after a tstore --scp prefill. Minion arm: the E-T4 line.PA[8:6] picks the memory shire; four memory shires sit on each side of the die. Force PA[8] and one DRAM pair should warm, 82 mm from the other — or none will, which means the interleave is finer.
memprobe_host with the dram_row parameters (--level 4 --stride 262144 --lines 1024, from gen_ops.py table) and PA[8] forced; snapshot the eight memory-shire counters with msmap in the same session.Mid-session the lab's airflow once dropped the die 12 °C at constant power. One wide frame of card, cooler and neighbour at 400 mm, at the start and end of every session, catches that.
Does this unit's noise fall as 1/√frames down to 0.4 mK in 10 minutes, or plateau? How often does the shutter fire, and does a manual fire reset its timer? Where are the fan lines in the room's spectrum?
The fitted chain has stages at 1.5, 4, 60, 150, 400 and 2,500 s. Which object is each — lid, heatsink base, fins, air? And aifoundry3's chain has never been fitted: this sets its power budget.
--stop-file and an 85 °C watchdog until the cap or 10 min, 15 min cool; three steps. Needs your explicit waiver of the 10 s rule.Where do compute watts land and where do memory watts land, in °C per watt, pixel by pixel?
sparsity_host --test fma --type fp32 --pattern none --values randn --shires 0xffffffff --per-shire 8 --seconds 5 --budget 6 --seed 1 (256 minions, 6.8 W, 1.6 W average). Memory at 1/28 Hz, 10 s on: the E-T2b enercat line with --seconds 10 --budget 11. 5 min unrecorded, then 30 min recorded each, in an unmoved mount.Eight operand patterns span 1.9 → 27.1 W of switching at identical instructions, clock and voltage. Fit each ROI's rise against watts plus the leakage feedback — the result is a per-pixel °C/W map.
run_horace_strict.sh <out> 80 84 4 3 1 7 "zeros ones pi signs mant uniform randn sparse50" "" on aifoundry2, and the same ladder on aifoundry3.predict_heat.py's predicted superlinearity, not a straight line: the existing ladder already has a −0.44 °C intercept from leakage feedback. The DRAM should stay flat — and that flatness is the result.Alternate A and B at 0.025 Hz with total power held constant: pure relocation shows at f, the common mode at 2f. Pairs: DRAM vs scratchpad stream; int8 matmul (9.98 W) vs DRAM stream (9.84 W); and the null — int8 on 1,024 minions vs fp32 on ~384 (same watts, same place).
Run 8 shires in the north-west corner against 8 in the south-east, 6.8 W each, centroids 14–18 mm apart, and look for the heat moving on the back of the PCB.
--shires 0x0101031b vs 0xc4c080c0. Capture the SP's per-shire IR-drop voltage map during both halves: the voltage map says where the current went, the camera says where the heat went.Ones on 1,024 minions reached 90 °C in 107 s after a hot predecessor and in 162–167 s after a zeros run, from the same 80 °C reading. The missing state variable is the heatsink; one ROI in starts.jsonl measures it.
A hop costs 131 fJ per bit, and a 6-hop read puts 49% of its power on the mesh rail. Does ~6 W moving from the SRAM arrays into the mesh show as a band across the die?
Rings between shires s and s+16 stall the service processor and make bursts read 45% low. The camera doesn't go through the SP. Part B: the contended hot line (+1.41 W) should show no hot spot.
An m-sequence of 4 s chips gives every pixel's impulse response, and the low-frequency rise of Z(f) measures how close the die is to thermal runaway without ever running away.
During a climb driven by one masked corner, the whole die should brighten while only the corner switches, so the corner's contrast should shrink as the die heats.
aifoundry3 sheds heat visibly faster. Same strict protocol, camera on each heatsink in turn; the slowest stage as an overnight idle equilibrium.
| Item | Why | Example | ~Price | |
|---|---|---|---|---|
| Super clamp, 10–55 mm jaws | The rigid anchor; 10 mm closes on a ~12.7 mm shelf tube | SmallRig 2058 | $25–40 | must |
| Articulating arm ≤ 140 mm | Short arms resonate less on a shelf full of fans | SmallRig 2065 (5.5 in) | $30–45 | must |
| Mini ball head, 1/4-20 | Fine aim within 20–30° of normal | any | $15–25 | must |
| Cradle with a cable clamp | The P3 has no tripod thread; the USB-C plug must never carry load | laser-cut at Studio 45, or the Printables “Thermal Master P3 Holder” | $0–10 | must |
| Magnetic base, 1/4-20 | Fallback anchor on a steel PSU case | any | $15–25 | must |
| USB 2.0 extension, 2 m A-to-A | After the included 0.5 m C extension and A adapter; longer passive C-to-C is out of spec | Anker, Cable Matters | $10–15 | must |
| Polyimide tape, 1 mil total | ε ≈ 0.9 on metal, insulating, 260 °C, removable | generic 1 mil; 3M 5413 is 2.7 mil (≈ 6 K offset) | $8–12 | must |
| Contact thermometer (K-type or DS18B20) | Solves ε and reflected temperature instead of guessing them | any | $15–30 | must |
| Matte-black foam board, zip ties, gaffer tape, paint pen | Ambient tab, background, strain relief, witness marks | craft store | ~$25 | must |
| First-surface mirror, 50 mm, Al or Au | > 95% at 8–14 µm — the fallback if the back of the PCB faces a CPU cooler. Household glass mirrors are opaque in LWIR | Thorlabs / Edmund | $25–40 | if needed |
| Raspberry Pi 5 + 1 TB USB SSD | Raw 16-bit frames need Linux; your box on the tailnet, 8.9 GB/h off the lab disk | Pi 5 4 GB | ~$160 | strongly |
| Hot-wire anemometer | Puts a number on the airflow changes | any | $25–35 | nice |
Budget version (~$50): a gooseneck clamp phone holder, polyimide, foam board, the USB path and gaffer tape. That covers day one and none of week one. Don't order SmallRig 2064 (an EVF bracket, not a clamp), the Manfrotto 244MINI (240 mm, too long), 3M Super 33+ vinyl (105 °C, leaves residue) or a PCIe riser.
starts.jsonl + ettelem at 10 Hz on aifoundry2/3board_w, not the commanded square wave. A clock offset then only shifts every pixel's phase equally.Tested here on synthetic data: 1,800 s at 2 Hz, shutter segments every 90 s with random offsets and slopes, quadratic drift, NETD noise. A true 3 mK channel came back as 3.07 ± 0.23 mK, σ on theory. Dropping the per-segment slope inflates the noise and lets drift leak into null pixels. Download lockin.py.
"""Lock-in / PRBS demodulation of a Thermal Master P3 raw stack, joined to ettelem telemetry.
temp : float32 (M,192,256) degC = thermal_u16/64.0 - 273.15 (rows 194..385 only),
already registered and boxcar-decimated to ~2 Hz
t : float64 (M,) host monotonic seconds -- REAL timestamps, never frame indices
seg : int32 (M,) increments at every detected FFC/shutter event
good : bool (M,) False for shutter epochs (+2 s) and cnt3 gaps -- drop, never interpolate
extra: (M,k) covariates: the REALIZED launch indicator or board_w, the residual board_w
amplitude term, camera body temperature, the ambient tab ROI
"""
import numpy as np
def design(t, freqs, harmonics=3, seg=None, seg_order=1, drift_order=2, extra=None, res_hz=None):
"""[per-FFC-segment offset AND slope | covariates | cos/sin per (freq, harmonic)].
With seg given, NO global drift polynomial: the per-segment piecewise-linear basis
already spans every global affine function, so adding drift^1 makes X exactly
rank-deficient (cond ~1e16) and np.linalg.inv returns garbage without raising.
seg_order >= 1 is mandatory: without the slope, sigma inflates 5x and a null pixel
reads 2.7 mK against a true 3 mK channel.
"""
cols, names = [], []
t = np.asarray(t, float)
if seg is None:
u = 2 * (t - t.mean()) / max(np.ptp(t), 1e-9)
for p in range(drift_order + 1):
cols.append(u ** p); names.append(f"drift^{p}")
else:
for k in np.unique(seg):
m = seg == k
tc = np.zeros_like(t); tc[m] = t[m] - t[m].mean()
sc = max(np.ptp(t[m]), 1e-9)
for q in range(seg_order + 1):
c = np.zeros_like(t); c[m] = (tc[m] / sc) ** q
cols.append(c); names.append(f"seg{k}^{q}")
if extra is not None:
for j, col in enumerate(np.atleast_2d(np.asarray(extra, float).T)):
cols.append(col - col.mean()); names.append(f"extra{j}")
tones, res = {}, res_hz if res_hz else 1.0 / max(np.ptp(t), 1e-9)
for f in freqs:
for h in range(1, harmonics + 1):
fh = f * h
for g in tones: # key by (f,h), never by formatted name
if abs(g - fh) < 2 * res:
raise ValueError(f"tone collision: {fh:g} Hz vs {g:g} Hz (res {res:.2g} Hz)")
tones[fh] = len(cols)
w = 2 * np.pi * fh
cols += [np.cos(w * t), np.sin(w * t)]
names += [f"cos{fh:.6g}", f"sin{fh:.6g}"]
return np.column_stack(cols), names, tones
def demodulate(temp, t, freqs, good=None, seg=None, ref_mask=None, harmonics=3,
seg_order=1, drift_order=2, extra=None, block=512, cond_max=1e8):
"""Per-pixel LSQ lock-in at several frequencies, chunked over pixels.
Returns {freq: dict(amp[K], phase[deg lag], sigma, snr)} plus '_cond'."""
if good is None: good = np.ones(len(t), bool)
tg = t[good]; sg = None if seg is None else seg[good]
eg = None if extra is None else np.asarray(extra)[good]
X, names, tones = design(tg, freqs, harmonics, sg, seg_order, drift_order, eg)
s = np.linalg.svd(X, compute_uv=False)
cond = s[0] / s[-1]
assert cond < cond_max, f"design matrix ill-conditioned: cond={cond:.3g}"
XtXi = np.linalg.inv(X.T @ X) # safe only after the cond assertion
P = np.linalg.pinv(X)
H, W = temp.shape[1], temp.shape[2]
npx, dof = H * W, len(tg) - X.shape[1]
beta = np.empty((X.shape[1], npx)); sigma = np.empty(npx)
ref = None
if ref_mask is not None: # common mode: unpowered taped patch
ref = temp[good][:, ref_mask].reshape(len(tg), -1).mean(axis=1)
flat = temp[good].reshape(len(tg), npx)
for i in range(0, npx, block): # 512 px x 3600 frames = 15 MB, Pi-safe
Y = flat[:, i:i + block].astype(np.float64)
if ref is not None: Y -= ref[:, None]
b = P @ Y; r = Y - X @ b
beta[:, i:i + block] = b
sigma[i:i + block] = np.sqrt((r * r).sum(0) / dof)
out = {"_cond": cond}
for fh, i in tones.items():
c, sn = beta[i], beta[i + 1]
sig = sigma * np.sqrt(0.5 * (XtXi[i, i] + XtXi[i + 1, i + 1]))
amp = np.hypot(c, sn)
out[fh] = dict(amp=amp.reshape(H, W),
phase=np.degrees(np.arctan2(-sn, c)).reshape(H, W),
sigma=sig.reshape(H, W),
snr=(amp / np.maximum(sig, 1e-12)).reshape(H, W))
return out
def mseq(n, taps):
"""Maximal-length LFSR, +-1, L = 2**n-1. n=7 taps=[6]; n=9 [5]; n=10 [7].
Periodic autocorrelation is exactly L at lag 0 and -1 elsewhere; check s.sum()==1."""
mask = L = (1 << n) - 1; reg, out = 1, []
for _ in range(L):
out.append(1.0 if (reg & 1) else -1.0)
fb = reg & 1
for tp in taps: fb ^= (reg >> (n - tp)) & 1
reg = ((reg >> 1) | (fb << (n - 1))) & mask
return np.array(out)
def prbs_impulse(temp, t, seq, chip, t0, good=None, detrend=2):
"""Per-pixel thermal impulse response by periodic cross-correlation.
Per-lag noise ~ NETD/sqrt(M) -- sqrt(2) better than single-frequency lock-in, and
L lags instead of one amplitude. cumsum(h) is the step response; fit Foster taus to
that. DC gain is lost with the mean -- take it from E-T13."""
if good is None: good = np.ones(len(t), bool)
L = len(seq); period = L * chip
tg = t[good]
b = np.floor(((tg - t0) % period) / chip).astype(int)
Y = temp[good].reshape(len(tg), -1).astype(np.float64)
V = np.vander(tg - tg.mean(), detrend + 1)
Y = Y - V @ np.linalg.lstsq(V, Y, rcond=None)[0]
acc = np.zeros((L, Y.shape[1])); np.add.at(acc, b, Y)
cnt = np.bincount(b, minlength=L).astype(float)
y = acc / np.maximum(cnt, 1)[:, None]
y = y - y.mean(0, keepdims=True)
S = np.fft.rfft(seq)
h = np.fft.irfft(np.fft.rfft(y, axis=0) * np.conj(S)[:, None], n=L, axis=0) / (L + 1)
return h.reshape(L, *temp.shape[1:])
mhz.minion left 600.sparsity_host polls --stop-file; for the others the guard is "don't launch the next burst", so keep ~3 °C of margin.ettelem; a second sampler makes them fail. Quit et-powertop and et-power-log.sh first and restart them after. Some mesh workloads starve the meter (sampler latency 22 → 150 ms, bursts read 45% low); watch took_ms.uptime, ps and the driver's open count; nothing holds the card over 10 s without your explicit waiver (E-T13, E-T17). The lock-in runs are 30–90 min windows: book them with Nate and note them on the rig.Five research agents covered the camera, the board and die floorplan, the repo's theories and tools, lock-in technique, and mounting. Three designers wrote independent plans (hypothesis-first, instrument-first, pragmatic), an editor merged them, and three adversarial reviewers checked the physics, every repo number and tool flag, and the camera facts. 58 of their corrections are applied: the shutter can't be disabled, frequency-multiplexing two workloads doesn't work here, and several command lines would have measured the wrong thing. The key repo numbers, the driver and the analysis code were re-checked by hand. Nothing was run on a card. The full plan (markdown, 130 KB) has every derivation, command and caveat.
et-soc1-prototyping: docs/findings/11-thermal-model.md (Foster chain, leakage), 12-heat-management.md (budget, time-to-90 °C ladder, −0.0 not gated), 14-card-behaviour.md (governor, telemetry, traps), 16-dvfs-and-leakage.md (31.79 W idle, 15.10 W unmetered), 18-on-chip-relay.md (relay), 19-observability-and-the-unmetered.md (droop meter, IR-drop map), docs/energy-manual/04a-fine-grain.md (per-class rail splits), workloads/*/host/main.cpp (flags), workloads/enercat/run_catalogue.py, workloads/memprobe/gen_ops.py, tools/ettelem/run_*.sh, docs/reports/data/2026-09-22-horace-aifoundry3/cards.json.ET-PCIe-Dev-Card-V3.pdf pp. 2–6 (layout, power tree, regulators, meter); ET Preliminary Datasheet Rev 1.0 §4, §7 (8 × 6 mesh, four memory shires per side), §9.1 (no thermal data published), §9.5 (45 × 45 mm package).