From d8ddea13ba11fd85768420cff584a0d3ee9fff9f Mon Sep 17 00:00:00 2001 From: Dennis Brentjes Date: Sat, 10 Oct 2026 15:26:39 +0200 Subject: [PATCH] Measure the throughput limit: window x RTT, not the adapter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Swiss dumping a disc over the adapter measures 850 KB/s (6.97 Mbit/s, 25.8% of the 27 Mbit/s EXI ceiling). The capture decomposes it completely: 18395 data frames / 9200 ACKs = 2.0 frames per ACK 18.84 MB / 9200 = 2048 B per round trip 22.78 s / 9200 = 2.48 ms per round trip 2048 B / 2.48 ms = 826 KB/s ~= the measured 850 So the flow is window x RTT bound and nothing else. Payload per frame is exactly 1024 B (wire 1078 minus 54 of headers) against a 2920 B window, so two segments fit in flight. Zero retransmissions means the 15-page ring never overflowed; 0.2% stall time means nothing waits on the disc; our W5100 bus engine sits at ~17% utilisation. The adapter is not the constraint. Levers, and the first two are not ours: a larger lwIP TCP_SND_BUF (scales almost linearly while window-bound), and full-MSS segments -- Swiss writes 1024 B at a time, so the same window and RTT at 1460 B would give ~1150 KB/s, a 1.43x gain for free. Third is a shorter round trip, which is GC-side interrupt dispatch plus lwIP, plus EXI lock contention: libogc2's bba_event_handler defers when EXI_Lock fails, so RX queues behind TX on the single channel. ⚠️ This REVERSES an earlier recommendation in the TODO. RX interrupt coalescing (NCRB.RXINTC, RXINTT) was listed as a throughput lever; for a window-bound flow it makes things worse, because delaying the ACK's interrupt lengthens the round trip and throughput is bytes-per-round-trip / RTT. The measurement showed the guess was backwards. Noted in item 3 as "implement for fidelity, not for speed". analyse-pcap.py now derives frames-per-ACK, bytes-per-round-trip and the implied RTT itself, identifies a window-bound flow, quantifies the full-MSS gain, and warns against coalescing -- so the next person gets the conclusion rather than the raw numbers. Self-test G replays this exact shape. Co-Authored-By: Claude Opus 5 --- docs/bba_compat_todo.md | 54 ++++++++++++++++++++++++++++++++++++ scripts/analyse-pcap.py | 51 ++++++++++++++++++++++++++++++---- scripts/test-analyse-pcap.py | 29 +++++++++++++++++++ 3 files changed, 129 insertions(+), 5 deletions(-) diff --git a/docs/bba_compat_todo.md b/docs/bba_compat_todo.md index df04f91..76e954c 100644 --- a/docs/bba_compat_todo.md +++ b/docs/bba_compat_todo.md @@ -168,6 +168,54 @@ For the record, libogc2 writes `TLBP = 0x0000`, `BP = 0x0001`, `RHBP = 0x000F` — exactly the old hardcoded layout, which is why this never showed up in practice. +## Throughput: measured 850 KB/s, and it is NOT the adapter + +**Measured 2026-10-10**, Swiss dumping a disc over the adapter, captured on +the PC and analysed with `scripts/analyse-pcap.py`: + +``` +GC -> PC 18395 frames, 19.83 MB, 850 KB/s (6.97 Mbit/s, 25.8% of the EXI ceiling) + 807 frames/s, mean frame 1078 B, retransmissions 0 +PC -> GC 9200 frames, mean 54 B (pure ACKs) +GC advertised window 2920 B; gaps >5 ms total 56 ms (0.2% of 22.78 s) +``` + +**The flow is window × RTT bound and nothing else:** + +| derived | value | +|---|---| +| data frames per ACK | 2.0 | +| payload per round trip | 2048 B | +| implied RTT | 2.48 ms | +| 2048 B / 2.48 ms | **826 KB/s** ≈ the measured 850 | +| payload per frame | **1024 B** (wire 1078 − 54 of headers) | +| window ÷ segment | 2920 / 1024 → **2 segments in flight** | + +So the sender can have 2048 bytes outstanding and each round trip costs +2.48 ms. **Zero retransmissions** means the 15-page ring never overflowed and +we drop nothing; **0.2% stall time** means nothing is waiting on the disc. +Our W5100 bus engine runs at roughly 17% utilisation at this rate. The adapter +is not the constraint. + +The levers, in order of effect, and **none of the first two are ours**: + +1. **A larger TCP send buffer on the GC** (lwIP `TCP_SND_BUF`). Throughput + scales with it almost linearly while the flow is window-bound. +2. **Full-MSS segments.** Swiss writes 1024 bytes at a time, so a 2920 B + window carries 2048 B of payload. At 1460 B segments the same window and + RTT would give ~**1150 KB/s, a 1.43× gain** for free. +3. **A shorter round trip** — throughput scales 1/RTT. The 2.48 ms is GC-side + interrupt dispatch and lwIP, plus EXI lock contention: libogc2's + `bba_event_handler` bails out if `EXI_Lock` fails and defers until the TX + path unlocks, so RX handling queues behind TX on the single channel. + +⚠️ **RX interrupt coalescing would make this WORSE, and that reverses an +earlier recommendation in this file.** `NCRB.RXINTC` asks for 2 packets per +interrupt and `RXINTT` adds a timer; both were listed as throughput levers. +For a **window-bound** flow they are not: delaying the ACK's interrupt +lengthens the round trip, and throughput is bytes-per-round-trip ÷ RTT. The +measurement is what showed this — the guess was backwards. + ## 3. `NCRB` (0x01) is not decoded at all **Status: ✅ DECODED 2026-10-10** — stored, readable, and exported as @@ -186,6 +234,12 @@ Bits, from libogc2: We interrupt per frame where libogc2 asks for every 2 packets, which is benign (more interrupts than requested). `PR`/`PM`/`AB` feed item 1. +⚠️ **Do NOT implement `RXINTC`/`RXINTT` coalescing as a throughput +optimisation.** Measurement showed the bulk flow is window × RTT bound (see +"Throughput" above), so delaying an interrupt lengthens the round trip and +*lowers* throughput. Implement it for fidelity if a driver depends on the +behaviour, not for speed. + `bba_register_file` **T15** asserts the export is `0x52` with `PR=0`, `PM=0`, `AB=1` — i.e. the real BBA would deliver own-MAC + broadcast only — which is the fact item 1 has to act on. diff --git a/scripts/analyse-pcap.py b/scripts/analyse-pcap.py index dbaea88..6510cfc 100755 --- a/scripts/analyse-pcap.py +++ b/scripts/analyse-pcap.py @@ -157,13 +157,54 @@ def main(): print(f" transactions + W5100 socket bookkeeping.") print() + # ── Is the bulk direction window x RTT bound? ──────────────────────── + # + # THE decisive statistic, and the one that is easy to miss: if the sender + # can only have `window` bytes unacknowledged, throughput is + # window / RTT no matter how fast the wire or the adapter is. Derive the + # round trip from the capture itself -- one ACK is one round trip -- so + # this needs no ping and no assumptions. + data_pkts = stats["from_gc"]["pkts"] + acks = stats["to_gc"]["pkts"] + payload = stats["from_gc"]["payload"] + if acks and payload: + per_rt = payload / acks + rtt = dur / acks + print(f"\nround-trip analysis (bulk direction GC -> PC)") + print(f" {data_pkts/acks:.1f} data frames per ACK, " + f"{per_rt:.0f} B per round trip, " + f"implied RTT {rtt*1000:.2f} ms") + print(f" {per_rt/rtt/1024:.0f} KB/s = bytes-per-round-trip / RTT" + f" <- compare with the measured throughput above") + if wins: + w = max(wins) + seg = payload / data_pkts + print(f" GC advertised window {w} B; segments average " + f"{seg:.0f} B, so {w//max(int(seg),1)} fit in flight") + if per_rt <= w * 1.2: + print(f" ⚠️ WINDOW x RTT BOUND: {per_rt:.0f} B in flight " + f"against a {w} B window.") + print(f" The adapter is not the limit -- the sender " + f"cannot have more outstanding.") + print(f" Levers, in order of effect:") + print(f" 1. a larger TCP send buffer / window on the " + f"GC (lwIP TCP_SND_BUF)") + if seg < 1400: + full = w - (w % 1460) or 1460 + print(f" 2. full-MSS segments: {seg:.0f} B " + f"segments waste the window;") + print(f" {full} B per round trip would give " + f"{full/rtt/1024:.0f} KB/s ({full/per_rt:.2f}x)") + print(f" 3. a shorter round trip -- throughput scales " + f"1/RTT") + print(f" ⚠️ RX interrupt COALESCING would make this " + f"WORSE, not better:") + print(f" delaying the ACK's interrupt lengthens the " + f"round trip, and") + print(f" a window-bound flow pays for that directly.") if wins: - print(f"GC advertised TCP window: min {min(wins)} max {max(wins)} " + print(f"\nGC advertised TCP window: min {min(wins)} max {max(wins)} " f"last {wins[-1]} B") - if max(wins) < 8192: - print(f" ⚠️ a {max(wins)} B window caps a single TCP stream at " - f"window/RTT; at 1 ms RTT that is " - f"{max(wins)/1e-3/1024:.0f} KB/s") if zero_win_spans: print(f"PC saw zero-window from the GC {len(zero_win_spans)} times, " f"total {sum(zero_win_spans)*1000:.0f} ms " diff --git a/scripts/test-analyse-pcap.py b/scripts/test-analyse-pcap.py index a2d257d..9b848c1 100755 --- a/scripts/test-analyse-pcap.py +++ b/scripts/test-analyse-pcap.py @@ -129,6 +129,35 @@ if "pcapng" not in (r.stdout + r.stderr): else: print("F pcapng refused with a conversion hint") +# Scenario G: window x RTT bound -- 2 undersized segments per ACK, which is +# the real shape measured on hardware (1024 B segments, 2920 B window). +# The tool must call it window-bound, and must warn that RX interrupt +# COALESCING would make it worse rather than better. +pkts, seq, t = [], 1, 6000.0 +for i in range(200): + pkts.append((t, tcp_frame(GC, PC, MAC_GC, MAC_PC, seq, b"\0" * 1024, win=2920))) + seq += 1024 + t += 0.0002 + pkts.append((t, tcp_frame(GC, PC, MAC_GC, MAC_PC, seq, b"\0" * 1024, win=2920))) + seq += 1024 + t += 0.0002 + pkts.append((t, tcp_frame(PC, GC, MAC_PC, MAC_GC, 1 + i, b"", win=65535))) + t += 0.0024 # ~2.5 ms per round trip +pG = os.path.join(tmp, "g.pcap") +write_pcap(pG, pkts) +out = run(pG, "--gc-ip", "10.0.0.64") +if "2.0 data frames per ACK" not in out: + fails.append(f"G: frames-per-ACK not derived\n{out}") +elif "WINDOW x RTT BOUND" not in out: + fails.append(f"G: window-bound flow not identified\n{out}") +elif "COALESCING would make this WORSE" not in out: + fails.append(f"G: missing the coalescing counter-warning\n{out}") +elif "full-MSS segments" not in out: + fails.append(f"G: did not suggest full-MSS segments\n{out}") +else: + print("G 1024B segments + 2920B window -> window x RTT bound,") + print(" full-MSS suggested, coalescing warned against") + print("\nFAILURES:\n " + "\n ".join(fails) if fails else "\nAll analyse-pcap self-tests passed.") sys.exit(1 if fails else 0)