OCaml Planet

OCaml Weekly News, 03 Nov 2026

Caml Weekly News — 2026-11-03T12:00:00-00:00

Local AI agents can now pass the IUCN Red List assessor exam

Anil Madhavapeddy's Blog — 2026-11-03T00:00:00-00:00

Shane Weisz become a certified IUCN Red List assessor last November, and then wondered whether Claude Code could pass the same exam he had just sat. It turned out that it could, with a similar mark to what Shane himself achieved! That was a big discovery at the time, but we kind of got used to frontier models being really good.

The advance of local models since then has been so rapid that they can now pass the exam as well alongside frontier models, and by running on a desktop-class machine! This is detailed in our new preprint on evaluating AI agents on the IUCN Red List assessor exam.

Why bother testing on an exam?

Conducting accurate species assessments is painstaking work. The IUCN Red List is the world's most widely used system for assessing the extinction risk of species, and it underpins conservation policy and action everywhere. But the assessments are severely resource constrained; <20% of plant species have been assessed at all, and over 30% of the assessments that do exist haven't been revisited in 10+ years. One bottleneck is the number of trained assessors available, which is why Shane's PhD and his Dash of Life project is looking at how to inject modern tooling to the Red List to help them out.

Modern AI can help assessors increase both the throughput and the quality of assessments, but it raises very reasonable concerns about trust and reliability, data security, cost, and the environmental impact of the datacentres doing the inference. The official assessor training exam is a convenient starting point for examinging these tradeoffs. The exam covers species categories and criteria, and mapping standards for extent and area of occupancy, and also regional assessment protocols. The questions are hard enough that Shane found them tough, and I didn't know the answers to most things!

What we found

We built a benchmark of exam questions and executed a spread of agents against it, each with the same official IUCN guidance documents that a human candidate has access to. We found that frontier models from Anthropic and OpenAI reliably clear the 75% pass mark, with the best of them (Claude Opus 5) reaching 96% mean accuracy, rising to 99% with a majority vote over five attempts.

The breaking result is that open-weight models running on local hardware now get a pass as well. Qwen3.8 Flash, self-hosted on a desktop Mac Studio on Shane's desk, attains 81% mean accuracy (87% with majority voting). That is a comfortable pass mark, which means assessment agents could plausibly be deployed on offline and mobile devices in the field. For many Red List partners, the location of sensitive species data (think the coordinates of the last few individuals of a poached orchid) is a blocker to using AI, so having a usable local alternative could be a game changer for conservation practitioners.

The exam is of course only a start. Passing it doesn't make a model a "species expert" any more than it makes a newly certified human one, and we're keen to encourage the responsible use of AI assistants within assessors' workflows (such as quality and consistency checks on draft assessments) rather than as a replacement for the humans.

Assessors in the loop

Our hope is that AI responsibly deployed, with assessors-in-the-loop and firmly in control, could be a powerful tool to meet the scale of the resourcing challenge in global biodiversity. But we need to collectively work through issues of transparency, traceability and auditability of anything an agent produces. Just as importantly, the human communities of assessors and specialist groups who do this work need to end up stronger and better connected as a result of these technologies, not more fragmented.

The paper is a collaboration between Shane Weisz, Michael Dales, Sadiq Jaffer, Steve Bachman at Kew. You can read the preprint on EcoEvoRxiv, and comments and thoughts are extremely welcome, to any of us.

References

Weisz et al (2026). Evaluating AI agents on the IUCN Red List assessor exam: Tradeoffs and opportunities for AI assistants in extinction risk assessments. EcoEvoRxiv. 10.32942/X2870H
Madhavapeddy (2025). Four Ps for Building Massive Collective Knowledge Systems. 10.59350/418q4-gng78

How to Use Conditionals: Terraform's if Statement Alternative

Terrateam Blog — 2026-11-03T00:00:00-00:00

The Terraform if statement doesn't exist. Conditional expressions do the same job, from conditional resources to gated modules and dynamic blocks.

Notes from week 44

Marc Elvers — 2026-11-02T09:00:00-00:00

Greenland is being processed into the Zarr store on a single L4; day10 stopped refreshing its base images and started carrying every package it might need; a five-year-old lwt workaround turned out to be the bug; monteverde’s drive was failing after all; and a kernel CVE was mitigated across 90 hosts in an evening.

`day10`

Base refresh breaks the invariant

day10 keeps a base image per platform and builds a layer per opam package on top of it. The whole design rests on every layer having been installed from the same package index as its siblings.

However, we need to keep the distributions, or at the very least the mirror list, up to date. To do this, ocluster-worker periodically ran day10 refresh-base which effectively runs apt update or equivalent for each distribution in each base image.

PR #31032, adding imguiml, failed 88 of 307 jobs and showed both consequences. On Debian and Ubuntu, 64 jobs died with apt reporting “Unmet dependencies”: the conf-freetype layer had pulled libbrotli1 1.0.9-2+deb12u1, sibling layers still carried the base’s +b6 stanza, and the merge of dpkg’s status files kept whichever came first. On Arch, four jobs died with pacman reporting a duplicate database entry: a layer’s whiteout over the base’s brotli-1.2.0-1 directory had been dropped by the merge, because the base is mounted beneath the merge as a lower layer of its own and the merge only looked at itself.

Those two are fixed in 0a54c2b, which keeps the highest version per package in the dpkg union using opam’s Debian-style comparison with the epoch split off first, and edaf118, which keeps a marker unless a later layer really provides the path.

Is `refresh-base` conceptually flawed?

obuilder uses a weekly refresh, which isn’t perfect, as there are days where mirrors or packages are unreachable/unobtainable. However, this does mean that the images themselves are consistent, whereas with day10, I think that isn’t always true when we start overlaying packages in unintended ways. With Ubuntu and Debian, I think it could work, but distributions with continuous updates, such as Arch, will continue to cause problems.

`prune --base --days N` replaces `refresh-base`

day10 now wipes the base image and all layers which have a depext every 7 days and rebuilds them. Consistency over the 7 days is guaranteed by prefetching all depexts at the point the base image is built, so any opam calls to install depext find the package already cached.

ocluster-worker runs the command on a schedule. Previously, the idle window actually detected the end of the idle window: the worker sat blocked on a Cap’n Proto call until a job arrived, and then the maintenance hook ran exactly when there was work to do. Now a thread wakes once a period, runs the prune if nothing is building, and otherwise logs that it will try again in an hour.

`day10 cp`

The copy step was originally a perfect sudo cp applied oldest to newest, but unfortunately, this isn’t so perfect when whiteout character markers are left. This is largely caused by the original refresh-base command, but the copy step is growing in complexity with an initial sudo find followed by a sudo cp with sudo rm -rf applied to marks. Some of the later changes also required day10 to run as root, which was a big step back. sudo cp was one thing, but requiring sudo day10 seemed excessive and broke my own day10 build . use!

Effectively, the updates have been recreating the overlayfs semantics in day10. Interestingly, the overlayfs limit of exactly one memory page for the mount options has been removed from the Linux kernel, so I can now stack unlimited layers rather than creating the hard-linked tree. I decided not to do that as the RISC-V machines still run 5.10 kernel (in some cases).

day10 now provides its own version of cp via day10 cp, which implements the exact semantics it needs. A trivial benchmark showed that day10 exec -- tessera-mosaic --help took 2.37 s after the change, down from 2.46 s before, so marginally faster and certainly not worse.

`fetcher`

The extra downloading of distributions, coupled with the HTTP GET storms the CI workers generate when the test matrix fans out to N distros and two compiler versions, would benefit from a cache.

I did a quick review of proxy caches which were available looking for these three specific properties: files live at <root>/<host>/<path>, unhashed, so anything can be found and deleted with ls and rm; one upstream fetch per URL however many clients ask at once, the rest waiting on the first; and nothing is ever refreshed or expired until an operator deletes it.

Clients are not proxied, because nearly every mirror is https and an https_proxy setting makes them tunnel TLS straight through, so the cache sees nothing. Instead, the URL is rewritten: https://deb.debian.org/debian/... becomes http://cache:8080/https/deb.debian.org/debian/..., and the cache terminates TLS to the origin itself.

opam can be configured to use it and similar tricks are available in the package managers.

opam option download-command='["fetcher" "download" "--cache" "10.0.0.1:8080" "%{url}%" "-o" "%{out}%"]'

It is on GitHub at mtelvers/fetcher. It is not deployed yet.

Other CI work

A CVE across everything

CVE-2026-98432 is a use-after-free in the IPv4 TCP handshake path. tcp_v4_syn_recv_sock hands the request socket’s IP options to the child socket without copying them, and a retransmitted SYN arriving on another CPU can read them after they are freed. It is reachable by any unauthenticated remote peer that can send a SYN carrying IPv4 options, it affects every kernel since 4.4, and the kernel CNA rated it 9.8. IPv6 is not affected.

Of the 121 machines surveyed, every one but monteverde was running a vulnerable kernel, and for almost all of them no fixed kernel had been released. Debian had shipped 6.12.113, so dopey, the only Debian host, was properly patched and rebooted that evening. monteverde happened to be on Ubuntu 26.10, whose 7.3.0-10 carries the fix, and was never vulnerable. For everything else, Ubuntu’s 6.8.0-149 for 24.04, 5.15.0-203 for 22.04, and 7.0.0-43 for 26.04 were all still sitting in -proposed.

The interim mitigation is an nftables table on the prerouting hook, so Docker-forwarded traffic is covered too, that counts and drops inbound IPv4 TCP carrying IP options, plus net.mptcp.enabled = 0.

Just under 100 machines were mitigated this way, and will stay that way until the fixed kernels are promoted.

A symlink, a `stat`, and a fix from 2021

Marek asked for symlinks on the Windows runners as odoc’s tests had been failing with a syntax error in cases/ocamlary.mli. git had checked the symlink out as a text file containing its target path. Git for Windows is installed with /VERYSILENT, which takes the installer’s defaults, and the default is core.symlinks=false. The catch is that git bakes that into every repository’s .git/config at clone time, so setting it system-wide fixed nothing for the 1,170 repositories already cached across the four workers. Unset in all of them, live, without pausing anything.

odoc’s checkout now had a real symlink, and the next run failed differently: Unix_error(ENOENT, "stat", "...\\test\\generators\\cases\\ocamlary.mli"), Internal error, the worker crashing while deleting the build context. Lwt_io.delete_recursively walks with lstat, and on Windows, when it finds a symlink, it then calls stat on it to decide between rmdir and unlink. stat follows the link. If the target has already been deleted, which it has whenever a link points at another file inside the tree and readdir visits the target first, that raises ENOENT and nothing catches it. odoc has exactly one symlink, from test into src, and src is traversed first, so every build of it failed, deterministically.

The first fix caught ENOENT and tried unlink then rmdir. However, a quick test on a Windows worker showed the rmdir leg never fired, as unlink always worked. That is weird because the entire Windows specialisation in that function was based on the assumption that rmdir was needed to remove Windows directory links and that unlink was needed for file links. Perhaps it was just that broken links could be removed with either? Testing on the worker showed this was not the case: unlink always worked. Of course, on Windows, there is a risk in assuming that the one machine you are using is representative across all Windows machines, so I was reluctant to make a wholesale simplification.

Searching the git history for ocaml/ocaml found d9cfda1e1a: Antonin Décimo, 16 September 2021 at 16:54, “making Unix.unlink remove directory symlinks on Windows”, which was released in 4.14.0, so the behaviour did change. Interestingly, earlier the same day Antonin modified the function in lwt.

As lwt requires 4.14, the entire special case is redundant on every runtime lwt supports, making the function identical on all platforms with one lstat per entry.

Raphaël’s review asked how we know the test reaches the bug, given it depends on readdir order, which was a fair criticism of my test. Based upon what I’d seen, NTFS always returned a sorted file list regardless of the creation order. However, Microsoft does not guarantee that, it just happens to be true for NTFS. If the underlying file system was FAT then it wouldn’t be. I rewrote the test to have two sibling directories cross-linked at each other, so that whichever is removed first leaves the other’s link dangling. ocsigen/lwt#1132.

core.autocrlf=true is the same installer default and was fixed at the same time. It had been putting a carriage return at the end of set -e in every shell script in every checkout.

macOS, again

On Tuesday morning, two more of the rebuilt arm64 workers had corrupted overnight: m1-worker-04 with its first adding existent segment to range tree panic and m1-worker-03 with a data abort inside the ZFS kext. That made three of four within five days of the 21 October rebuild.

The Intel side is quieter. The /private/tmp sweep from #62 was sized against 16 GB a day; under a large batch the real rate was a 3.8 GB Homebrew source tree an hour, and i7-worker-04 was back down to 32 GB free. #63 halves the window to six hours.

OCaml 5.5.2 and new distributions

The rollout that follows an OCaml release. ocaml-dockerfile 8.4.6 went out with the Ubuntu 26.10 change and is in opam-repository; docker-base-images#362 picks it up along with ocaml-version 4.1.5, adding 5.5.2 and 26.10 and dropping 25.04, 25.10 and Fedora 42; opam-repo-ci#490 and ocaml-ci#1076 follow, and the day10 branches of both were rebased on top.

FreeBSD, forty days on

Before rosemary was rebuilt with OCaml 5.5.2, one last reading of the vnode fix from September: vnlru_failed frozen at 25 for forty days, through two more traffic surges that took the load average to 33 with eight jails and the store to 21,183 datasets before it pruned itself back to five thousand. No warning ever logged.

grumpy

“193 updates could not be installed automatically” turned out to be a release upgrade from jammy to noble that had stopped halfway: base-files, dpkg and libc6 from noble, apt, systemd and the kernel still jammy, both sources lists present, and noble’s t64 library renames needing removals that unattended-upgrades will never do. The root filesystem was 7.5 GB with 1.3 free, and the volume group had 149 GB unallocated, so 40 GB online and the upgrade finished, then on to 26.04.

A side find for every 26.04 host: the garbage on the serial console at boot is systemd 259’s OSC 3008 context escape and an XTGETTCAP query that serial-getty@ sends because TTYReset=yes. A drop-in with TTYReset=no stops it.

Tessera

Greenland for EPFL

With monteverde back in service, I have started processing #408: dark ice on the Greenland ice sheet, a bounding box from 55 to 42 degrees west and 59 to 71 north, for 2017, 2019 and 2023.

I used mtelvers/ferry to create a pipeline similar to the one I used for 17MPU. This will process 412 shards across UTM zones 21 to 24, 5,513 live sub-windows of 6,592. 114 of the 2017 shards were already in the store from earlier work, so 1,122 remain.

The ice sheet itself is interesting in the logs. Interior shards get no usable Sentinel-1 at all, 165 of 165 dates unavailable, and the model falls back to Sentinel-2 only; the download log has 224,846 lines rejecting HH/HV polar-mode scenes against the model’s VV/VH.

As of 8am this morning: 67 shard-years uploaded, 17.3 minutes a shard, no failures overnight, about 83 shards a day, finishing around 15 November.

dClimate’s engine against mine

On a call with dClimate earlier in the week, we discussed the scarcity of GPUs across cloud providers. We suggested using the AMX to do inference on CPUs. However, when they looked at the numbers, dClimate’s optimised Python pipeline was significantly quicker.

I could not remember a recent benchmark on their inference engine, so when monteverde was available, I tested the inference on a single tile vs my CUDA encoder on the NVIDIA L4.

engine	per tile
dClimate v2 Large, pipelined	129.0 to 130.3 s
dClimate v2 Large, serial loop	129.8 to 131.6 s
my CUDA encoder, `TC_CHUNK=64`	117.0 s

Cambridge machines

monteverde

monteverde was reporting NVMe errors despite a spotless SMART record and the drive firmware at the latest available. I gathered logs so the issue could be reported to Dell. However, after a single reboot, the box came up in the initramfs: the PS1010 now failed every read, including LBA 0, with Internal Error rather than the SGL rejection. A controller reset, a physical reseat and a full power cycle changed nothing.

Dell replaced the drive with a Samsung-based PM1743. The rebuild on Saturday had its own obstacles. The department’s PXE rescue entries fail in GRUB with “out of memory”, so the route was a manual-install entry, a shell, a static address typed by hand because nothing offers monteverde DHCPv4, and the 26.04 server ISO written raw onto the new drive and booted from the old GRUB. Root on the BOSS mirror this time as that is a hardware RAID mirror. tank as a stripe of the two data drives, 27.9 TB, with home, scratch and the container stores on it.

On Saturday, Anil upgraded it to 26.10 with do-release-upgrade -d, and within an hour the console was logging Data SGL Length Invalid again. Always on the surviving PS1010, never on the PM1743, on the same bus. We are currently running with nvme.sgl_threshold=0, which forces PRPs for everything. Hopefully, Dell will give us a preemptive replacement.

hopper: three releases in a day

hopper, the Grace Hopper box, was on 22.04 and needed to be updated to 26.04. This has to be done on-site, as the fans are so loud that we need to ensure the server room is empty before rebooting. The upgrade was an uneventful step through 24.04 to get to 26.04 with the latest NVIDIA drivers.

daintree’s crash was a power cut

daintree had been restarted, which I initially assumed was an OOM crash which turned into a reboot. The amber light on the front led to racadm, and the iDRAC’s lifecycle log reported: “iDRAC firmware was rebooted with the following reason: ac”, at 22:17 on 25 October.

hopper’s previous boot ended uncleanly at 22:14 the same night, marpe’s at 22:14, and grumpy rebooted at 22:22. The common factor was that all these machines were in the same rack.

On the way back up daintree ran four firmware jobs that had been staged in May, including a BIOS from 1.12.4 to 1.17.1. The amber light itself is “Unable to authenticate System FPGA/CPLD” at every power-on since at least September. And the reboot did not change the kernel: still 6.8.0-63, with 447 updates pending.

marpe’s backplane

marpe came up from the same power cut halted at a BIOS prompt: “Comm Error: Backplane 1”. The third backplane assertion in three months, roughly monthly. Via the serial console, I accepted the configuration to boot it, then found the one combined power and signal cable that runs from the backplane through the latch between the fans to the front edge of the board, and reseated it. If the sensor stays quiet until December, the cable was the fault.

UID standardisation

daintree’s power cut was the opportunity to renumber ae491 and sj514. monteverde’s reinstall meant recreating its users, so they went in at their canonical numbers from the start, with mte24 finally renumbered there too. zf281 was back on daintree with 27 processes before his turn came, and remains the other half of the 1012 collision. cae37’s R jobs are still running.

Other

conservation-copilot: the R hang

On the VM, the Celery worker was in a fork-crash loop: 494 kills in ten minutes. Restarting everything brought the service back, but why did it fail? Fortunately, there are detailed logs, so I could replay the last query, which recreated the crash.

The issue is that when the query runs for more than the hard time limit, the billiard.Pool tries to kill the process, but the R worker blocked in native code didn’t exit and everything piled up behind it.

I replaced the nested pool with a plain subprocess: pickle in, JSON out, SIGKILL on timeout, which R cannot ignore. PR #24, merged. The R hang itself is still unexplained.

fuel

I revisited my fuel web app, as the data was missing. Claude wrote it, and I asked for a fix and a review of the project, explaining that it was written six months ago by an earlier version of the agent.

The app was substantially rewritten, taking it from a single file to an API client, a station cache, and a thin server, each with an interface, and moving refresh into a background fibre that publishes an immutable snapshot.

The client adds per-request timeouts, re-authentication on 401 or 403, backoff retries on 429 and 5xx, result-typed errors instead of escaping exceptions, and a connector that tries IPv4 before IPv6 and falls through on failure. A failed or partial refresh now keeps the previous snapshot and retries with backoff, the frontend lives in its own HTML file embedded at build time and polls a status endpoint until data is ready, logging uses proper Logs sources, the stations endpoint supports ETag and 304.

console: Making OCaml Command-Line Tools Look Good

Thomas Gazagnaire — 2026-11-02T00:00:00-00:00

A command-line tool can print lines, which stay in the terminal's scrollback but look plain. Or it can take over the screen like a full-screen app, which looks good until it exits and takes its output with it. I wanted the looks of the second and the scrollback of the first, so I wrote console, an OCaml library for tables, trees and live progress rows that print into the scrollback. Its first version, 0.9.0, is now on opam:

$ opam install console console-eio

It comes in two parts. console is a set of pure, composable widgets. Each one is a three-dimensional object, drawn in the room it is given (a width and a height) at a given time, so the same widget can be printed once or animated. There are enough of them to write a command-line game where a camel jumps over lambdas, or a stock ticker. console-eio is the driver that feeds them the clock, the keyboard and the terminal's size changes, through Eio for now. Both need OCaml 5.4, on Linux or macOS.

examples/camel/main.ml, recorded by k7.

A trading day on an invented exchange: examples/stocks/main.ml, recorded by k7.

It looks like a TUI, but you can scroll

The recordings in this post look like full-screen terminal applications, but none of them is one. A full-screen program switches the terminal to its alternate screen, which suits an editor. A build or a test run is different, as its output is read after it ends, in a CI log or by an LLM agent that ran it. So console redraws only the few rows at the bottom that are still running, and everything a program printed is still there when it ends.

OCaml is a great language for command-line tools, and I have written or maintained a few of them over the years, opam among them (Louis Gesbert later made its output much nicer). But I have always looked jealously at the tools of other ecosystems, and I have brought one to OCaml before, with Tailwind without Node. Docker's BuildKit, cargo, uv and pnpm print lines into the scrollback and redraw only a small live region below them, and they look good doing it. I wanted the same for OCaml, with a pure OCaml API and output versatile enough for all my tools, including the one I am writing to deploy applications to satellites with SpaceOS.

OCaml already had some of the pieces. printbox prints boxes, tables and trees, and progress draws progress bars. But I could not combine them, because a box cannot redraw and a progress bar cannot sit in a table. console has a single type of view for both, so the live rows are built from the same tables, trees and panels as the output that stays in the scrollback. Most libraries keep the size and the time in mutable state, as Bubble Tea does in its model. A console widget takes both as arguments, so it can nest in any other widget and still move.

console borrows a lot from Python's Rich, which it is modelled on, and from indicatif in Rust and Charm's lipgloss in Go. Full-screen frameworks such as Charm's Bubble Tea or OCaml's notty solve a different problem. The README's related work lists the rest.

To see how far a single type of view goes, I imitated the output of the tools I use every day. None of these is the real tool: each is a short console program that draws its rows the way that tool does, and its source is in the repository's examples directory. So you can now pick your favourite package manager, and have your OCaml tool look like it:

opam install

cargo build

pnpm install

docker buildx build

brew install

nix build

uv sync

apt install

Tasks and their logs

A program reports its work as tasks. While a task runs, its row shows its latest lines in a window of ~log_lines of them: its Logs messages, the lines given to Task.log and what it prints with Fmt.pr. The window folds away when the task succeeds, as BuildKit's does. When the task fails, it stays under its ✗ line:

open Console

let build root =
  Task.run root "compile" (fun step ->
      Task.log step "ocamlopt parser.ml";
      Logs.warn (fun m -> m "parser.ml: unused variable x"));
  Task.run root "test" (fun step ->
      Task.log step "test_parser: 11 passed";
      Task.log step "test_lexer: expected 3, got 4";
      Task.fail step);
  Task.fail ~message:"1 test failed" root

let () =
  Eio_main.run @@ fun env ->
  Console_eio.run ~clock:(Eio.Stdenv.mono_clock env) ~label:"Building" build

The rows go to standard error, which leaves standard output to the program. Piped into another program with both, nothing is redrawn, and each line is printed once (left). On a terminal, the rows of the tasks still running are redrawn in place, under a table of the steps that changes as they go (right, from another example). A warning logged outside a task stays above the rows once that task succeeds:

$ ./build.exe 2>&1 | cat
• compile
  › compile: ocamlopt parser.ml
  ! compile: parser.ml: unused variable x
✓ compile  0.0s
• test
  › test: test_parser: 11 passed
  › test: test_lexer: expected 3, got 4
✗ test  0.0s  failed
✗ Building 0.0s (2/2)
✗ 1 test failed

examples/display/main.ml, recorded by k7.

Themes

console comes with a bunch of themes, all shown in the README, and you can define your own. The default theme uses the terminal's own colours. Others follow existing colour schemes, such as Dracula and Solarized. Theme.matrix and Theme.rainbow are animated.

Theme.solarized_light

Putting views together

Tables, trees, panels and bars can all be put inside one another, a bit like boxes in an HTML page. Here, for instance, a tree of files sits next to a small panel, which has a sparkline inside it:

open Console

let tree =
  let leaf s = Tree.v (View.text s) [] in
  Tree.view
  @@ Tree.v (View.text "lib")
       [ leaf "view.ml"; leaf "table.ml"; leaf "panel.ml" ]

let build =
  Panel.view ~title:(View.text "Build")
    (View.vcat
       [
         View.text "time   1.2s";
         View.hcat
           [ View.text "trend"; Bar.sparkline [ 3.; 5.; 2.; 8.; 6.; 9. ] ];
       ])

let () =
  Console_tty.setup ();
  Fmt.pr "%a@." View.pp (View.hcat [ tree; build ])

lib           ╭──── Build ────╮
├── view.ml   │ time   1.2s   │
├── table.ml  │ trend  ▂▄▁▇▅█ │
└── panel.ml  ╰───────────────╯

The same goes for bar charts, and for the live rows themselves. So a program can keep a summary table above its running tasks, or give each of its builds a column of its own:

Bar.chart and Bar.sparkline in a panel and a table

Two builds side by side, with View.hcat

Not only progress bars

The live rows can draw more than progress bars, like the camel at the top of this post. I also built a simulated Winamp 2 player out of console's ordinary widgets. Two panels hold the player and the playlist. Vertical bars draw the spectrum analyser, whose bands are a few sines and some seeded noise, so every run draws the same frames. Horizontal bars draw the volume, the balance and the seek bar, and plain text laid out with View.hcat and View.vcat gives the LCD digits and the scrolling title. A theme of its own then gives it Winamp's colours, with the analyser's palette from Winamp's default skin:

open Console

let rgb = Color.rgb
let green = rgb 0 232 0

(* Winamp's default analyser colours (viscolor.txt), from the bottom up. *)
let viscolor =
  [
    rgb 24 132 8;
    rgb 41 206 16;
    rgb 148 222 33;
    rgb 222 165 24;
    rgb 214 90 0;
    rgb 239 49 16;
  ]

let theme =
  Theme.v ~name:"winamp" ~accent:green ~palette:viscolor
    ~muted:(rgb 150 150 150) ~background:(rgb 41 41 57) ~border:Border.single
    ~header:Style.(bold + fg (rgb 214 214 222))
    ()

let () = Console_tty.setup ~theme ()

examples/music/main.ml, recorded by k7.

Keeping the terminal at the edge

The goal is for console to work with the libraries OCaml command-line tools already use. A view prints with Fmt on any formatter, and Console_tty.setup takes the same style renderer as Fmt_tty, so a --color flag from Fmt_cli works unchanged. Logs works unmodified too, as the tasks above show.

For tables and trees, that is easy, as each of them is just a value that a formatter function prints, a bit like the view in the Elm architecture. The live rows are trickier, as they depend on time and on the size of the terminal. So the display never reads the clock or asks the terminal for its size itself: it is given a function for each, and a formatter to write to. console-eio provides them for Eio programs, and the tests provide a buffer and a clock that they move by hand.

This is the same idea as in nqsb-TLS, the TLS stack of MirageOS. Its core takes bytes and returns bytes, and leaves the network to its caller. console is less strict, as its display still writes to the formatter it is given instead of returning what to write. But the clock and the terminal stay outside, and that is what makes the recordings reproducible.

Recording the animations

A library like this one is shown by its animations, and I changed the code every day while writing it. Recorded by hand, with VHS or asciinema, the images in the README would have stopped matching the code within a week. So I wrote k7, a small language-agnostic tool, to record them (see last week's post). Under a mocked clock, a console program's output no longer depends on how fast the machine is, so k7 draws the same SVG image, animated with CSS alone, on every run. Each image is then a test. dune draws it again from its program and diffs it against the one in the repository, so the images change with the code.

What's next

I will move my own tools to console over the next few weeks, before cutting a 1.0 release, starting with the ones in my Homebrew tap and the SpaceOS CLI. I tried a few ideas over the last six months in an experimental playground, but the GitHub repository is now the official one, and the one released on opam. I will delete the playground once all my tools have moved. Feedback is welcome on its issue tracker.

.plan-26-44: Commons People

Anil Madhavapeddy's Blog — 2026-11-01T00:00:00-00:00

With term in full swing, I caught the traditional head cold and mostly lost my voice after giving many lectures this week! It's inevitable for me to catch one every first term; I hear a chorus of coughing from the 1A FoCS students by the second week, and I fall like clockwork in the third. The week has also been extremely busy keeping up with a huge deluge of security issues that have required 100s of our open source infra machines to be upgraded. The security pressure on open source projects right now is relentless, unfortunately.

Still, it's been very fun being back in Pembroke again after sabbatical and I wrote in more detail about three different events this term there, covering the computer science subject dinner, the weekly Evidence TAP "samosa socials" to encourage interdisciplinary conversations, and admitting two new honorary fellows.

Two of the Part II students have started keeping public weeknotes, which I'm delighted to see! Matt Occleshaw is writing his notes on the embedded OCaml DSL for OpenStreetMap. Finley Stirk is restarting his weeknotes after his UROP work this summer on GPS uncertainty in GBIF point labels for Tessera. I'm watching Matt's progress on OpenStreetMap with interest since I've also been looking into OpenStreetMap mirroring for my UK label database below. And Cyrus Omar has also refreshed his website with weeknotes for his sabbatical.

In the rest of this week's notes, I headed to London to keynote the OpenUK Space Tech Radar launch and began a comprehensive UK label map for Tessera that could be the basis for a collaborative wiki via Tangled, and Tessera was also shortlisted for an OpenUK award. There are plenty of fun papers and talks in progress, and my links around the web as usual.

Keynoting the OpenUK Space Tech Radar

I also took a break from lecturing and headed down to London on Thursday to give the opening keynote at the OpenUK Space Tech Radar launch, held at the (very) swanky Fieldfisher offices.

My talk was on using open geo-foundation models like Tessera to find life on Earth. I was also delighted to meet Simon Wardley at long last, since I've almost crossed paths with him many times after his early work on virtualisation, containers and now on strategy maps! This event follows on from the AI Openness report conversations with OpenUK earlier this year.

A UK label map for Tessera

The crowd at the Space Tech Radar launch was well into finding ways to use Tessera. On the train back, I started pulling together the proper label database with a specific focus on the UK. My aim is to harmonise the many sources of ground truth labels we use for habitat and urban mapping into one place, since every dataset I encounter has subtle variations in schema, coordinate conventions and notion of uncertainty. This builds on the OpenStreetMap, GBIF and iNaturalist raw mirror I set up a couple of weeks ago.

The UK has an incredible wealth of open data available, and I discovered that the government portal has been turned into a National Data Library of the sort that I speculated about last year! So far my label db involves a large sync of:

OpenStreetMap for the built and natural environment features.
GBIF species occurrence records, which ties in with Finley's project on their GPS uncertainty.
ESA WorldCover land cover classes that cover 2020/2021 at a 10m resolution (a good fit for Tessera).
The Environment Agency's hydrology data via their API portal, for rivers, flows and water levels across England. Wales, Scotland and Northern Ireland aren't covered by these but I'm working on those.

After that, I spidered the NDL with some (carefully rate-limited) local agents and found various APIs for:

Lidar surveys: 2 m DTM composite, first-return DSM composite, vegetation object model; and tiles via the survey data download. These give us very fine terrain models but only over some years.
The National Forest Inventory 2023 and 2022, trees outside woodland from Defra, and ancient woodland from Natural England.
Habitats and soils data along with a priority habitats inventory, the Living England habitat map, peaty soil extent, agricultural land classification; and a CASI and LIDAR coastal habitat map.
How sites are designated by Natural England with a sites of special scientific interest, the national nature reserves, Ramsar, Country Parks; and RSPB's Important Bird Areas and reserves and heathland extent and potential. Some of these are under the Open Government Licence, and others are research use only.
Then plant atlas info from 2020 such as tetrad presences, hectad presences by period, hectad trends, native status, conservation status, and phenology.
There are also ancient tree registries from the Woodland Trust Ancient Tree Inventory and the Tree Register champion trees.
Going back to water, I got more info from the Environment Agency on main rivers, storm overflows 2020-2025, historic flood warnings, coastal erosion risk; and the water quality archive. This complements the EA hydrology data above and is all available via nice APIs or downloads.
Landfill data, both currently authorised and historic site boundaries
Air quality got a look in as I have an air quality project idea with 1km background concentrations from 2003-2024 for NO2, NOx, PM, ozone, SO2, and benzene; and even AURN hourly site data.
I also grabbed base maps from the Ordnance Survey OpenData on terrain types, Open Rivers, Greenspace, OpenMap Local, Boundary-Line, Open Roads, Zoomstack, and Open Names. Phew!

I'm just putting this enormous (and growing) dataset together into one DuckDB instance and a tileserver along with GBIF. It can't be easily redistributed I think, as there are a lot of competing licence terms in there I need to figure out first. However, just the list above is quite useful to get started. I put together a quick Tessera WorldCover 10m downstream task and hooked it up to the Esri Wayback service (to verify historical satellite data) and it's looking very promising so far.

Hosting and sharing these on Tangled

For the longer term, we'll need a way of sharing these datasets online. I'm very excited that Akshay Oppiliappan launched a big upgrade of Tangled code hosting with lots of new features. Most importantly, the self hosted code hosting knots that I run for both Recoil and the Computer Lab are now upgraded to a fancy new Rust based one that is not only faster, but also supports Git Large File Storage!

Akshay is planning a rollout of private repositories in Tangled in the next month or so, which means that we'll have all the pieces necessary to store large files under version control in infrastructure we run, and to share it selectively with a shared social graph using the AT Protocol. This could revolutionise how we can harmonise all those datasets above into a shared commons. Now that Cyrus Omar is settled in, we've been having regular brainstorms about how our earlier ideas on Fairground and Bastion could fit together here.

Tessera shortlisted for an OpenUK award

Tessera has also been shortlisted in the software category of the OpenUK Awards 2026, alongside titans like Cilium, Flux CD and StackHPC. The awards ceremony and gala dinner is at the House of Commons on Thursday, so I'm going to don my tuxedo and practise my Oscars^H^H^Hacceptance speech on behalf of the team (and cry theatrically if we lose, of course).

Papers and whatnots

I finally got back to the Computer Lab and started fixing up my office, and got Cyrus Omar settled in with his card. We discovered the new Cavendish building has an excellent cafe, bringing back the physics tradition of cheap(ish) healthy lunches!

Beatrice Spence and Arissa-Elena Rotunjanu gave a splendid practice run of their ZoologyCon 2026 oral presentation to the Zoology group here in Cambridge. Their talk is on DoME, a low cost and high throughput rig for the 3D digitisation of insect collections, with Anna Yiu, Tiffany Ki and Edgar Turner. The conference runs from 4th-7th November in Bucharest, with their slot on the Wednesday afternoon in case you are attending!

Max Carroll finished his POPL 2027 camera ready changes for his type slicing paper and we submitted those to the portal. I've also been helping Josh Millar with his latest work on test-time adaptation, and Patrick Ferris on next generation shells, so there's been a lot of fun computer science in the past few weeks! A lot of my contribution here is just storytelling these days, while the heavy technical lifting is taken on by the key authors.

Links around the web

This was the week that Patrick Stewart passed away, so I of course switched over to drinking Tea, Earl Grey, hot.

Various discussions about the use of AI in academia rumble on here in Cambridge, with the ever changing landscape being difficult for universities to keep up with. John Palfrey wrote an excellent note on the recent resignation of the Provost of Dartmouth College over undisclosed AI use in his writing:

Given the power of the AI tools that exist today and all the more so given the astonishing speed of recursive self-improvement in the frontier labs, the development of new knowledge is increasingly not being done by human brains, but rather directed by humans and performed by machines.

If I’m right in this regard, then that places even more pressure on setting the right set of rules for academic research and publication — such that we neither limit ourselves unduly in terms of developing new knowledge, nor do we mislead one another or our readers when we fail to disclose the roles we played as human scholars as compared to the machines that have aided us. -- John Palfrey, Nov 2026

My own use of AI varies dramatically based on context (cold turkey ban if trying to learn something, minimal if working on existing projects and mostly for code review, and extensive frontier use for new explorations), so I'm going to put up a web page explaining my personal methods. It changes weekly right now...

References

Madhavapeddy (2025). Foundations of Computer Science. 10.59350/qms3q-ymn65
Madhavapeddy (2025). Thoughts on the National Data Library and private research data. 10.59350/fk6vy-5q841
Madhavapeddy (2025). Socially self-hosting source code with Tangled on Bluesky. 10.59350/r80vb-7b441
Omar et al (2025). A FAIR Case for a Live Computational Commons. Association for Computing Machinery. 10.1145/3759536.3763802
Carroll et al (2026). Bidirectional Type Slicing. arXiv. 10.48550/arXiv.2607.12197

Northflank Security Incident Update (October 30, 2026)

Terrateam Blog — 2026-10-31T00:00:00-00:00

Northflank reported a security incident on October 30, 2026. Terrateam and Stategraph rotated every secret kept in Northflank shortly after and were not impacted. You do not need to do anything.

Migrating Mollymawk to mkernel, mnet, and vif

Robur Cooperative — 2026-10-30T00:00:00-00:00

Moving Mollymawk from MirageOS and Lwt to our new OCaml 5 unikernel ecosystem.

Tessera: A New Era of Open-Source Earth Intelligence from Space

Tarides — 2026-10-29T00:00:00-00:00

We've all seen satellites drifting by at night, and wondered how many are actually up there. With how long they have been observing Earth, it is unsurprising that satellites have accumulated a vast library of images covering the globe. In a world increasingly focused on identifying and mitigating human impact on the natural world, these images constitute an untapped data resource.

Sadly, most conservation groups, field biologists, and local policymakers don't have a supercomputer on stand-by to help them trawl through that data and analyse trends. This is where the AI model Tessera, co-funded by Tarides and developed by a team headed by the University of Cambridge, comes in.

What is Tessera?

Tessera is an open-source AI foundation model which can parse through petabytes of satellite data to reveal patterns and changes over time. Tessera lets scientists designate habitat maps and highlight the changes they want to monitor over time. It can be used to reveal environmental changes, such as crop patterns, habitat loss, and deforestation.

The software processes millions of satellite images from the European Space Agency into a compact, easily usable format for scientists. The result is that, instead of needing a supercomputer, researchers have access to global-scale analyses from their laptops.

Tessera's algorithm learns to understand the 'fingerprint' of a landscape, knowing what it should look like, which allows it to fill in cloud-covered images with the missing values. For each 10-by-10-metre of land on Earth, the software creates an uninterpreted sequence of 128 numbers called 'embeddings'. All of these embeddings are available on demand in the database GeoTessera.

A recent example comes from the Scottish Highlands, where a Cambridge-led team used aerial surveys to build detailed 3D maps showing the beginnings of large-scale native woodland recovery across the Cairngorms National Park. Tarides supported that work, and the data feeds into Tessera. You can read the University of Cambridge write-up.

To learn more about the project, you can read the article published by the University of Cambridge, the open access paper, and watch the webinar on Tessera.

Tessera & Tarides: A Natural Fit with our Work in Space

We are proud to have sponsored Tessera's development, since it is open source and has no paywalls by design. As Tessera's co-lead, Anil Madhavapeddy, professor of planetary computing at Cambridge, puts it: the goal is to "democratise satellite data for people who lack massive computer resources, whether that's an underfunded conservation group here in the UK, field biodiversity researchers in the majority world, or planners figuring out climate adaptation strategies in a rapidly warming world."

Tarides is dedicated to building secure and sustainable software with a positive impact, and supporting the Tessera project is a natural fit. It also complements our work on satellite software. Our sister company, Parsimoni, has been putting OCaml and MirageOS into orbit through SpaceOS, and we've built Notafs, a filesystem that lets Irmin run on MirageOS unikernels aboard satellites. Earlier this year, we also kicked off our part in the CEOS project, building next-generation Earth observation satellites.

Tessera is a piece of that same puzzle; it's one thing to get reliable, secure software into orbit and collect data, and another to make sense of what that data is telling us once it's back on the ground. Whether it's the satellite hardware and unikernels collecting the imagery, or a foundation model turning that imagery into insight for a conservation group with no cloud budget, the focus for us is the same: open tools should make space and Earth science more accessible, not less.

What's Next & Stay in Touch

Explore Tessera's embeddings yourself at geotessera.org, and share your thoughts with the team behind the model.

Stay in touch with us on Bluesky, Mastodon, and LinkedIn or sign up to our mailing list to stay updated on our latest projects. We look forward to hearing from you!

From subject dinners to samosa socials at Pembroke College

Anil Madhavapeddy's Blog — 2026-10-29T00:00:00-00:00

It's been a busy term in Pembroke College, with lots of events happening. I get asked quite often 'what it is that Cambridge Colleges actually do', so here's a sampling of three very different events this term!

Pembroke Computer Scientists subject dinner

We have around four computer science undergraduates per year group, as well as masters and PhD students, which makes around 20 Pembroke students. I'm one of the undergraduate Directors of Studies for the subject alongside the wonderful Carl Henrik Ek and Amanda Prorok. We organise the day-to-day subject supervisions which are a distinctive part of Cambridge study life.

We also held our first subject dinner of the term last week where we convene as a group. For these events we usually invite a guest to give us a brief talk that we then discuss over dinner. This year we are lucky to be hosting Cyrus Omar for the whole year as an Associate Scholar at College, and so he delivered the pre-dinner talk. Cyrus specialises in 'live programming', and he took us through a tour de force of live demos of how to build logics that update as you type them, and also encourage thought rather than requiring a lot of factual recall.

We also congratulated three of our undergraduates who have had papers accepted in international conferences recently. Sophia Chiang published a paper on whether robot-led support conversations align with human therapists, and Arissa-Elena Rotunjanu and Beatrice Spence had a talk accepted on their insect scanning rig at ZoologyCon 2026 in November. Well done everyone!

I was also a little shocked to hear from the undergraduates about their recent experiences applying for internships. Apparently a very large majority of the screening interviews are now with no human interaction and are pure AI-driven coding interviews, often with several rounds of screening before they ever talk to a real person. This seems like a terrible, terrible way for our youth to be spending their time, but they're left with little choice due to how competitive the available roles are.

Pembroke samosa socials in the Reading Room

As the Evidence TAP project picks up speed across Education, Computer Science, Zoology and Plant Sciences, it's been obvious there's a need for inter-departmental dialogues that go beyond our regular group meetings. I therefore contacted our wonderful catering manager Nina Rhodes at College, who offered to supply us with a healthy dose of samosas (everyone loves a samosa) and proper chai as the winter chill sets in.

I then invite all of my colleagues from across the departments on a weekly basis to join Sam Reynolds, Mélanie Gréaux, Sadiq Jaffer and Eleanor Toye Scott -- the senior postdocs on the project -- and me and the other professors to just hang out and get out of our regular comfort zones. I encourage people in the invitation to think about long-term directions for our research across groups, but also short-term things we can do today to help guide the tsunami of change occurring right now due to the impacts of AI on society.

The first four weeks have been successful beyond my expectations! I invited about 80 people, expecting about 10 or so to show up weekly. Instead, the room has been packed with 30-40 people most weeks, with the number of samosas having to be increased to cope. Discussions have been equally wide-ranging: we've had fundamental restructuring of climate change cooperation on the agenda, as well as applying the conservation copilot to our beloved River Cam, to how planetary wikis might reshape policymaking if they were as easy to use as WhatsApp, and also learning that salt marshes are super cool habitats that can be found just next door in Norfolk.

We had representatives show up from the Centre for the Study of Existential Risk, from the Centre for Human-Inspired AI, from archaeology and history and philosophy and even bellringers. It's going so well that I wonder if we need to have recommended reading lists to get people's creative juices flowing even more. And all this conversation is without a hint of any booze, as the chai is working wonders!

Pembroke honorary fellows with Rob Macfarlane and Raj Thakker

And last but not least, we admitted two new Honorary Fellows to College who I had the pleasure to dine with. This is a class of Fellowship granted to former members of College (in this case, both former undergraduates) who have greatly distinguished themselves in their careers. Professor Raj Thakker is a doctor who has advanced the treatment of endocrine tumours, kidney stones and several bone diseases, and made the trip all the way from Oxford for this dinner!

I was delighted to meet Rob Macfarlane, whose book Is a River Alive? was my Christmas reading. It's an incredible dive (pun intended) into the cultural backstory behind almost any body of freshwater, ranging from the Ecuadorian rainforests to Cambridge springs to Australasia and India (Chennai!). Rob was as erudite and effusive and passionate about environmental protection as anyone I've met, and I came out of the dinner triply energised. He is also delivering the mammoth Reith Lectures this year on the four elements, with the fire lecture happening in a couple of weeks. A must-watch!

Colleges are a wonderful little community

We also do bigger events of course, like the meeting of chief scientists earlier in the year or the AI evidence synthesis workshop, but I thought I'd highlight some others here that are a bit off the beaten path.

A lot of these events serve a simple purpose: to let us mingle across generations and expertise, with the flexibility to use our judgement towards both delivering good ideas to the world and the creative destruction of bad ones. It's a rare privilege, and it's why I love my job very much, even if it's completely knackering at this time of year. Speaking of which, time to prepare for the next Foundations of CS lecture tomorrow...

References

Madhavapeddy (2026). Discussing effective conservation with all the UK Chief Scientists. 10.59350/qjrmv-38130
Madhavapeddy (2025). A fully AI-generated paper just passed peer review; notes from our evidence synthesis workshop. 10.59350/k540h-6h993
Madhavapeddy (2025). Foundations of Computer Science. 10.59350/qms3q-ymn65
Omar et al (2025). A FAIR Case for a Live Computational Commons. Association for Computing Machinery. 10.1145/3759536.3763802

Revisiting restarting `pulseaudio`

Richard Mortier — 2026-10-28T00:00:00-00:00

OK, so I have recently again started to have problems with PulseAudio vs PipeWire confusing SwayOSD, again.

But this time the fix seems simpler! Which is nice. Essentially: disable pulseaudio, enable the rest, and that’s it. The runes this time appear to be:

systemctl --user disable pulseaudio.service pulseaudio.socket pulseaudio-x11.service
systemctl --user enable pipewire.service pipewire.socket pipewire-pulse.socket swayosd.service
systemctl --user restart pipewire.service pipewire.socket pipewire-pulse.socket swayosd.service

As ever, YMMV. But that seems to sort it for me, as well as feeling cleaner. Which is, as ever, nice.

OCaml Weekly News, 27 Oct 2026

Caml Weekly News — 2026-10-27T12:00:00-00:00

Notes from week 43

Marc Elvers — 2026-10-26T09:00:00-00:00

ocaml.org memory and CPU improvements; the first Tessera block processed straight into the Zarr store; a CephFS kernel bug that returned zeros for files that were fine; and day10 updates.

ocaml.org

Same symptom, different cause

On Sunday, ocaml.org was serving Backend fetch failed again. The file descriptor leak in gluten that caused the same error on 3 October was fixed and deployed, and it was verifiably not the problem, as the number of open file descriptors was less than 100.

However, the kernel OOM-killed the server three times, with memory usage ranging from 6.2 to 7.3 GB each time. Looking at it early on Monday showed the backends were at 3.5 to 3.8 GB with 290 MB free on a 15 GB host. Each OOM kill is a window during which Varnish returns a 503 until Docker brings the replica back.

top showed that the CPU for each backend process was locked at 100%.

74% of requests were for documentation pages, cache misses outnumbered hits four to one, and the top user agent, a single Chrome-on-macOS string, came from 3,874 distinct addresses, most likely a scraper farm, as we have seen before.

The logs showed that packages without docs created multiple upstream requests. A request for /p/<pkg>/latest/doc/ walked every version fetching status.json from docs-ci. One request for ocaml-system made 65 upstream fetches and took 516 seconds to produce a 404. And every doc page downloaded the package’s whole odoc search index to compute a cache-busting digest.

Profiling the production image

I pulled the exact production image by digest and replayed 1,500 real documentation URLs from Varnish’s log at 32-way concurrency, with Accept-Encoding: gzip exactly as Varnish sends it. That reproduced the main thread at 100% and 2.5 second average latency.

Every documentation page renders the package’s entire module tree in its left-hand navigation, and on 41 sampled pages, that tree accounted for a median of 97% of the bytes. The median page is 492 KB uncompressed, p90 2.7 MB, p99 8.2 MB, and the largest 22 MB; core and frama-c pages are 6 to 8 MB of HTML wrapping about 35 KB of content. The navmap data was cached but re-rendered to HTML on every request.

Per 1,500 requests, the backend spent 82 CPU-seconds with gzip and 52 without gzip, so compressing those pages in pure OCaml on the single Lwt thread accounted for 37% of that time. Of the rest of the time: major GC 27%, memcpy and page faults 32%, a fresh TLS handshake per upstream fetch 8%.

The allocation side was worse: 338 pages in 20 seconds mapped 9.5 GB in large blocks and unmapped 9.5 GB, about 28 MB of allocation per 1.2 MB response, approximately a 23-times amplification of data served. The recursive template returns each level’s string into its parent, so the sidebar is copied once per nesting level, then through Buffer doubling, html_escape, the Dream body and gzip. The runtime image is Alpine, and musl’s malloc mmaps every block over about 128 KB and unmaps it on free, which is where the thousand mmap calls a second came from.

Gzip moved to Varnish

To ease the load on the backend services, I moved the gzip to Varnish by configuring vcl_backend_fetch to drop Accept-Encoding: gzip and setting do_gzip on text responses in vcl_backend_response. Varnish compresses in its worker threads with zlib and stores the gzipped object. Measured locally against the production image that was about 40% off the backend’s CPU. Updating the production server resulted in a 20-30% reduction, indicating that the CPU thread was no longer at 100%.

The sidebar as a fragment

Cuihtlauac opened #3854, memoising the rendered tree per package version. I built it locally and gathered performance metrics comparing it to the unpatched version: CPU per response 54 to 42 ms, a 22% saving, but the memory churn barely moved, dropping from 474 to 426 MB/s, and of course, RSS went up. Caching the tree removes the render but not the five copies of it, and the sidebar still accounted for 97% of every byte the GC had to sweep. I added my analysis on to the PR.

Then I tried another idea: #3858 replaces the tree with a small placeholder carrying the current page’s path and an htmx hx-trigger="load" that fetches the tree from /p/<name>/<version>/doc-sidebar, served with Cache-Control: public, max-age=600. The tree now uses the same HTML for every page of a specific package version, which will be cached by Varnish. The browser reads the current path and uses Alpine.js to expand the branch and mark the current module.

	before	after
backend CPU per page	54 ms	16 ms
average page	1,210 KB	130 KB
large-block churn	474 MB/s	65 MB/s

The fragment itself costs 26 ms and averages 948 KB, once per package version per ten minutes. Cuihtlauac offered to add the backend memo from #3854 on top, but measurements show little benefit at the cost of doubling the RSS.

A fetch cap that leaked a socket

Cuihtlauac also wanted to merge #3767, his fix for #3765 which caps the upstream fetch to 50 MB and adds a HEAD request for the index’s ETag instead of downloading it to hash.

When reviewing it, I noticed that the “response too large” path returned without draining or closing the cohttp body, so every rejected fetch leaked a socket. To prove it leaked, I lowered the cap to 100 KB and requested a selection of pages: 44 rejections, file descriptors from 6 to 50, 44 upstream sockets still open five seconds later. Cuihtlauac pushed a fix with an explicit connection closed on every exit path. Running the same test again showed that the file descriptors held steady.

Grafana dashboard

ocaml.org should report metrics to our Grafana dashboard, but the recent data was absent. This turned out to be an issue with the startup sequence. The node exporter on the host binds the Docker bridge gateway so the containerised front end can reach it. docker0 does not exist until Docker has started, and the systemd unit has no ordering on docker.service. At the last reboot, node exporter started before Docker and exited immediately. I have added the ordering dependence and a Restart= directive.

Memory that grew with every package version

Both PRs went out with Cuihtlauac’s usual morning updates on the 21st. With Grafana now able to provide feedback on memory usage, it was clear that RSS was growing at about 80 MB per hour. The production instance had served 23,472 distinct package versions since the last restart, out of roughly 30,000 that exist. Replaying 3,000 doc URLs locally showed that RSS grew with each page fetched. However, replaying the same 3,000 did not cause a second increase.

The status cache in documentation.ml was a Name.Map of Version.Map holding the parsed status.json for every version ever requested, with a TTL on freshness but no eviction. Each record carried the package’s complete file list, which the code only searches for README, LICENSE and CHANGES. Looking at the worst case, awso lists 115,896 files needing 36 MB of heap.

I replayed all 23,472 versions, one page each, sampling RSS every two thousand, and that is the header chart. The deployed image climbs to 2,809 MB, consistent with the 3.4 to 3.5 GB production reached on the same traffic.

#3861 keeps only the records which are needed and moves the status and search-index caches onto the same Bounded_cache the page and sidebar caches already use. With these changes, the memory usage is around 600 MB.

Cuihtlauac caught something I had missed: package.state is written with Marshal and includes these caches, and I had changed their layout without bumping Info.version, so an old state file would pass the version check and be reinterpreted.

Before and after

Three hours after the final deployment compared with the previous days:

	before	after
main thread busy	55 to 104%	10 to 11%
CPU per response	65 to 95 ms	11 ms
responses over 10 s, of the last 20,000	thousands	0
RSS per backend	3.5 to 3.8 GB, climbing	1.8 to 2.1 GB
open file descriptors	hundreds	13
Lwt pool threads	786	18 to 34
OOM kills	3 in two days	0

Varnish was serving the sidebar fragment from cache about 85% of the time.

`day10`

Post dependencies, built

Shon’s opam-repo-ci#489 from the day revdeps went live was the merlin test suite failing on day10 with Program ocaml-index not found in the tree or in PATH while passing on the main pipeline. The cause was day10 issue #7: the dependency tree used one name set for two purposes, which packages are in the build and which must be built first, and post names were removed from it so that a cycle through a post dependency (ocaml and ocaml-base-compiler name each other) could still be sorted. However, that also removed them from the build. A post dependency reached no other way was solved for and then absent from the switch.

8d1dadd added the package’s own post dependencies to its tree for tests. I measured the solution of every package in opam-repository on both compilers. No build graph changed and nothing moved between solvable and unsolvable. Every test graph that moved belonged to a package declaring a post dependency which is ten on 5.5.1 and eight on 4.14.2. merlin built ocaml-index as a layer beneath it and its tests passed.

1fbc0e5 extended the same treatment of post dependencies to plain builds, because issue #6 turned out to be the same bug. ocaml-option-llvm’s only dependency is ocaml-variants {post}; with that edge dropped, the tree was one node. day10 now solves for and builds ocaml-variants.5.5.1+options, which is what opam does under --update-invariant. 43 build graphs on 5.5.1 and 62 on 4.14.2 change, all for post-declaring packages; frama-c gains its conf-graphviz, and a package depending on frama-c does not. This is a deliberate divergence from OBuilder, which starts from a switch with a compiler already in it, cannot install the option there, and simply reports this as a skip.

A depopt that was not filtered

Hugo opened day10#8 on Thursday morning, and diagnosed it in the report. On the js_of_ocaml 6.5.0~beta2 pull request, every wasm_of_ocaml-compiler,tests job died before building anything, with Assertion failed in topological_sort. These numbered about 44 jobs across the distributions. wasm_of_ocaml-compiler depends on out-channel-redirect {with-test}, and out-channel-redirect has a depopt on wasm_of_ocaml-compiler {with-test}. In opam, with-test is true only for the package under test, so for a mere dependency that depopt evaluates to nothing and there is no cycle. day10 reads depopts through OpamFormula.all_names, which ignores the filter.

I was surprised that this had not come up before, as depends had always been filtered through the package’s environment and depopts never had. Reviewing the 1,345 package versions that have depopts, 747 are filtered, and the fix changes none of them. The first package where this set of conditions occurred was the one added under the PR. This has been addressed in 7dd1238.

Whiteouts lost

A job on doris building conf-libXft.1 on Arch died with

cp: cannot overwrite non-directory '.../lower/var/lib/pacman/local/pkgconf-3.0.7-1' with directory '.../fs/var/lib/pacman/local/pkgconf-3.0.7-1'
day10: internal error, uncaught exception: File "bin/linux.ml", line 238, characters 8-14: Assertion failed

A build’s layers are merged into a single lower directory by hardlinking them in turn. A layer is an overlayfs upper directory, and a path the build removed is recorded there as a 0:0 character device, a special whiteout marker, rather than being absent. The merge carried those markers across as they stood. The issue arises as cp will not copy a directory over a device node, so a layer that removed that name earlier failed at the assertion. When the copy order happened to put the directory first, the marker was silently dropped by cp. This caused pacman to report two versions of pkgconf in its database.

877f330 applies the layers in the order they were installed, base-most first, with each layer’s whiteouts removed from the merge before it is copied and the copied markers dropped again afterwards. I was concerned about a performance hit with this change, so I ran an analysis. Running both find and cp was marginally slower, but the majority of the cost was sudo. However, there was an easy optimisation: cp takes any number of sources and applies them left to right, so a run of clean layers can be done in a single sudo cp, and the markers are found by a single sudo find. Previously, a hundred-layer merge was a hundred sudo cp invocations taking around ten milliseconds each, whereas now it’s just two commands. This has better than halved the time taken for my test layer from 1.70 s to 0.61 s.

Dependents whose own solution holds the target

David asked why the revdeps for ocaml_intrinsics_kernel.v0.17.3 were testing alt-ergo.2.5.1. Both day10 and OBuilder list it as a reverse dependency. This turned out to be a legitimate static dependency through base, ppxlib 0.25.1 and earlier, ppx_blob and alt-ergo-lib. It looks strange as ocaml_intrinsics_kernel.v0.17.3 does not appear in the final day10 solution as ppxlib.0.13.0 cannot install on 5.5, and the solver takes ppxlib.0.39.0, so the job’s actual solution contains neither base nor the package under test.

day10 runs the solve, so could drop these packages. However, after having written that, this changes the coinstallability part of the revdep test.

OBuilder’s revdep job pins the target and then installs the dependent, so it tests that the two can share a switch, including conflicts through shared dependencies. day10’s dispatch installs the dependent alone. Interestingly, the result set is the same.

The way forward here is either to accept that day10 and OBuilder test different things, or, my preferred solution, to make day10 install both packages in the switch, thereby proving the coinstallable result that OBuilder tests.

The `dune` alpha, resolved as build hygiene

ocaml/dune#16617 from last week was closed. The Dune team’s reading is that removing OCAMLPATH from actions improves build hygiene by eliminating dependency ambiguity, and that the failures day10 found are due to misconfigured builds in the respective packages. Shon catalogued hxd, GT and bogue, sent fixes upstream, and bounded their tests to dune < 3.25 in opam-repository.

CI and infrastructure

Three Apple Silicon macOS workers ZFS issues

Three of the four arm64 workers had the same kernel panic as on 1st October, zfs: adding existent segment to range tree, after which the kext is evicted and zfs create obuilder/state fails. The recovery was the same as last time: erase the ZFS volume and rebuild.

Homebrew compiles everything now

Jan noticed ar: libCMakeLib.a: No space left on device on one of the Intel workers. Since Homebrew 7.0.0 in September demoted x86_64 to tier 3, there are no Intel bottles at all, so cmake 4.4.4 was built from source with -flto=thin in /private/tmp. That is the APFS system volume, which is outside of the OBuilder space monitoring.

I cleared the 316 stale source trees across the four Intel workers. ocurrent/macos-infra#62 adds an hourly launchd job that removes Homebrew build directories older than twelve hours, which is safe because the workers run at capacity one, and truncates ocluster.log in place at 128 MB keeping a 32 MB tail.

Ubuntu 26.10

Stonking Stingray was released on 15 October. ocaml-dockerfile#276 adds it at tier 2 and moves the Latest alias to it. Stonking Stingray packages OCaml 5.4.1, bubblewrap 0.12.0, gcc 15.3, and still uutils coreutils, so the GNU coreutils swap from 25.10 has been extended to it. Ubuntu 25.04, 25.10 and Fedora 42 are deprecated as past end of life.

Tessera

17MPU, as shards

A request came in for MGRS square 17MPU in Ecuador for 2017 through 2025, and after the recent difficulty with the NPY->Zarr conversion, I decided to process it directly into the Zarr store rather than producing NPY tiles first.

The square covers 12 shards, 17:246 to 248 by columns 10 to 13, and the pipeline for it uses mtelvers/ferry again. All the processing was done on monteverde: download the d-pixel data for a shard-year with tessera-dpixel --shard, encode it on the L4, upload it with tessera-zarr-upload, one item per shard-year, 108 items.

108 shard-years, 0 failures, 80 hours on the clock because I left it running after it finished.

It started at 13:27 on the 22nd and finished at 14:31 on the 24th, 49 hours, with nothing failed. The averages across the run were 8.6 minutes to download a shard-year, 33 minutes to encode it and 76 seconds to upload.

The 9,899 tile downloads on pima averaged about 306 seconds for 828,000 pixels. A shard-year averaged 8.6 minutes for 16.8 million pixels, which is roughly 12 times the throughput per job. A sub-window sits on the zone’s own UTM grid, so the Sentinel-2 granules are plain windowed reads of the COGs, whereas a 0.1-degree tile has to be reprojected for each date.

Twelve full shards instead of a polygon

The square itself touches 110 of the 192 sub-windows in those 12 shards. The question was whether to process the polygon or the full shards, as writing sub-windows back into the shard later is more work. I decided to process 12 complete shards.

The 100 km MGRS square in red over the twelve 41 km shards in blue. Blue sub-windows are the 110 inside the square; grey are the 82 only in the full shards. The green is Ecuador from the World Bank boundaries, and the coast is 80 km to the west.

The inverse tool

ucam-eo/geotessera will not give you 0.1-degree NPY tiles from a Zarr store; it exports GeoTIFF and marks the NPY tiles as deprecated. The request wanted tiles. So tessera-zarr-tile has been added to mtelvers/ocaml-tessera-tools and is the inverse of the uploader. A tile is a pure integer window of its centre zone’s grid, because the converter pasted each tile’s axis-aligned UTM bounding box at an integer offset, and the tool reads that window and writes it back in the archive’s layout with the SHA256 sidecar.

Reading a window out of a 1.5 GB shard needed partial reads in mtelvers/ocaml-zarr which was added in commit 6ba7b75. The sharding codec reads the index with a single ranged read from the end of the object, then only the inner chunks that the region touches are coalesced into runs and decoded in parallel, with the S3 store issuing its ranges concurrently.

The code was tested with a round trip. Extracting grid_-1.05_51.05 for 2017 from the store and comparing it with the original NPY before it was converted to Zarr showed 824,106 of 824,900 pixels as identical. Every difference was within 22 pixels of an edge, where geotessera later pasted the neighbouring tile. I extended the testing across other tiles and years for confirmation.

The 17MPU request was then served from the store: 100 tiles per year for each of the nine years were downloaded on daintree at about 15 seconds per tile.

The 12 shards of the 17MPU block, 164 by 123 km of Ecuador, one panel per year. Bands 0 to 2 through the store’s 2024 stretch; the reservoir at the top is the Daule-Peripa.

Cambridge machines

Files that read as zeros

James reported four files on /maps that read as zeros. A buffered read of certain 4 MiB objects, the same six indices in all four files, returned zeros, and an O_DIRECT read of the same range returned the data.

To me, this was a critical issue. You need to be able to trust the storage system to return the data you wrote.

So rados get returned the same as O_DIRECT so the data was there. But rados stat showed the affected objects physically short of 4 MiB on the OSDs, by 16 to 48 KiB.

I tried to read the data from another machine, sherwood, and that resulted in the same corruption. I wasn’t expecting this, as much of my investigation had centred on daintree’s memory pressure at the time the files were written. Eventually, I tried another machine, monteverde, and was surprised to find that the files read correctly, but that’s the opening that’s needed in this sort of investigation, as it opens the door to finding the problem.

import os, sys, hashlib

if len(sys.argv) != 2:
    sys.exit(f"usage: {sys.argv[0]} <path-under-cephfs-ec-mount>")

path  = sys.argv[1]
OBJ   = 4 * 1024 * 1024          # CephFS default object_size (= stripe)
SHORT = 16 * 1024                # bytes of stripe-1 we deliberately leave untouched

try: os.unlink(path)
except FileNotFoundError: pass

fd = os.open(path, os.O_CREAT | os.O_WRONLY, 0o644)
os.ftruncate(fd, 2 * OBJ)                            # 2-stripe sparse file
os.pwrite(fd, b'\xaa' *  OBJ,           0)           # stripe 0: full
os.pwrite(fd, b'\xaa' * (OBJ - SHORT),  OBJ)         # stripe 1: last 16 KiB untouched
os.fsync(fd); os.close(fd)

# drop_caches: next read must come from the OSD, not the writer's page cache.
os.system("sudo sync && sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'")

with open(path, 'rb') as f:
    f.seek(OBJ); buf = f.read(OBJ)
fd = os.open(path, os.O_RDONLY | os.O_DIRECT)
os.lseek(fd, OBJ, 0); direct = os.read(fd, OBJ); os.close(fd)

print(f"BUF stripe-1  nonzero={sum(1 for b in buf    if b):>10,}  "
      f"sha={hashlib.sha256(buf   ).hexdigest()[:16]}")
print(f"DIR stripe-1  nonzero={sum(1 for b in direct if b):>10,}  "
      f"sha={hashlib.sha256(direct).hexdigest()[:16]}")
assert buf == direct, "BUG: buffered read differs from O_DIRECT"
print("OK -- reads agree")

The above code reproduces the problem: create an 8 MiB file, write stripe 0 fully, write stripe 1 short by 16 KiB, sync, drop caches so the next read comes from the OSD rather than the writer’s page cache, and compare a buffered read of stripe 1 with a direct one.

On sherwood: buffered nonzero=0, direct nonzero=4177920. Without the cache drop it passes, which is why a fresh-file test looked fine. Running the same script, on the same pool, across four hosts: sherwood on 6.8.0-142, daintree on 6.8.0-63 and kinabalu on 6.8.0-124 all fail; monteverde on 7.3.0-9 and hopper on 6.5.0-1024-nvidia-64k pass.

It is a Linux kernel bug, not a Ceph one. bugzilla.kernel.org #219237, a regression in the generic netfs read path when a subrequest straddles a remote object shorter than the stripe, introduced with the iov_iter rework and fixed by David Howells in September 2024, first released in 6.12. It needs an erasure-coded data pool, a mid-file object physically shorter than its stripe, and a first read after cache invalidation.

The fix for the three vulnerable hosts is the HWE kernel.

Ceph 19.2.3 to 19.2.6

To try to fix the issue above, I upgraded Ceph from 19.2.3 to 19.2.6. The upgrade ran from 07:52 to 18:06. 138 daemons: five monitors, two managers, 86 OSDs, four MDSs.

Clocks out of sync

While watching the Ceph update, I noticed that daintree’s clock was slow by nearly 5 minutes. Sherwood and monteverde were 34 seconds fast, sequoia 21 behind. All four had systemd-timesyncd enabled and pointed at ntp.ubuntu.com, which cannot be reached from the CL network, and had been silently not syncing. Kinabalu, on chrony, and hopper were within half a second. I copied the local time source, ntp1a to ntp1d.cl.cam.ac.uk, from hopper’s configuration to Chrony on the four hosts. I set no makestep, so nothing jumped.

Other

The arm32 nightly

Last week’s note ended with the nightly rebase of the 32-bit backends failing on the Monday morning. The cause was ocaml/ocaml#14980, merged on the 18th, which adds native atomic exchange and compare-and-swap with inline emits for amd64, arm64, riscv and power, and conflicted with the ppc32 changes in the Power emitter. arm32 and i386 have no inline emits for these, so 80ad12a453 routes Catomic_exchange and Catomic_compare_exchange through C on both, the same way caml_atomic_fetch_add already goes, and does them natively on ppc32 with lwarx and stwcx..

That took three more nights to go green. First, the exhaustive match in emit.mlp required that the cases be listed, even though selection never generates them. Then PPC64BE failed because upstream’s configure now requires a C++11 compiler, and the Debian ports worker had no g++. Then it failed again because the freshly installed g++ 16 emits .base64 directives and the worker’s binutils 2.42 does not know them; 2.44 does. Green on the 25th on all four architectures.

Conservation Copilot

Redeployed the conservationevidencecopilot.com project to its VM to update to the latest code changes.

Endeavour transfer re-test

Endeavour upgraded their internet connection, so I repeated the transfer test from September: 512 MiB of random data by rsync over ssh. A single stream is 4.7 MB/s, unchanged. Four streams give 11.0 MB/s, and eight give 17.8, still rising, so the cap is per connection rather than the link. In tiles, that is 9 an hour on one stream and 36 on eight, against the 136 an hour one node consumes.

k7: Keeping CLI Animations in Sync with Your Code

Thomas Gazagnaire — 2026-10-26T00:00:00-00:00

I have always been a fan of asciinema and of Charm's VHS. asciinema records a terminal while you use it, and VHS types a script of commands into a terminal and films the result. I wanted their animations in the READMEs of my projects, but I always forgot the syntax of VHS's tape files, and I always mistyped a command halfway through an asciinema recording (and then started again). I also always forgot to re-run those recordings when my API changed, and they got out of sync quickly.

Fortunately, most of my projects already check their command-line output with cram tests, transcripts of shell commands and of what they print, which dune, OCaml's build system, runs as tests. Making those tests produce animated images was a short stretch, and then, why stop there? The result is k7 (how the French write cassette, if you say it out loud), a command-line tool that records any program, whatever its language, as an SVG image animated with CSS alone.

Here is ls, and sl, which I regularly enjoy mistyping. Each took one command, such as k7 rec -o sl.svg -- sl, which generates an animation you can embed in any HTML page (and in your projects' READMEs).

ls, as I meant to type it.

sl, as I actually typed it.

Animations that dune checks

I was not the first to draw a terminal as an animated SVG. termtosvg (now archived) and svg-term-cli do it too. What k7 adds, I think, is that it both writes and plays dune's cram tests.

A cram test is a text file that reads like a terminal session. Each command is indented and starts with $ , the output it should print sits below it, and any other line is a comment:

Print a greeting:

  $ printf 'hello\n'
  hello

dune test runs the commands of each test in a new empty directory and compares what they print with the file. When the output changes, dune shows the difference, and dune promote writes the new output into the file.

k7 fits into that loop at both ends. k7 rec -o hello.t -- printf 'hello\n' writes the test above (without the comment). In the other direction, k7 play types the commands of a cram test into bash and draws what happens. The image at the top of k7's README comes from doc/demo.t, drawn by k7 play -o doc/demo.svg doc/demo.t. Dune runs the same file as a test, so the test fails exactly when the image needs drawing again, and drawing it again takes one command.

doc/demo.t from k7's repository, typed into bash by k7 play.

The same screen as VHS, every time

k7 rec runs a program on a pseudo-terminal and feeds what it prints to a terminal emulator, matrix, from Thibaut Mattio's mosaic project at Invariant Systems. I wanted k7 to draw the same screens as VHS, which runs xterm.js in a browser, so I fixed the places where matrix and xterm.js disagreed and sent the fixes upstream.

Each time the program changes the screen, k7 keeps a copy of it, at most 50 times a second (as in VHS). Each distinct line of text is defined once, each frame refers to the lines it shows, and a single CSS @keyframes rule slides the strip of frames, jumping to each one at the time the program drew it.

This keeps the file small when most lines stay put: the image of ls above takes 4 KB. A train crossing the screen is the worst case, and the image of sl takes 356 KB. The GIF that VHS made of the same sl, on a terminal of the same size, takes 2.1 MB. VHS drives a headless browser and encodes the frames with ffmpeg, and on my laptop it took 53 seconds, against 8 for k7 rec, about 7 of which were sl itself.

A program like sl has no cram test, since its output is an animation rather than text. For those, k7 rec -o sl.cast -- sl keeps the recording itself, in asciinema's asciicast v3 format, and k7 play draws it, or any asciinema recording, without running anything.

Recording is not deterministic, but drawing is. Record sl twice and the two recordings differ, because the program never prints at quite the same times. Draw one recording twice and you get the same file, byte for byte. So dune can check a committed image against the recording beside it:

(rule
 (target sl.generated.svg)
 (action
  (run k7 play -o %{target} %{dep:sl.cast})))

(rule
 (alias runtest)
 (action
  (diff sl.svg sl.generated.svg)))

A tape syntax the compiler remembers

That leaves the VHS tape syntax I kept forgetting. k7 plays .tape files too, and in OCaml, the K7 library builds tapes as values instead, so the compiler remembers the syntax for me:

let tape =
  K7.tape
    ~settings:K7.[ Font_size 18; Width 1000; Height 560 ]
    ~outputs:[ "demo.gif" ]
    K7.[ type' "dune test\n"; wait Prompt; sleep (Ms 2000) ]

K7.tape checks the tape against the rules of VHS's parser as soon as it is built, theme name included, where VHS only notices a misspelt theme when it runs the tape. Its tests hold it to vhs validate on 3000 random tape lines, as I held my Tailwind rewrite to Tailwind itself.

Trying it

k7 0.9.0 is in my Homebrew tap, for macOS and Linux, and on opam for OCaml 5.4 and later:

$ brew install samoht/tap/k7
$ opam install k7

A few edge cases are not supported (the README lists them), but for most uses it seems to work fine. If k7 draws something differently from VHS or asciinema, please open an issue with the recording (k7 rec -o bug.cast -- yourprogram makes one).

.plan-26-43: Wireguard meshes over troubled water

Anil Madhavapeddy's Blog — 2026-10-25T00:00:00-00:00

Teaching with Forester is going well, while the Patch the Planet week turned up new OCaml issues as other security reports pushed me towards a WireGuard mesh. On the mapping side, I built my first Android app for a field trip, talked human-in-the-loop cartography with a real cartographer, and applauded Tessera at GTC Berlin from home base.

Teaching with FoCS and Forester is a success

The Foundations of Computer Science lecture course is in full swing now, with the Forester interactive toplevels working extremely well (link). I've been switching to them from slides to illustrate points interactively during the lectures or in response to student questions, and it all follows very naturally from the main text.

I think a good next step would be to have more direct Jupyter integration so that the web toplevel can be "upgraded" in place into a persistent session. Cyrus Omar is in Cambridge now for his sabbatical so I can ask him for UI advice on this front!

The student feedback mid-course has also been very positive, and judging from the questions they are grappling with the content manually and not using LLMs, which continues to be a big relief. LLMs are good for summative goals, but terrible for any formative learning experiences without a lot of care going into how they're used, so it's great to see the self-motivation in our Cambridge undergrads.

Patch the Planet week for OCaml

A few weeks ago we set up the OCaml Patch the Planet week at Trail of Bits, which paired security engineers with us for the week of 19th October. The week did turn up several new potential issues in OCaml, and I thought that the process was very smooth and hopefully an indication of the sorts of cross-peer-review we might do more of for critical Internet infrastructure in the future.

We agreed on the scope beforehand of which repos would be scanned, and then set up a private channel on the OCaml Zulip, with Samuel Moelius posting daily updates as he discovered issues. We had lots of help from Török Edwin from Tarides who continuously triaged and guided the findings; Edwin was already fixing Scrutineer results alongside Hannes Mehnert and so was up to speed on the general LLM-security-issue workflow.

Other security reports continue to flood in

Elsewhere, a flood of security reports across various other projects has left me really underwater trying to deal with it all. Can't really even talk about their nature as just a rumour is enough now to trigger exploit generation from bad actors. The PHP vulnerability basically took down half the Internet (including several UK governmental services). One bright spot was the BBC article which fêted the PHP developer who maintains the Debian packaging; this must be the first mainline news piece I can remember celebrating an open source packaging maintainer.

Unfortunately, my inbox continues to be flooded by LLM-generated CVs, requests for internships, and phishing attacks to the point where a legitimate query about working with me is almost certainly getting lost. As Patrick notes, we're going to have to think hard about the nature of open source contributions now and how we choose to engage. I'm not in any way blaming the people using LLMs to apply for jobs and positions (it's a competitive world out there), but the social signals we used to depend on are now obsolete and I'm not sure (beyond personal recommendations) what the best way to fix this is.

Hiding services behind a WireGuard mesh

One thing I did do due to all the security activity is to set up a private WireGuard mesh to take my self-hosted services off the public internet. This is still in a test phase, and I'm not actually using Tailscale for this. Tailscale is a brilliant piece of software, but has fairly complex auth infrastructure to make it work. Instead, since my network is mostly static I'm going very very simple and using WireGuard p2p links directly. I'll write this up more in the coming weeks when things are a bit less mad.

On matters of mapping

I did get a chance to get some mapping related work done in the middle of everything else!

My first Android app for the field

I caught up with Alastair Tse who showed me how to use Claude to build effective design previews (e.g. Material Design) before writing a single line of code. That's then enough to one-shot a full mobile application that runs on my Android! I used it to build a custom app that supports offline labelling for Julia P.G. Jones, who is off on a field trip to Madagascar soon and wanted to try out Tessera on the ground.

Human-in-the-loop cartography

Aneesh Naik, David Coomes and I met Perrin Remonté at the CCI, who builds the most incredible maps for a variety of regions and purposes. We discussed how to inject more human creativity into Tessera-style mapping. One cool dream would be to have dynamically updated maps filled in with real data, but with the artistic vision authored by a cartographer. Human-in-the-loop mapping seems very achievable these days with LLM assistance, letting the human focus on the end goal while the data backfill happens through coding agents.

Validation, however, remains the difficult piece, as we're finding with habitat mapping and processing through vast data sources!

Tessera at GTC Berlin by proxy

Sadiq Jaffer travelled to GTC Berlin to represent team Tessera; I couldn't make it as I've been lecturing solid all week. We had a poster there but also a session on "AI for Science in Action: Using Today's Accelerated Discovery and What's Next" with a lively panel discussion; the video is on the site.

This was a great follow-up to AI for Science 2024 (how fast things have moved in just two years), and all the recent NVIDIA events have given us exciting leads on how to get access to sufficient compute resources to work on the next generation Tessera v3 models that we're cooking up at the moment.

References

Madhavapeddy (2025). Foundations of Computer Science. 10.59350/qms3q-ymn65
Madhavapeddy (2026). Just a rumour of a bug is enough to find a security exploit these days. 10.59350/tngsm-6rx23
Madhavapeddy (2026). Using Forester to turn Foundations of CS into interactive evergreen lectures. 10.59350/sjcvd-hb857
Madhavapeddy (2026). .plan-26-39: End of my sabbatical, and I feeeel fine!. 10.59350/veb7j-vh038
Madhavapeddy (2024). Royal Society and DeepMind host AI for Science Forum. 10.59350/0znpc-fw825

Tracing Dune, with a side of graphs

Ahrefs — 2026-10-21T19:53:28-00:00

The following aims to be a record (and advertisement!) of my recent work on observability in the Dune build system.

A bit about me: I’m Nat, and I’m usually a formal methods PhD student at Imperial College London. This summer I had the privilege of working in Ahrefs’ mid-end team, who focus on improving developer experience of the tools the wider company uses the most.

In my case, this was Dune. Looking over Slack discussions and GitHub issues, the core complaint I observed was difficulty understanding what Dune can see, and why it does certain things at certain times. I started gently with some improvements to dune describe— these are a group of commands for reporting things that Dune knows about a project, such as build rules and external dependencies. Once I had a feel for Dune's internals, I began my main project: a “graph and trace viewer”, to explore the structure of a Dune build alongside its trace.

Some Dune background

For the rest of this to make sense, let me teach you a little about Dune’s internals. Dune relies on key abstractions over the build process; most relevant to us are dependencies and rules. Dependencies are anything that can be required to build something else, and include:

Files, such as source files to compile, input files for tests, etc.
Directories and all the files within
Environment variables, which can affect the build process or test execution
File selectors referring to multiple files & directories
Aliases that expand to other dependencies

Build targets are a subset of dependencies; dune build takes a list of file paths and aliases to be built, falling back to a @default alias. Then we have Rules, which declare a list of targets and, once their dependencies are built, describe the steps to build those targets with an Action.

Let’s suppose we have an OCaml program, sum.ml, that takes numbers from standard input and prints their sum, and we want Dune to build some files using it. We start with some rules — in their most basic form — that declare a target and describe the action to build it; this makes some files with lists of numbers.

; Rule 1
(rule
 (target nums1.txt)
 (action
  (bash "sleep 0.25; seq 5 > %{target}")))

; Rule 2
(rule
 (target nums2.txt)
 (action
  (bash "sleep 0.5; seq 10 10 30 > %{target}")))

Notice how the dune file syntax directly corresponds to the concepts I explained a moment ago; the rule stanza declares a rule that builds the specified target by executing the given action (with no dependencies in this case).

We then have rules to sum those lists, which — as specified with the deps field — depend on the list of numbers used as input, and our sum program.

; Rule 3
(rule
 (target sum1.txt)
 (deps sum.exe nums1.txt)
 (action
  (bash "./sum.exe < nums1.txt > %{target}")))

; Rule 4
(rule
 (target sum2.txt)
 (deps sum.exe nums2.txt)
 (action
  (bash "./sum.exe < nums2.txt > %{target}")))

Then, a rule to sum those results; see how the target, dependencies and action are declared implicitly this time, instead of via the target, deps and action fields.

; Rule 5
(rule
 (with-stdout-to total.txt
  (bash "cat %{dep:sum1.txt} %{dep:sum2.txt} | %{exe:sum.exe}" )))

Finally, we need to tell Dune to build the sum program with the familiar executable stanza. Significantly more things happen implicitly here, with Dune introducing rules and dependencies not just for sum.exe, but for sum.ml, its compiled bytecode, imported libraries, and even the OCaml toolchain itself.

; Rules 6+
(executable
 (name sum)
 (libraries unix))

The build graph, then, is the directed, acyclic graph connecting all the rules and dependencies in the build; for our simple project, it’d look something like this:

These abstractions — rules and dependencies — allow Dune to be remarkably concise and flexible, while offering parallelised, incremental builds. A cost of this, however, is separation between the mental model of a build (as declared in dune files), the true graph of rules and dependencies, and the build’s execution in practice. The build graph quickly grows large, obscuring the why, when and how of each build step.

Proposal: Graph-aware build trace & enhanced Perfetto UI

Here’s my attempt to address this problem: an enhancement of Dune’s existing trace infrastructure, and an extension of the Perfetto trace viewer integrating Dune’s build graph.

The Dune-aware Perfetto trace viewer, exploring the trace of building Merlin.

To use it, run a Dune build with graph tracing enabled and output a Perfetto trace.

$ DUNE_TRACE=+graph dune build @target
$ dune trace perfetto -o my-trace.perfetto

Then, load the result in our modified Perfetto UI. In addition to Perfetto’s standard offerings, this trace viewer provides a number of Dune-specific features, including:

Information about the relevant rule, dependency, or build directory is shown when selecting relevant trace events
Build-graph information can be queried alongside the trace via Perfetto’s SQL capabilities
First-class support for Dune data in Perfetto’s Data Explorer (experimental), to create and reuse complex queries and data visualisations

Check out the screencast below for a guided tour of the interface. For more information (and the source code), check out our forks of Dune and Perfetto.

https://medium.com/media/77fe83a9709599d244d67665239889ea/href

Crossing the desert

Of course, there were many twists and turns in making this tooling a reality, so please enjoy an abridged account of my summer’s work. There are three key areas of implementation: reporting the build graph in Dune’s trace, importing it all into Perfetto, and extending Perfetto’s UI to make it useful.

Emitting traces in Dune

The first step is getting Dune to report the build graph. Rather than creating a new channel for this data, I used Dune’s existing tracing infrastructure, where messages are logged to an internal file and later converted to a readable format. This was generally harmonious with Dune’s internals, though there were a couple of interesting hitches owing to Dune’s tendency to be incredibly lazy: the majority of the codebase uses a monad for delayed execution (to put off work until it’s actually needed) and memoization (to avoid doing the same work twice).

While a dependency build or a rule execution can be required by many dependants, I thought it’d be useful to report the “forcer” of each of these — the single thing that first requested them. This is naturally racy, non-deterministic data, so how can one thread it through the memoized functions at the center of the build engine? As it turns out, the core monad for organising execution is also a reader monad, allowing variables to be set for nested computations; this is mainly used for error handlers, but it fits our purposes nicely, if arguably a bodge. Each rule and dependency declares itself the forcer of anything built under it, only propagating through to operations that haven’t yet been triggered.

When considering failing builds, however, Dune’s laziness really got in the way. Internally, a rule doesn’t list its dependencies upfront: they can only be determined by evaluating the “action builder”, which collects dependencies as they’re required. If building a dependency fails, then the resulting exception prevents the final dependency set from being reported; this severs that target and all its transitive dependents from the build graph, making it practically useless. I’ve come up with a couple of solutions to this, though I’m not particularly satisfied with either.

My first response was to try and recover the dependencies with the action builder’s lazy mode, which in most cases, collects dependencies without building them. The problem is with that “most”, as this creates fringe cases where the lazy pass can progress further than the eager one and trigger a build, meaning the build result would be different when graph tracing is enabled.

My other approach was to introduce a third “eager-reaching” evaluation mode for the action builder, which remembers the result (including exceptions) and the encountered dependencies of memoized steps, reporting the dependencies even on failure. This prevents the overshooting problem, but it can result in a little bit of repeated work if an action builder is evaluated in both eager and eager-reaching mode. Also, call it superstition or impostor syndrome, but making this kind of change to such a core part of dune just has… bad vibes? Maintainers, feel free to tell me off.

Consuming in Perfetto

With Dune reporting the build graph, the next concern is the interchange between Dune and Perfetto. My instinct was to use the JSON Chrome trace output that Dune already supports. This led to some problems, largely stemming from the trace format being quite loosely defined — the only official documentation I could find is a relatively brief Google Doc. The biggest issue is overlapping event durations. The documentation specifies that trace events should only overlap when properly nested, which makes sense for synchronous per-thread traces, but not for something as deeply asynchronous as Dune. The documentation does mention asynchronous events, but Perfetto doesn’t support it when importing from Chrome trace JSON.

So, I thought, why not speak to Perfetto in its own language? That being, its protobuf. It gives more fine-grained control over trace output, integrates better with Perfetto, and, as will soon become relevant, gives more opportunities to optimise performance. The question then became, how? While ample OCaml tooling for protobuf exists, Dune is deliberately light on dependencies as it has to be able to bootstrap itself. On the other hand, given my time constraints, it’s not as though I could hand-roll an encoder myself!

This is where Claude interjected and, despite my scepticism, one-shot an encoder in about half an hour. If a dependency-free Perfetto protobuf encoder is useful to anyone else, I’ve hosted the library separately.

Now that we have all our trace data, complete with the build graph, we can finally start looking at the traces we’ve made…

…in juuust a moment…

…ah.

What was Perfetto doing that whole time? Well, Perfetto tries to process every debug field on every event—which I use to carry build-graph data-for later retrieval — into an “arg set” register. Then, in the Dune plugin, we try to pour all that build-graph data into Perfetto’s SQL engine, in a shape that’s useful to us. My target for this work is Ahrefs’ monorepo, and I shouldn’t have been surprised that Perfetto couldn’t handle a standard CI build trace — weighing in at 2.5GB at this point — without some serious optimisation. We tackle these inefficiencies in Dune, when converting the trace.

First, we scrape the build graph data from individual rule and dependency events — leaving them with the bare minimum to associate with the relevant rule or dependency — and collect it into a “build-graph blob”, which is emitted in chunks on special instant events. This also includes an intern table, cutting down on duplicated strings.

Second, a lot of the data overwhelm is caused by the sheer number of edges in the build-graph; about 2.8 million on my monorepo trace. Many dependency sets tend to be similar, but not quite identical. The obvious response is to delta-encode dependency sets (expressing them as additions and removals to other sets), but this would require a slow recursive query to resolve dependency edges and make reverse dependencies impossible to follow. Instead, we find core sets of dependencies shared between multiple rules, and express the resulting dependency sets as additions to those cores. This gives a 6x space saving; it’s not as strong as delta-encoding’s 20x, but a worthwhile trade to keep edge queries performant.

These changes, combined with string interning at the protocol level and gzipping the result — both of which Perfetto natively supports — cut that trace from 2.5GB to under 90MB, fully loading in about a minute.

Extending Perfetto’s UI

At last, we can think about the interface for actually exploring the data. I find that iterating on UI can be quite time intensive, and I’m not familiar with the Mithril framework that Perfetto uses, so I leaned a bit more heavily on Claude for this part.

The Perfetto UI is an odd beast when it comes to extensibility; it supports plugins, but they must live in the source tree. A lot of the UI was pretty close to what I wanted, but not quite. Some areas (like the data explorer) needed some extra extension points to work, where others (like the SQL query interface) were more economical to just clone with the desired tweaks. I did keep a log of upstream changes for the sake of hypothetical pull requests but, in retrospect, it’d probably make the most sense to stop worrying about the boundary between the Dune plugin and the rest of Perfetto, and turn it into a “Dune trace viewer” hard fork. This is, in fact, just what Jane Street did with magic-trace.

In conclusion

Reflecting on the project, the strongest use found so far is diagnosing parallelism bottlenecks; there’s the lwt example shown in the screencast, but also a real issue discovered in the Ahrefs monorepo. What some colleagues observed as Dune sometimes stalling at random likely turned out to be a bunch of executables linking at once, and throttling the dev server's write buffer (I discovered this when I made a dev server unusable for everyone else one morning 🙂). It was precisely because the graph trace revealed which processes stalled, and what targets their rules were building, that we discovered the root cause.

The hope is, of course, that this work can be more widely useful for understanding and optimising Dune builds. Bear in mind my constraints in this project:

Three months is pretty tight!
While I have some familiarity with build systems, I am by no means an expert in them.
I’m also new to Ahrefs, and a monorepo like theirs will naturally be too nuanced for an intern to get a handle on (I recall one of the onboarding instructions was “Do not even try to understand our system architecture”).

This was what made the Perfetto approach attractive; I could focus on providing all the relevant data — at the scale of the monorepo — and laying the groundwork to explore it for those who come after.

Naturally, these constraints mean there are still some gaps to be filled. A large one is tracing watch-mode and multi-context builds; currently, everything assumes each rule and dependency is handled once. There are also some edge cases for what triggers builds: most happen because of rule dependencies, but there are a number of scenarios like dune files with variable expansion and dynamic-includes. It’d also be nice to perform some amount of headless analysis on a trace: for example, CI could flag when a build parallelises less than a previous one, and point to the rules that might be responsible. Last, and most important, is upstreaming; requiring people to pin their Dune isn’t particularly sustainable, but this work makes some non-trivial changes to a codebase that is, quite rightfully, developed with caution.

I’d like to thank my friend and colleague Sacha Ayoun, not just for his contributions to the OCaml community, but for his unyielding personal support in entering this internship and beyond. Of course, huge thanks go to Ahrefs for giving me this opportunity to work with such an amazing team. I was generously invited to join the company at this year’s OCaml workshop in Paris to meet some of my colleagues in person, and it was wonderful to find some fellow Nix d̶e̶g̶e̶n̶e̶r̶a̶t̶e̶s enthusiasts. Finally, I thank my mentors, Louis Roché and Raphael Proust, for creating an environment in which I felt reassured and empowered to see this project through.

Tracing Dune, with a side of graphs was originally published in Ahrefs on Medium, where people are continuing the conversation by highlighting and responding to this story.

OCaml Weekly News, 20 Oct 2026

Caml Weekly News — 2026-10-20T12:00:00-00:00

opentelemetry 0.92

uniker.nl, our new website for unikernels

Robur Cooperative — 2026-10-20T00:00:00-00:00

A new workflow to develop and deploy unikernels

Proofreading in Vim, the way Word does it

Marc Elvers — 2026-10-19T15:00:00-00:00

I have been trying out Grammarly, but the editors it integrates with are not the ones I use. I want to write Markdown in vim on Linux and have the same features.

So the question was whether there is a basic Markdown editor that integrates with it, or an alternative that does, or whether I could just call the Grammarly API from the editor I already have.

There is no API

I liked the idea of calling an API, but it is the option that has been dead the longest. In July 2023, Grammarly announced that it was shutting down its Text Editor SDK, and switched it off on 10 January 2024. Grammarly for Developers stopped taking registrations before that.

All the Vim and Neovim integrations I could find were built either on that SDK or on the private endpoints the browser extension uses. The language server that nvim-lspconfig used to list is archived, and mason marks it deprecated. There is no official public API, and there has never been a Linux desktop application.

So the only way to get Grammarly on Linux is a browser extension, which isn’t really the answer I was looking for.

Could the machine do it on its own?

The interesting alternative was a local model, mostly because everyone wants to play with a local model!

My machine does not have a GPU, so this was largely a non-starter for me. Generating each token means streaming the whole model’s weights through the CPU, so the speed is set by memory bandwidth rather than by core count. This machine is two Xeon E5-2640 v4s, twenty physical cores, AVX2 but no AVX-512, but with plenty of RAM. A 7B model quantised to four bits is about 4.5 GB, and that much memory traffic per token puts generation at roughly 6 to 10 tokens a second. Reading the prompt is compute-bound instead, and runs at something like 40 to 60 tokens a second.

Put a 150-word paragraph in and take a corrected one out and you are waiting 20 to 40 seconds. A thousand-word document is a few minutes.

That is fine for a deliberate “proofread this before I commit it” but not really the interactive “while you type” feedback that Grammerly offers.

The tool that was already installed

The next question was whether my subscription covers the Claude API, or whether this would just be a case of piping text into the claude executable. It is the latter. A Pro or Max subscription covers claude.ai and Claude Code, which is what the claude binary uses, not API credits.

claude -p --no-session-persistence --tools "" --model sonnet --effort low \
  --output-format text \
  --system-prompt "You are a copy editor. Fix spelling, grammar and punctuation only. \
Do not change meaning, tone, Markdown syntax or line breaks. \
Output only the corrected text with no commentary." < para.md

That took 3.6 seconds on a test paragraph, against my estimate of 20 to 40 seconds that a local model would have taken. It’s important to remember, though, that the text leaves the machine, and every call draws on the subscription’s usage limit. However, I publish everything on GitHub or on my blog so there’s no privacy issue for me.

Back where I started

At that point I had a way to produce corrections and no way to see them. What I actually want is the lazy underlining Word does, and a shell pipe does not give me that. It felt like going round in a circle.

It was not, because the underline layer is a separate problem from the thing that produces the findings, and Vim already has a solution. A standard Vim 9 build has +job to run something in the background, +textprop to attach a highlighted span to a range of text that moves correctly when you edit around it, and +popupwin to show a floating window at the cursor. That is the whole of Word’s interface.

proof.vim

With these components I was able to write proof.vim, 323 lines of Vim9 script in one file, with no plugin manager and no dependency beyond a logged-in claude on $PATH.

cp proof.vim ~/.vim/plugin/

Open a Markdown file and pause. After four seconds idle, which is Vim’s updatetime setting, or whenever you save, the buffer goes to Claude in the background through job_start and each problem span comes back underlined. You carry on editing while it runs. Rest the cursor on an underline and a pop-up gives you the reason and the suggested fix, which is the hover behaviour from Word.

The system prompt asks for nothing but a JSON array, where each element has the faulty text copied verbatim, the replacement, and a reason under twelve words. Asking for the original text verbatim rather than a line and column is what makes the spans survive the fact that you have been typing since the request went out: the plugin finds the string and attaches a text property to it, and Vim then keeps that property in the right place on its own.


`:Proof`	check the buffer now
`:ProofClear`	remove the underlines
`:ProofAuto`	toggle checking on idle and on save
`]g` / `[g`	next and previous suggestion
`gh`	show the reason and the fix
`gy`	accept the fix under the cursor

The settings are g:proof_model (sonnet by default), g:proof_effort (low), g:proof_auto, g:proof_filetypes (markdown, text, tex, plaintex, mail, gitcommit, rst and asciidoc), g:proof_max_bytes (40,000, so it refuses to send a large buffer by accident), g:proof_no_maps and g:proof_highlight.

Underlining text

Vim’s built-in spell checker has been there since version 7, with underlining as the default in GUI Vim. Adding set spell spelllang=en_gb to your .vimrc turns it on. In a terminal, though, Vim’s defaults mark misspellings with solid coloured backgrounds instead, which produced unreadable cyan and pink blocks.

The rules predate undercurl and use solid background blocks: red for SpellBad, blue for SpellCap, cyan for SpellLocal, magenta for SpellRare. For example, the cyan block above was SpellLocal, with the pink underneath being Markdown’s own syntax colouring showing through.

As usual I was working in a tmux window which needed to be told that the outer terminal understands underline styles and colours:

set -as terminal-features ",xterm-256color:usstyle"

vim had to be told the escape codes directly, because the tmux-256color terminfo entry has no undercurl capability at all:

let &t_Cs = "\e[4:3m"               " start undercurl
let &t_Ce = "\e[4:0m"               " stop undercurl
let &t_AU = "\e[58;5;%dm"           " underline colour (256-colour index)
let &t_8u = "\e[58;2;%lu;%lu;%lum"  " underline colour (RGB)

And the four spell groups had to be redefined as coloured undercurl styles with no background, reapplied whenever a colour scheme loads, since loading one resets them.

One last thing was that both grammar and spelling were underlined in red, so you could not tell which was which. Grammar suggestions from Claude are now orange and spelling stays red, with blue for capitalisation, magenta for rare words and cyan for the wrong regional spelling. Vim’s own suggestions are still on z=.

The plugin is available at mtelvers/proof.vim

Notes from week 42

Marc Elvers — 2026-10-19T09:00:00-00:00

day10 reverse dependencies are live and found a dune regression; the aardvark 2017 run finished; and the v2~beta1 Zarr store was rebuilt on a grid that no longer depends on its own data.

Revdeps on `day10`, turned on

Last week I wrote that the opam-repo-ci side of the day10 reverse-dependency work was finished but not deployed. The workers had already been updated to support the revdeps command last week, so this week it should have been a simple case of pushing opam-repo-ci commits from origin/day10 and upstream/live.

However, the update in ocurrent.0.8.0 had raised its constraint to prometheus-app >= 2.0, meaning that current_web.dev did not compile, as CollectorRegistry.collect is now synchronous rather than returning a promise.

Prometheus.CollectorRegistry.(collect default) >>= fun data ->
Error: This expression has type ... snapshot ...
       but an expression was expected of type 'a Lwt.t

My day10 version of opam-repo-ci has ocurrent pinned to proof-of-work PR ocurrent/ocurrent/pull/475 so that needed to be rebased.

How long a build takes

In week 35, I ran a controlled experiment comparing build times across 36 builds on a pair of identical RISC-V workers. With revdeps on, I now have a large-scale test, particularly as there is the dune 3.25.0~alpha0 release opam-repository#30850 open. dune has more dependents than anything else in the repository.

	OBuilder	`day10`
revdep builds	47,356	49,253
distinct packages	14,021	14,491
hardware	11 workers, 328 slots	doris alone, 64 slots
wall clock	20h 50m	10h 31m
summed job time	5,036 job-hours	400 job-hours
median job	242s	2s

The wall clock is the least fair number, as doris was also working through the revdeps for every other open PR at the same time. The summed job time gives the actual time spend much more clearly: 400 job-hours instead of 5,036.

As a second comparison, later in the week, the conf-perl change in #29613 has around 3,000 dependents per compiler, so a single pull request produces about 6,000 day10 jobs. In one hour, doris finished 11,695 of them.

183 differences

Of course, speed is very much secondary to agreement with obuilder. Of the 47,312 debian-13 x86_64 twins, 43,568 passed on both, and 3,297 failed on both. That leaves 264 where OBuilder failed and day10 did not, and 183 the other way.

The 264 are almost all OBuilder infrastructure: 150 are Failed to get sources, a curl that did not complete, 72 are skips, and 13 were cancelled.

The 183 break down as follows:

cause	jobs
tests run without the package installed	88
an optional dependency present at build time	38
`topiary` and a missing `share` directory	20
`OPAMCONFIRMLEVEL` in expected output	6
`post` dependencies absent from the test layer	6
a package reading `%{env:PWD}`	4
newer dependencies from the master merge	5
apt, and a crash	3
unexplained	13

A `dune` alpha that stopped exporting `OCAMLPATH`

The 88 all have the same issue: a test that steps outside _build and asks the switch for the package by name. For example, bogue runs ocamlfind query bogue; hxd’s cram test writes a dune project with (libraries hxd.string) and builds it; GT and ostap run mdx, whose top level does #require; and flint loads dllflint_stubs.so through lib/stublibs.

They fail under day10 and pass under OBuilder because of an ordering difference. day10-install --reinstall removes the old copy and then builds; opam’s own reinstall builds first, then removes, then installs. OBuilder’s job therefore still has the previous install in the switch while the tests run, and day10’s does not. You can see it in the logs: OBuilder prints compiled, removed, installed, and day10 prints installed, removed, then extracts and builds.

Outside of obuilder/day10 what happens in a plain container with just opam?

docker run --rm -it ocaml/opam:debian-13-ocaml-5.5
opam install dune.3.25.0~alpha0
opam install -t hxd.0.5.0     # Error: Library "hxd.string" not found
opam install hxd.0.5.0
opam reinstall -t hxd.0.5.0   # passes

While creating the reproduction steps, I initially used dune 3.24.2 and was unable to reproduce the problem. It was that insight which made me consider whether this was due to dune release. GT and bogue behave the same way.

Up to 3.24, dune set OCAMLPATH=<root>/_build/install/<context>/lib, from make_root_env in src/dune_rules/super_context.ml. In 3.25.0~alpha0, that is gone, replaced by a per-action layout that is only set up for actions declaring (deps (package ...)). One rule shows it:

(rule (alias runtest) (action (system "echo OCAMLPATH=${OCAMLPATH:-unset}")))

  dune 3.24.2:        OCAMLPATH=/tmp/e/_build/install/default/lib
  dune 3.25.0~alpha0: OCAMLPATH=unset

So a fresh opam install -t breaks for eight packages, on their current releases as well as old ones: bogue on all thirteen versions, hxd on nine, plus flint, GT, ostap, embedded_ocaml_templates, ocaml-index and topup. Reported as ocaml/dune#16617.

`opam switch create` does not create `share`

topiary depends on the Rust compiler, not the OCaml one, and its installation does a bare mkdir "%{share}%/topiary". In a switch with no compiler, there is no share directory, because opam switch create does not make one. The compiler build normally creates it as a side effect of writing share/ocaml/build-id, which is why this has never come up: under obuilder the compiler is part of the base image so is always there.

I opened ocaml/opam#7205 to get opam to create share (and etc, which was the only other one missing) when the switch is created. However, the CI went red, because the tests assert share being created and owned by the first package that needs it and reclaimed on removal. I read that as the test documenting deliberate design and withdrew the pull request. Kate’s reply was that the tests only show the state of things at a particular point, not necessarily the intent, and asked me to reopen it.

The other end is opam-repository#30938, which adds mkdir -p to every topiary version. At the moment, a package that needs the share directory should create it.

Other issues

I documented these by creating issues on day10 so the history and reasoning is preserved.

mtelvers/day10#5. OPAMCONFIRMLEVEL=unsafe-yes was exported into the container, where nothing reads it: day10-install authorises the depext sudo itself and installs system packages through OpamSolution.install_depexts rather than the opam CLI. What it did reach was package tests. An opam asked for the 2.0 CLI prints a warning that the variable was ignored, and an expect-style test that captures opam’s output fails on the extra line. That accounted for dune-release and satyrographos.

mtelvers/day10#6. day10 health-check ocaml-option-llvm.1 on an OCaml 5.5.1 base reports [NOTE] success with a dependency graph containing one node, having built nothing. opam, given the same tree, refuses the install outright. It only reproduces against a tree with #29582 merged onto master; on plain master the package has no solution at all. This is currently still open.

mtelvers/day10#7. post dependencies are resolved when day10 solves and then never built. merlin’s test suite calls %{bin:ocaml-index}, which is a post dependency, and gets Program ocaml-index not found in the tree or in PATH. catala needs conf-ninja the same way. Shon opened ocurrent/opam-repo-ci#489 about exactly this the day revdeps went live, which is a fair thing to notice first. This is still open.

mtelvers/day10-install#1. imguiml’s src/dune reads %{env:PWD}, and day10-install execs build commands directly, so PWD is whatever it was at startup. I had assumed opam sets PWD, but it does not. It chdirs and passes the environment through, and plain opam install imguiml.v1.90.6 fails identically on 2.1.6 and 2.5.1. OBuilder only looks correct because bubblewrap’s sandbox.sh is a shell, and a shell resets PWD when it starts. The bug is the package’s, and %{project_root} is what it wants. Closed.

One more thing I have not created an issue for is that opam-repo-ci’s is_compiler_package recognises three names, ocaml-variants, ocaml-base-compiler and ocaml-compiler, so the ocaml-option-* and ocaml-options-* packages, which carry flags: compiler, never get --update-invariant and come out “not available” on every compiler variant. The list is duplicated in the OBuilder spec and in the day10 job builder.

Thousands of directories in `/tmp`

Builds started failing early on with Unix.Unix_error(Unix.ENOSPC, "mkdir", "/tmp/opam-xxx-258365").

day10 didn’t create them, or did it? day10 binds to the opam library and read_command_output builds a stdout path of its own and passes it in, and opamProcess deliberately excludes a caller-supplied stdout from the files it unlinks afterwards, so the log of every such command stays. opam tries to rmdir the directory on exit, but it is never empty, so the rmdir fails and the directory stays.

day10 polls OpamSysPoll for the core count and the OS version before it does anything else, and because Os.default_jobs is a top-level binding, even day10 --help does it. That is two such commands and one abandoned directory per run, and turning revdeps on multiplied the number of runs by a large factor.

Commit 0c0e340 adds the directory as a cleanup resource so a signalled run drops it as well.

opam pin removed

day10 had an opam.template whose only purpose was pinning six opam libraries to a fork, carrying one patch: copy_file preserving the source’s mtime, so a configure script does not come out newer than its configure.ac and send make looking for autotools that are not installed. See ocaml/opam#7196.

The pin is no longer needed, as all the work is done in day10-install; this allows day10 to move from 2.4.1 to 2.6.1. Commit f775e2a.

Tessera

The aardvark run is finished

mtelvers/ferry, which runs the aardvark tile pipeline, has got to the end of it. All the tiles have been built, encoded, shipped and uploaded, virtually all done on a single NVIDIA L4.

v2~beta1 zarr

The last NPY store which needs to be regenerated is v2~beta1. I followed the playbook from v1.1: delete the data and refill it with Fargate spot instances. This appeared to work but there were several zone-years which didn’t exit cleanly. I’ve learnt my lesson, and do not allow Fargate to automatically resume failed jobs. The logs showed:

Stats catch-up for shard (189,7) failed: index 16 is out of bounds for axis 1 with size 16

shard_coords_for_tiles and build_shard_index walk range(col//SHARD, (col+w-1)//SHARD + 1) with no clamp to the grid. When a zone’s grid width is an exact multiple of the shard size, and utm33’s v2 grid is 65,536 pixels, which is exactly sixteen shards, a floor-placed tile on the eastern edge emits shard column 16 when the last valid one is 15. The write path already clips pixels, so the shard contains no data and nothing visible goes wrong, but the coordinate still reaches the statistics mask, causing an exception. That kills the fold mid-update, leaving the count at its -1 sentinel, and every later shard then reports that the statistics need rebuilding.

It is now clamped in commit 742d8ee, and v1 escaped only by geometry: its utm33 grid is 69,632 pixels, which is seventeen shards and not a knife edge.

A manifest a month out of date

I ran an audit on v2~beta1 before I started and again after it finished. Unfortunately, this showed that the manifest.parquet, which is dated 11 September, only lists 305,184 tiles. The bucket holds 373,767. 68,583 tiles uploaded since then had never been converted.

Regenerating the manifest needed geotessera-registry s3scan, which would not run: its endpoint defaults to the read proxy, which cannot serve an anonymous ListObjectsV2. Pointing at the real S3 API, it returned 373,765 tiles.

The grid depended on its own data

Fortunately, the v2~beta1 cost was less than $100, but it’s still annoying to need to rerun it again so soon.

However, there’s a further issue, the existing store can’t hold the new tiles and init_store sized each zone’s grid to the bounding box of the tiles that happened to exist when it ran, so new tiles that extend the coverage invalidate the grid and force a full regeneration. Plus, the preview pyramid is keyed off zone coordinates, an RGB regenerate is needed too. For example, zone 14’s old grid was northern hemisphere only; the new manifest has tiles at about 67 degrees south, overflowing it by 1,037,833 pixels and zone 15 overflows by 777,040, zone 12 by 572,259, etc.

Given the partial state of the v2~beta1 dataset, rerunning it again now would just mean we’d need to rerun it again later the next time the data changed. I can see why it does it like this. It doesn’t create empty space containing just metadata.

5871e3b makes the store deterministic based upon the UTM zones. _compute_zone_grid_deterministic projects the zone’s full six-degree longitude band across the whole range of latitude into the zone’s CRS and snaps to the shard size, so the grid depends only on the zone number and nothing else. Every zone now comes out at 69,632 by 2,002,944 pixels, which is 17 by 489 shards, and it contains all 373,765 tiles across all 60 zones with zero overflow. Empty shards cost nothing, because a sharded store with skip-existing never writes an all-nodata shard. Adding tiles is now an incremental fill.

Transverse Mercator stays well behaved even at the poles, because the band is only three degrees either side of its central meridian, and the data does go to 89.95 degrees south, beyond UTM’s nominal limit.

Both commits are on geotessera#448, which has grown from three commits to five.

Wipe, rebuild

Then the whole v2 store was wiped and rebuilt on the new grid, from the regenerated manifest: 540 fill jobs, nine years by sixty zones, on Fargate Spot. About 250 task-hours of a 16 vCPU, 120GB task at roughly $0.37 an hour, so somewhere near $90 of compute. On demand, it would have been three times that.

The audit I ran afterwards reads the store’s own objects rather than any commit marker, so it says what is actually there.

v2~beta1 after the rebuild: 83,887 scales shards over nine years. Colour is real embedding data, bands 0 to 2, sampled once per 41 km shard. Red is a tile position that has data in some other year but not in this one.

83,887 scales shards, and the embeddings array matches exactly: every zone-year has the same count in both, and no zone-year is flagged as holding shards in one array and not the other. Thirteen shards that the NPY says should be present are not, across seven zones, and the audit names them.

What the picture mostly shows is not the conversion but the delivery. 2017 and 2024 hold 32,767 and 27,674 shards between them, and the other seven years have between 2,367 and 4,300 each. So almost everywhere with data in 2017 is red in 2019, and that is the shape of v2~beta1 itself rather than anything the rebuild did or failed to do.

Fargate performance

I mentioned last week the variable performance I’d been seeing. This week, here are some log snippets from Zone 44 for 2017. After initially progressing well it got spot evicted, and I restarted it:

First restart

    Shards: 1,014 land, 1,014 in store, 0 to write, 54 stats catch-up read(s) (8 workers)
    Zone 44 y2017: 3/54 (6%, 0.0/s, 1m47s elapsed, eta 30m22s)
    Zone 44 y2017: 4/54 (7%, 0.0/s, 2m37s elapsed, eta 32m54s)
    Zone 44 y2017: 5/54 (9%, 0.0/s, 3m56s elapsed, eta 38m33s)
    Zone 44 y2017: 6/54 (11%, 0.0/s, 4m24s elapsed, eta 35m18s)
    Zone 44 y2017: 7/54 (13%, 0.0/s, 4m49s elapsed, eta 32m23s)
    Zone 44 y2017: 8/54 (15%, 0.0/s, 5m57s elapsed, eta 34m12s)

Then nothing further in the log.

Second restart

    Shards: 1,014 land, 1,014 in store, 0 to write, 54 stats catch-up read(s) (8 workers)

Then nothing further in the log.

Third restart

    Shards: 1,014 land, 1,014 in store, 0 to write, 54 stats catch-up read(s) (8 workers)
    Zone 44 y2017: 3/54 (6%, 0.0/s, 2m50s elapsed, eta 48m14s)
    Zone 44 y2017: 4/54 (7%, 0.0/s, 4m41s elapsed, eta 58m38s)

Then nothing further.

Fourth restart

    Shards: 1,014 land, 1,014 in store, 0 to write, 54 stats catch-up read(s) (8 workers)
    Zone 44 y2017: 12/54 (22%, 1.2/s, 10s elapsed, eta 35s)
    Zone 44 y2017: 27/54 (50%, 1.3/s, 20s elapsed, eta 20s)
    Zone 44 y2017: 40/54 (74%, 1.3/s, 30s elapsed, eta 10s)
    Zone 44 y2017: 53/54 (98%, 1.0/s, 55s elapsed, eta 1s)
    Zone 44 y2017: 54/54 (100%, 0.9/s, 58s elapsed)
    Stretch stats: 91,650,006 pixels from 54 shard(s) folded in

Done in under 1 minute.

Two-colouring the zones

The last phase is the global RGB preview. The cross-zone stretch is done: 66 billion pixels sampled, plus 39 per-zone stretches for the zones dense enough to deserve their own. Then each zone is reprojected into the global pyramid, and unfortunately that cannot simply be fanned out across all sixty, because adjacent UTM zones share boundary chunks and two neighbours writing the same seam chunk at once leaves colour gaps along the join. Running it unconstrained and sequential is safe and takes about eighteen hours.

The sixty zones form an even cycle, so they two-colour. All thirty odd zones run concurrently, then all thirty even ones, and no two neighbours are ever in the same wave and completed in around an hour. The coarsening pass that builds the rest of the pyramid has finished since, so v2~beta1 is done.

A root key rollover that broke a `docker pull`

A deployer job failed to pull an image:

dial tcp: lookup registry-1.docker.io on 127.0.0.53:53: server misbehaving

The deployer was fine. The target host runs systemd-resolved with strict DNSSEC validation, which the other deployer targets do not. On 11 October, the root zone began signing its DNSKEY set with KSK-2024, and systemd 249 ships only the KSK-2017 trust anchor and does no automatic key update. Once the cached root DNSKEY expired, at 02:47 UTC on 12 October, every validation chain failed. Go reports the resulting SERVFAIL as “server misbehaving”, which is unhelpful enough to send you looking at Docker.

Putting the DS records into /etc/dnssec-trust-anchors.d/root.positive and restarting resolved fixed this.

Dashboards on status.ocaml.ci.dev

Jon’s ocaml-docs-ci monitoring was a standalone docker-compose stack with its own Prometheus and Grafana. It is now incorporated into status.ocaml.ci.dev: the docs.ci scrape job covers both dill and sage, since the two swap roles, with a host label relabelled from the address, and the dashboards are file-provisioned into their own folder with every query scoped to the job and host.

Fetching 9KB of /metrics from dill takes between 3 and 44 seconds, with the occasional 502, while sage answers the same request in 50 milliseconds. That looks like the daemon’s main loop blocking under load rather than anything in the monitoring.

FreeBSD, under a real load

The ZFS changes in ocurrent/obuilder#220 have now been running for three and a half weeks. The median “package unavailable job” time rose from 74 seconds to 125 on 13 October, then came back down through 122, 107 and 88 to 75 by the 17th.

The change was just reflecting the system load at the time. rosemary had a load average of 44 with seven jails on the 13th and 1.73 million vnodes, and the build store ballooned to 19,755 datasets.

Perhaps we are able to cautious call this fixed?

UID standardisation

Four more accounts moved onto the 30xxx scheme across the EEG machines, leaving only one UID collision. The recurring obstacle is that users think they have logged out, but lingering tmux or screen sessions combined with VS Code processes.

`occ`: a function per section

mtelvers/occ is a C11 compiler in OCaml, written to be read, and sized to be able to build the OCaml runtime and pass OCaml’s test suite. This week -ffunction-sections was added. Each function goes into a section of its own, named .text.<name> exactly as gcc names them, so a linker told --gc-sections can drop the ones nothing refers to. Under -g, the compilation unit no longer has a single contiguous address range, so it records a list of them with DW_AT_ranges in .debug_ranges rather than a low and high address that would span an empty .text.

The more interesting part is what the flag found in occas, occ’s own assembler, since both faults are only reachable by a call that crosses a section boundary.

A call to a function the same file defines locally was being given a PLT32 relocation. Nothing can interpose on a local definition, so gas uses an ordinary pc-relative relocation against the symbol’s section instead. Within a single section, the displacement is calculated at assembly time, and no relocation occurs, which is why this had never shown. 136 of the 239 runtime corpus units differed from gas with the flag on, and none do now. The second fault is on RISC-V, which marks every section holding code with the instruction set it belongs to, whereas occas wrote one mark for the whole file. It writes one per section now, and none at all for a .text left empty.

With the flag on, the test set is identical to gas in all cases on x86-64 and, on RISC-V, in all but one file: the known divergence where gas leaves a function’s size to the linker as an add-and-subtract pair and occas works it out. The whole OCaml tree built with CC="occ -ffunction-sections" gives 1,629 passed, 60 skipped, and nothing failed, which is the same as what it gives without the flag.

The Guide to Maintaining Cloud Infrastructure Compliance

Terrateam Blog — 2026-10-19T00:00:00-00:00

Maintaining cloud infrastructure compliance depends on what happens before the apply, not just what a scanner catches afterward. See what that requires in our guide.

.plan-26-42: Tails of the unexpected

Anil Madhavapeddy's Blog — 2026-10-18T00:00:00-00:00

Term is in full swing, so this week is mostly about lecturing and talking at the CCI, followed by a tour of some of my typical termtime meetings on topics from AI governance, the CRI management team, an NVIDIA AI safety event and samosa socials at Pembroke.

Lectures and term are go!

I'm back in the swing of lecturing again after a year out, and greatly enjoying delivering Foundations of Computer Science once more!!! The use of Forester with interactive toplevels has been extremely useful during the lectures, as I can just tab away from my slides to the web toplevel to illustrate some point, which the students can also follow along with.

A quick poll of the students reveals that a number of them are using the Forester version of the notes in lieu of the PDF version. Thus far, several have also commented to me that they're greatly enjoying avoiding LLMs while learning how to navigate functional programming, which is a great relief to me as an educator!

Chasing your own (list) tail

As happens every year, there's a new and interesting question raised during lectures that I don't anticipate. This year, it's how the OCaml head and tail of a list work.

In lecture 3, I explained how lists are implemented as a singly-linked data structure, starting from a head value and ending in an empty list sentinel. The type of val hd : 'a list -> 'a makes it obvious that the function needs to raise an exception if the input list is empty, since a value of 'a cannot otherwise be constructed. However, the question was why the tail function val tl : 'a list -> 'a list can't just return an empty list instead of raising an exception?

I had to look this one up, and couldn't find any language for which tail([])=[], although GHC does have versions which don't raise. Talking to Simon Peyton Jones, he reckons that we shouldn't have a List.tl function in the first place, and should always prefer pattern matching to deconstruct a list. There were lots of other opinions online too.

MPhil and Part III projects

I've been meeting a steady stream of students about MPhil projects, although I'm right at my supervision capacity with all the Part IIs this year. Still, I've been having a fascinating time hearing from students with a variety of backgrounds and probably won't be able to resist taking a couple on. I suspect they'll be more on the formal verification (Lean/OxCaml) side than Tessera, but do get in touch if you're interested in any of them.

Talking Tessera at the CCI

David Coomes and Sadiq Jaffer and I also gave a really well-attended talk at the CCI covering recent advances in Tessera. This was apparently the most people ever at any of the CRI talks, which is quite the honour! The discussions were as robust as you might expect; there's a huge amount of interest in the upcoming v2 embeddings, as well as change tracking models, and also the application to marine ecosystems.

One key point we tried to settle was how to pronounce 'TEssera' or 'TesSERA', and the audience was invited to either tap their head or rub their cheeks to signal their preference. Unfortunately I forgot which was which but we did get a nice audience picture of it:

Some typical termtime meetings

Since I haven't had much time to write up my coding this week, I thought I'd highlight some really fun different policy meetings I had, since I'm often asked what a 'typical' week looks like. It's never the same during termtime!

AI governance with CSaP

The Centre for Science and Policy here organises 'policy fellow' meetings for academics to interact directly with civil servants from around the world. I had the pleasure of meeting with Keita Azuma from Japan’s Cabinet Secretariat this week.

Keita Azuma is Chief Policy Officer for Emerging Technologies and Global Affairs in the Office of the Cabinet Secretary for Crisis Management, Government of Japan. He leads Japan's strategic engagement with international AI governance, coordinating closely with the United Nations and the OECD, including the UN Office for Digital and Emerging Technologies, UNODA, UNDP and UNU, and served as lead organiser of side events at the STI Forum.

[...] Within Japan's crisis management architecture, he has played a central role in integrating emerging technologies — including satellite imagery, AI analytics and drone-based reconnaissance — into the government's response to large-scale disasters, accidents and security crises, and has worked on countering mis- and disinformation during such events.

Keita and I had a really wide-ranging discussion ranging from the rollout of LLM technologies for policymaking (as we are working on in the Evidence TAP) through to how forecasting models like the latest advances in Tessera might apply to climate adaptation strategies in Japan. I also learnt that Japan is hoping to host the 2028 AI summit after next year's in Switzerland. After my whirlwind Indian AI summit experience earlier this year, I definitely want to attend that one!

The Conservation Research Institute management team

Cambridge is full of acronyms, and none more so than the Cambridge Conservation Initiative (CCI), home of UNEP-WCMC, IUCN, RSPB and of course the CRI. The CRI is the codename for the University members of the CCI, and we have regular meetings about how best to organise the interdisciplinary research. My attendance has been sporadic in the last year as I was on sabbatical (except for the fantastic CRI offsite in Norwich) but I very much enjoyed seeing everyone again!

NVIDIA AI Safety and Security event

Then for something completely different, Sadiq Jaffer and I nipped down to London for an invitational from NVIDIA to discuss their announcements on investments in AI safety in the UK. There was a closed event earlier where we heard a "fireside chat" with Blaise Metreweli discussing the matter with Jensen Huang (sadly, Chatham House rule so no notes on this).

I did get to meet many security researchers there though, including my first contact with the AI Safety Institute who have an excellent red team blog about breaking things. I've been working on a side project about protocol specification that I showed a few people, to encouraging feedback. More on that in future weeks!

Since pictures weren't allowed at these meetings, Sadiq Jaffer and I instead celebrate below at King's Cross' notable achievement of multiple Loo of the Year awards, which is, I'm sure you'll agree, not crap at all.

McSamosas at Pembroke

Last but not least, this term as part of the Evidence TAP project kicking off at Pembroke, I've been hosting 'samosa socials'. We're working across so many departments that many of our collaborators never really get to see each other in person. One of the great things about the College system is that we can quite easily find a room and invite 20-30 people without issue.

Our wonderful catering team led by Nina Rhodes has been supplying a steady stream of samosas, pakoras and chai, and we've made this a regular event in Michaelmas every Monday at 4pm. I expected numbers to dip after the first one, but we ended up with over 40 (!) people from 8 or more departments showing up. Discussions ranged from the AI safety announcement I mention above, to how we might locally help with the Cam rights of the river, to how a policymaking platform with LLMs could be crafted to fit how politicians work (messaging!), and how global cooperative policymaking might be implemented. I am delighted to see many of the Pembroke undergrads attending as well, so I'm hopeful we can get some Cambridge Green Blue activities going next term if the momentum continues to build!

If you'd like to attend this and are local in Cambridge, just drop me a note and I'll send you the details!

Other than this, I've been moving all my home directories to our Ceph cluster to test out if using a network filesystem is actually practical these days. We're trying to get 10Gbps in our Computer Lab offices so that our ragtag collection of GPU/CPU machines can network together at a reasonable speed. On my list next week is to promote the OpenStreetMap/GBIF/iNat mirror that I have on my machine to something more easily accessible via a Wireguard mesh, so that Michael Dales and Aneesh Naik can use it for their habitat mapping work.

References

Madhavapeddy (2026). At the AI Impact Summit in Delhi: people, planet, progress. 10.59350/6vc5q-mbk23
Madhavapeddy (2025). Foundations of Computer Science. 10.59350/qms3q-ymn65
Madhavapeddy (2026). Using Forester to turn Foundations of CS into interactive evergreen lectures. 10.59350/sjcvd-hb857
Madhavapeddy (2025). The Cambridge "Green Blue" competition to reduce emissions. 10.59350/y1g67-aq825

How to run a Terraform audit: The Four-Layer Framework

Terrateam Blog — 2026-10-17T00:00:00-00:00

A Terraform audit has to cover code, run history, state, and backend access, not just login logs. Here's the four-layer framework and the tools that help.

What Is Terragrunt? How It Simplifies Terraform

Terrateam Blog — 2026-10-17T00:00:00-00:00

What is Terragrunt used for? See how it wraps Terraform, kills config duplication, and orchestrates multi-module deployments in real pipelines.

The Changing Craft of Programming with John Crepezzi

Signals and Threads — 2026-10-16T17:35:12-00:00

John Crepezzi has spent his career building tools for other programmers—at Tumblr, at Square, and then at GitHub, where he worked on the merge queue and the early versions of Codespaces. He came to Jane Street to build a code search engine for OCaml. A week into the job, ChatGPT came out, and he’s been working on AI tooling ever since. In this episode, Ron and John talk about how a three-day prototype grew into a firmwide platform for agentic coding, how AI can break the social contract around code review, and how to build tools that support both agent workflows and human understanding.

You can find the transcript for this episode on our website.

Some links to topics that came up in the discussion:

QBasic
Pirates of Silicon Valley
Hoogle — the Haskell API search engine that inspired the original code-search project
GitHub merge queue and Codespaces
Sorbet — gradual typing for Ruby; Hack — Meta’s typed PHP
Active Record
Bubblewrap and passt/pasta — sandboxing tools
seccomp BPF
Claude Code and OpenAI Codex CLI
Model Context Protocol and Agent Skills
Perfetto and magic-trace
Antithesis
Bonsai — Jane Street’s OCaml web UI library
Signals & Threads: Language Design with Leo White

Thought Experiment #1

Thomas Gazagnaire — 2026-10-16T00:00:00-00:00

Yesterday evening I was at the first Thought Experiment, a new series of informal evenings on how "AI is changing the way we code, find vulnerabilities, and prove theorems". It was held at the Paris School of AI, which turns out to be a few hundred metres from the Tarides office, and it was cheerfully organised by Caterina Urban and Karthik Bhargavan, with whom we are sending formally verified post-quantum cryptography to space.

Message sequence charts for agents

Benedikt Bollig (CNRS, LMF) asked whether AI agents can be made to coordinate safely. I read his papers on message sequence charts (MSCs) during my PhD, when I was working on scenario languages built on the same formalism, so I was curious to see where he would take them twenty years later. The answer is agentic workflows. An MSC draws each participant as a lifeline and each message as an arrow between two of them, and that is a good fit for a system of agents: the message-passing structure around the LLM calls is where deadlocks and type-mismatched messages live, and that part can be checked. His paper with Matthias Függer and Thomas Nowak (ISoLA 2026) turns this into a small language in which every choice and every loop is decided by one named participant, who tells the others which branch was taken. A projection then generates each agent's local program from the global chart, deadlock-free by construction whatever the LLM calls return. There is a Python implementation called ZipperGen.

Writing an exchange down once and cutting out each participant's part has a long history, from the realisability of MSCs studied by Alur, Etessami and Yannakakis to the multiparty session types of Honda, Yoshida and Carbone; I spent a few years myself generalising these techniques to more abstract partial-order automata. They stayed in papers because a global specification is long and precise, the proof that an implementation follows it is longer still, and few projects could afford to write either by hand. Over the past year that cost has collapsed, because models now write the verbose parts. So the same ideas are resurfacing twenty years later, with concrete applications, and I find that fascinating.

I have also tried to revive those ideas recently, in a new framework I called ocaml-fsm (it now handles symbolic state machines too, so I should think of another name before a proper release :p). It is early days and very much a prototype, limited to Mazurkiewicz traces for now, with full pomset automata cooking. Used as an oracle, though, it has already found bugs in reviewed and tested code, such as a security association in our CCSDS stack that could be neither created nor deleted. Compared with Benedikt's agents, it works one level down, on the finite control of protocols such as HTTP/2, SSH or the CCSDS security layer, and at the same level in mq, my merge queue, where landing a branch is one description of the flow between the caller, the hook that admits the entry, the run that holds the lock and the resolver that takes its conflicts. The rule is one exhaustive OCaml function. The tool evaluates it over the finite domains of states and inputs, and reports the states the code can never reach and the arrows it can never take, each with the shortest sequence of messages that shows it. Fsm.Global.project does the projection, cutting each party's machine out of one description of the whole exchange, on the boxed-pomset theory Loïc Hélouët and I worked out in 2007 (so the talk felt like familiar ground). A keepalive protocol is written once for both parties, one arrow per message, and the sender of a message the standard leaves optional is the one who decides:

let keepalive ping =
  Fsm.Global.v ~name:"keepalive" ~initial:Fresh ~roles:[ Client; Server ]
    ~pp_role ~controls:[ Fresh; Live; Over ] ~pp_control:pp_phase
    ~arrows:
      [
        Fsm.Global.arrow ~sender:Client ~receiver:Server Hello;
        Fsm.Global.arrow ~sender:Client ~receiver:Server Ping;
        Fsm.Global.arrow ~sender:Client ~receiver:Server Bye;
      ]
    ~pp_message ~error:pp_error ~finals:[ Over ]
    (fun phase (arrow : (role, message) Fsm.Global.arrow) ->
      match (phase, arrow.message) with
      | Fresh, Hello -> Fsm.goto Live ~effects:[]
      | Fresh, (Ping | Bye) -> Fsm.refuse Not_open ~effects:[]
      | Live, Hello -> Fsm.refuse Busy ~effects:[]
      | Live, Ping -> ping ()
      | Live, Bye -> Fsm.goto Over ~effects:[]
      | Over, (Hello | Ping | Bye) -> Fsm.refuse Finished ~effects:[])

Fsm.Global.project then gives each party its own automaton. The client may send a ping or not; the server takes it whenever it comes.

# let () = show_party Client "send ping to server";;
live on send ping to server: goto live
live on send ping to server: reject quiet
latitude on send ping to server in live: Example 5.1, recommended false
# let () = show_party Server "receive ping from client";;
live on receive ping from client: goto live
no latitude

The realisability check runs when the description is built, so a test that constructs it is enough to catch it (the types do not, yet). If accepting the ping could lead to two different phases, the server could not tell from the message which one the client chose, so the constructor refuses the description. The message names the cell, the ping at live, and the two phases, live and over, that the receiver would have to choose between without being told.

keepalive (fun () ->
    Fsm.may ~clause:"Example 5.2"
      [ Fsm.goto Live ~effects:[]; Fsm.goto Over ~effects:[] ])

Proofs on machine code, and the oracle problem

Úlfar Erlingsson (Google Cloud Security) had the best title of the evening, "Petrifying the dungheap: Better software through bookkeeping". His starting point is that finding bugs has become so easy that the software we all build on stands revealed as a dungheap. We see it in the OCaml ecosystem too, from two directions. The frontier models now write OCaml well (a few of us wrote last year about what it would take), so the bug and security reports reaching the OCaml compiler and the libraries around it have become noticeably better, and a report now tends to come with a patch that compiles and a test that fails without it. And Scrutineer, Andrew Nesbitt's agentic scanner, finds plenty: by the end of September a dozen OCaml developers were triaging on Anil's instance and twenty-four fixes had been merged in ocaml-tls, ocaml-x509, mirage-crypto and others. Type safety definitely helps here, but there is still a surprising number of bugs lying around in our nice OCaml libraries.

Úlfar's bookkeeping is for the code that cannot be rewritten in a safer language, because its performance and memory footprint have to stay where they are, and because even its bugs are part of its contract: fix one and something that relied on it breaks. The idea is to prove pre- and post-conditions directly on the machine code, with neither the source nor the compiler in the trusted base, and to use the proofs to find memory-safety bugs in software as it ships. The proofs are generated, which raises the question every generated artefact raises: is the program the proof was built for the program that ships? His answer is the cheapest oracle there is, a byte-for-byte comparison of the generated binaries.

I met the same question porting Tailwind to OCaml, where models wrote much of the code. The first check was bytes against the reference compiler's output, then a CSS-aware diff, because the reference emits spellings that differ without mattering. But a model can change a test and the code it checks in the same commit, so the check that decides has to share no code with either tool. In my case it is Chrome rendering both stylesheets and comparing pixels. Úlfar's bytes and my pixels do the same job. Once a model generates the artefact, the oracle has to sit outside anything the model can touch, and the simpler it is, the easier it is to trust.

AI tools for democracy

Lê Nguyên Hoang (Télécom Paris) closed with "AI tools to democratically govern AIs". He started from the attacks: AI is being used against democracies everywhere, in disinformation and in the recommendation systems that amplify it. His point is that the same tools can make democracies harder to attack, with better recommender systems and tools for building consensus, and Taiwan's g0v civic-tech community was his running example. (He is also behind Tournesol, a participatory recommender.) I am halfway through Plurality by Glen Weyl and Audrey Tang, so the talk landed on prepared ground.

It also made me finish a small thing I had been playing with, a page on French public money that lays out the official figures, each number linking to the Eurostat or Insee cell it came from and the arithmetic in between. It is a visualisation tool and nothing more. I have a few other ideas for this kind of data (one of them is a map of every commune, below), and I will write about them if they become something.

The home page of the public-money page. Each underlined figure opens its source.

The 35,000 communes, each coloured by the indicator per resident furthest above the French average (energy, water, farmland, building materials, commuting wages, jobs, industrial CO₂), year by year from 2006 to 2027.

OCaml now has type recovery: when type checking goes wrong, but keeps going

Tarides — 2026-10-15T00:00:00-00:00

Merlin, the foundational library that underpins the reference implementation of the OCaml Language Server Protocol provides a wide range of practical features, including commands, code completion, and much more. It also implements more subtle mechanisms that are essential to delivering a smooth and responsive editing experience. Indeed, the editing environment for a programming language closely resembles the front-end of its compiler. However, whereas a compiler operates on complete, syntactically valid programs, an editor must continuously cope with programs that are incomplete, temporarily inconsistent, and often ill-typed.

To handle these transient states, Merlin vendors a modified version of the OCaml compiler front-end (primarily the parser and the type checker), extending it with two recovery mechanisms:

Parsing Recovery, which synthesises AST nodes to allow parsing to continue even in the presence of syntax errors.
Typing Recovery, which synthesises typed AST nodes, allowing type checking to continue even when type errors are encountered.

This article will focus specifically on the second mechanism, typing recovery, and how we upstreamed this feature to the mainline OCaml compiler.

Typing Recovery in Action

The recovery mechanisms' behaviour is easy to observe in practice. Consider, for example, the following OCaml program:

let x = "ten"
let y = x + 20
let z = (y + (2 * x)) && true

Here, we are faced with three type errors. First, we attempt to add x and 20, which makes no sense since x has type string. Next, we attempt to multiply x (once again) by 2, which is also invalid. Finally, the expression (y + (2 * x)), which is expected to have type int, is instead used as a boolean value.

When compiling this program, the OCaml compiler reports only the first type error in the compilation unit:

File "lib/test.ml", line 2, characters 8-9:
2 | let y = x + 20
            ^
Error: The value x has type string but an expression was expected of type int

This is a very sensible design choice. Although OCaml has a highly sophisticated type system, the language's pervasive type inference means that a single mistake can easily trigger a cascade of secondary errors, obscuring the compiler's primary goal: ensuring correctness.

However, applying the same strategy in a code editor raises two major (and closely related) issues:

An editor that reported errors one at a time would provide a rather poor user experience.
More importantly, if type checking or parsing stopped at the first syntax or type error, every feature provided by Merlin that relies on compiler structures – namely the parsetree and the typedtree – would immediately become unavailable. This would be particularly frustrating because, while actively editing a file, the program is almost always in a temporarily invalid state, whether syntactically, semantically, or both.

For these reasons, the modified version of the OCaml compiler front-end embedded in Merlin always produces valid ASTs (Parsing Recovery) and typed ASTs (Typing Recovery), allowing the subsequent analysis phases to continue operating on programs that are still being written, whilst collecting errors:

These two recovery mechanisms, although arguably among the most interesting features of the language server, are also one of the main reasons why upgrading Merlin to support a new OCaml release is such a challenging process. This effort is known as OCaml Release Readiness. For now, let's take a closer look at why this is the case by focussing specifically on Typing Recovery.

Adapting Typing Recovery Across OCaml Versions

Without going into too much detail, upgrading Merlin to a new OCaml version involves applying patches coming from two different sources:

Compiler patches, corresponding to the various changes made to OCaml over time;
Merlin patches, which introduce the recoveries.

Because Typing Recovery is deeply intertwined with the various modules involved in OCaml's type-checking process, applying patches from two different sources (with different development cycles) makes version upgrades particularly tedious and introduces two major challenges:

Each upgrade takes time, because the vendored version of the compiler front-end must be adapted while preserving all recovery information and fixing the analysis issues naturally introduced by the evolution of OCaml;
Handling new recovery cases can be tedious for the people responsible for the upgrade process. Indeed, it is common to apply patches from the compiler side without having a complete understanding of their underlying rationale. For example, OCaml 5.5 recently introduced a new feature that was highly anticipated by the community: module dependent functions. This addition changes how function parameters are handled and requires careful consideration of how recovery should behave in these new situations.

For these reasons, Typing Recovery could become unstable (within reasonable scenarios, of course), sometimes introducing small regressions and sometimes lacking support for newly introduced features.

Reducing the Differences Between Merlin and OCaml

Reducing the differences between Merlin's vendored version of the front-end and the one maintained by OCaml is therefore a crucial task. It not only helps reduce Merlin's maintenance burden, but also gives OCaml maintainers more control over how they want recovery to be handled within the type checker.

Of course, this represents an additional responsibility for the maintainers (another one!), but in practice, developers introducing new features in the type checker usually have both a deep understanding of their implementation and a good intuition about how recovery should behave in the presence of errors.

Integrating Typing Recovery into the Compiler

At the time of writing this blog post, the Typing Recovery mechanism has been merged into the compiler. In this section, we will briefly describe its re-design and direct integration into the compiler's type checker.

Typing Recovery has been part of Merlin since its very beginning. Although its implementation relies on very clever encoding, it has aged over time, and its underlying design has become increasingly difficult to understand. Integrating it into the compiler, therefore, provided an opportunity to rethink the mechanism, clean it up, and make its behaviour easier to understand. The goal was to enable a non-intrusive integration, giving maintainers the tools they need to confidently evolve and maintain it in the future.

How Recovery Works, at a High Level

The core of the OCaml type-checker is implemented as a set of recursive functions responsible for typing the different syntactic categories of the language, notably expressions, patterns, type declarations, modules, and classes. When a type error is encountered, these functions propagate an exception that interrupts the construction of the typing derivation. As mentioned above, the compiler adopts a first-failure reporting strategy, reporting only the first encountered error.

To go beyond the first-failure reporting strategy, Merlin instruments the main recursive functions of the type-checker in order to intercept typing exceptions. When an exception is raised, it is caught so that it can be recorded (logged), and typing then resumes at the recursive call level (locally) on the other independent subterms of the program. Thus, even if the typing of a subexpression fails, independent subtrees can still be analysed, which makes it possible to report multiple typing errors within a single analysis.

Resuming recursion allows new errors to be discovered, but it is not sufficient to produce an exploitable Typedtree. When a sub-derivation fails, Merlin synthesises an artificial Typedtree node whose typing judgment is that expected by the context in which it is inserted, while also preserving the fragments of the derivation that could be constructed before the typing failure, even if their attachment to the rest of the tree is no longer known. These nodes help preserve the structure of the Typedtree returned by the type-checker and maintain language server functionality despite the presence of errors.

However, these generic mechanisms are not sufficient to cover all language constructs. Some phases of the type-checker perform preliminary checks before recursively typing their subterms. When these checks fail, recursion is never engaged, and no partial derivation is produced. For example, record expressions:

{ my_fst_field = e1;
  my_snd_field = e2 }

The type-checker first attempts to resolve my_fst_field and my_snd_field. If, for instance, my_fst_field does not exist, it gives up without attempting to type e1 and e2. This strategy is suboptimal: subexpressions are never analysed even though they could contain useful information or independent errors. To address this issue, Merlin delays the error by assuming that the field exists.

Merlin then temporarily adapts the type-checker by considering that an unknown label is not necessarily a fatal error, but may correspond to a field that will be added later. Typing then proceeds under this assumption, which makes it possible to produce partial derivations for subexpressions and to improve the quality of diagnostics.

Another API for Raising Type Errors

As the purpose of Typing Recovery is essentially to be managed by Merlin, we do not want to make it the default behaviour; it can therefore be enabled using a flag (-typing-recovery) passed to the compiler (which will be enabled by default by Merlin).

In order to make error handling explicit, we introduce two functions intended to replace direct calls to raise in the type-checker. When the -typing-recovery flag is not enabled, these functions behave exactly like raise. In Typing Recovery mode, they record the exception in a dedicated structure, which will then be processed by the recovery handler.

Typing_recovery.log_and_raise records the exception before re-raising it. It is used when processing cannot continue locally and captures the general case;
Typing_recovery.log_or_raise records the exception without re-raising it when the context is able to perform local recovery, for example, by synthesising an artificial node or by continuing the typing of other subterms; it captures more fine-grained cases.

This distinction makes explicit the points where recovery is possible and those where the exception must necessarily be propagated. The API guides the implementation of the type-checker and makes recovery decisions visible when adding or modifying typing rules.

Today, we do not have a formal criterion to systematically determine which function to use. The integration into the OCaml type-checker was therefore guided by a large set of unit tests and empirical validation. In practice, the choice between the two functions depends on the expected behaviour of recovery, and is validated by observing the produced diagnostics. This lack of a general criterion aligns with the observations of the paper Merlin: A Language Server for OCaml (Experience Report), which notes that recovery policies would benefit from a more systematic theoretical framework.

Forcing User-Facing Error Reporting

In Merlin’s historical implementation, the selection of exceptions eligible to trigger a Typing Recovery scenario was mainly based on an empirical approach. The reimplementation in the compiler made it possible to identify a simple design invariant: only exceptions intended to be reported to the user (user-facing errors) should be recorded by the recovery mechanism. Conversely, internal type-checker exceptions, as well as simply re-raised exceptions (reraise), must not be captured.

To introduce recovery as a systematic approach, we aim to channel user-facing exceptions so that they are always handled through a pair of functions log_and_raise and log_or_raise. To achieve this, we rely on a property of the OCaml language: exceptions are represented by the extensible type exn. This representation makes it possible to control the introduction of new exception constructors via module interfaces. By making certain exception constructors private in a module signature, it becomes possible to forbid their construction outside this module, while still allowing them to be deconstructed through pattern matching or exception handlers. This technique makes it possible to structure the flow of user-facing errors and to enforce a systematic passage through the associated logging functions.

- exception Error of error
+ module Error : sig 
+   type exn += private In_context of error
+ end = struct
+   type exn += In_context of error
+
+   let log_or_raise err = 
+     Type_recovery.log_or_raise (In_context err)
+
+   let log_and_raise err = 
+     Type_recovery.log_and_raise (In_context err)
+ end

This transformation also makes it straightforward to rely on the compiler to guide the implementation by identifying and handling every possible recovery case individually and provide a precise API to handle recovery with or without resumption.

Closing Words and Acknowledgements

Although there are still some differences between Merlin's vendored front-end and the one maintained by OCaml, this change has already significantly reduced the number of Merlin patches that need to be applied. In the future, this will greatly simplify the process of upgrading to new OCaml versions. We recently provided a Merlin branch that anticipates the 5.6 release and relies on the new Typing Recovery mechanism!

Although the next version of OCaml will expose the -typing-recovery flag to enable this mechanism, the goal is not for developers to use it directly, but rather for it to be leveraged by tools such as Merlin.

Historically, Typing Recovery was designed and maintained by Merlin's two creators, Frédéric Bour and Thomas Refis. It was also maintained over the years by Ulysse Gérard. The integration into the compiler was funded by Jane Street. The review, conducted by Thomas Refis (who contributed greatly to knowledge transfer and design discussions), was funded by The OCaml Software Foundation. We would also like to thank Florian Angeletti for his quick review, guidance, and detailed explanations.

This work is part of the broader effort to reduce the gap between OCaml and Merlin, and it is one of the most significant steps in that direction. We believe that the integration was carried out in a smooth and non-intrusive way, minimising the impact on maintenance work while drastically reducing the difficulty of future version upgrades. This will allow us to focus on delivering the best possible experience and continuously improve the OCaml code editing experience.

Stay Tuned.

Stay In Touch!

Tried out the new update? Connect with us on Bluesky, Mastodon, Threads, and LinkedIn or sign up for our mailing list to stay updated on our latest projects. You can also connect with other OCaml users on Discuss. We look forward to hearing your thoughts!

Improving how Mollymawk handles data

Robur Cooperative — 2026-10-14T00:00:00-00:00

Refactoring Mollymawk's internal data structures and how it persists data.

OCaml Weekly News, 13 Oct 2026

Caml Weekly News — 2026-10-13T12:00:00-00:00

Flex Model: an event-driven feature engineering system

Jane Street Tech Blog — 2026-10-13T00:00:00-00:00

Neural networks are a tool we use heavily in many trading strategies. But raw market data can’t be fed to a neural net as-is. It’s an unstructured input that needs normalization and feature extraction first.

Notes from week 41

Marc Elvers — 2026-10-12T09:00:00-00:00

NPY to Zarr has been completed for v1 and v1.1; day10 revdeps; Tessera tools packaged and failures in bench.ci.dev.

The re-convert that cost too much

Last week’s one-pixel fix was merged, which meant the published Zarr stores needed rebuilding. The zarr/v1 store on source.coop is 540 zone-years, sixty UTM zones across nine years, and it holds both the original NPY and the Zarr. We cannot delete the NPY while the Zarr store has a known issue, but storing both is an additional expense. The original NPY-to-Zarr conversion cost under $500.

On Tuesday afternoon, I launched the full re-convert as an AWS Batch fleet on spot pricing, running the converter with --rewrite-existing-shards.

The problem, which I understood about eighteen hours too late, is that --rewrite-existing-shards is not resumable. A spot eviction does not cost you the remaining work; it costs you the whole zone-year from zero. Light zone-years finished within a single spot session. Dense ones, which is broadly all of 2024 and utm32 to utm37 in all years, never accumulated any progress at all. They were evicted, restarted, evicted, restarted.

So on Wednesday morning, I switched to on-demand at 3.2 times the spot rate, thinking that paying a little more for a few hours would be acceptable, since even the biggest zone-year is only 4 hours. However, there was clearly an additional problem as the jobs didn’t complete on schedule. As the spend increased, and with no end in sight, I killed everything and went to bed.

508 of the 540 were already right

The next morning, before re-running anything, I took stock of the cost and what had been completed. This showed that I had spent roughly five times the original, and I was still only at 365 of 540 zone-years succeeded.

The converter writes the shard data first and computes the statistics afterwards. Scanning the failed jobs’ logs for those whose write bar had reached 100% with no remaining ETA gave 143. Add the 365 that succeeded and 508 of 540 already held correct, seam-fixed embeddings. Only 32 genuinely needed rewriting.

I read the stretch_stats_count chunks from S3 with boto3 and Blosc-decompressed them. Every write-complete slot held -1, the marker that says “statistics not computed”. Every successful slot held a real pixel total.

Looking further, at how --rewrite-existing-shards works, it reads the NPY data, writes the Zarr data in parallel with N workers, then serially rereads the entire dataset it just wrote to compute the statistics. Conversely, writing NPY into an empty store computes the statistics as it writes; therefore, there is no secondary statistics pass.

The statistics build pass can be reached if there is no existing statistics data or marker at all. Resetting a zone-year to stats-only means deleting seven small objects:

utmNN/stretch_stats_count/c/{t}
utmNN/stretch_stats_shards/c/{t}/0/0
utmNN/stretch_stats_sum/c/{t}/0
utmNN/stretch_stats_prod/c/{t}/0/0
utmNN/stretch_sample_count/c/{t}
utmNN/stretch_sample/c/{t}/0/0
utmNN/stretch_sample_scales/c/{t}/0

A non-zero count merges the old sample back in; the sums and products are additive, so the fold adds to what is stored; and a non-empty shard mask sends the fill back down the rebuild branch.

To make the fill resumable, the easiest option was to delete the existing data only in the partially rewritten zones.

The summary was 365 already done, 32 cold refills and 143 stats-only passes. The 32 cold refills ran between half an hour and 4.3 hours each.

The three commits

I opened PR ucam-eo/geotessera#448 to update the behaviour.

The rewrite branch now scans the grid once, gives the writers a non-zero sample cap so they return per-shard statistics, and submits every present shard the rewrite does not touch as a catch-up read in the same pool. The serial backfill at the end of each zone-year is no longer present. The progress line also now ends in a newline rather than a carriage return, which is why hours of stall went unnoticed in a redirected Fargate log.
Shards read back are folded a batch at a time, default 64, so a kill loses at most one batch. A failed read leaves its own shard unmasked rather than condemning the whole slot. The backfill writes a resume marker beside the store, not in it, one object per zone-year, so parallel tasks never contend, and continues from it on a retry.
The pool now hands results back in batches as they complete, for writes and catch-up reads alike, and each batch is folded immediately. An eviction loses one batch instead of everything held in memory.

Stuck jobs

Some jobs that had been running for several hours appeared to stop reporting progress relatively soon after they started. Perhaps a dozen progress lines being produced at a rate of 2-3 updates per minute, then nothing for hours. Fargate reported the job as RUNNING, but why was there no output?

Also, the typical job rate was 0.1 shards per second, but some jobs reported a rate of 0.0 shards per second in the log, with ETAs running into weeks.

In both cases, killing the jobs and restarting fixed the issue, underlining the importance of being able to resume the jobs.

I still do not have a root cause. The working theory is an S3 read or write hanging with no timeout, under roughly 24 jobs times 8 workers against source.coop, but a solo job with no neighbours stalled the same way, which rather spoils that explanation.

I ended up with several bash scripts that reported which jobs were running and could stream the logs. When jobs stalled, I had a script to kill and restart them.

Observability

The direct consequence of my complete lack of trust in the process is mtelvers/fargate.

The left pane is a live table of every job: zone, job definition revision, uptime, seconds since its last log line, state and progress string. The right pane is the selected job’s CloudWatch stream. Every job’s stream is tailed continuously, so the arrow keys flick between logs instantly instead of waiting for a fetch. A running job that has written nothing for more than sixty seconds is marked STALLED in red, and the header counts them. K or Delete terminates the selected job after a confirmation. Jobs that leave the queue stay listed, dimmed, for fifteen minutes with their final status.

The project was written entirely agentically. My prompt asked the agent to read the code base from a couple of existing notty projects and the shell scripts. This was the biggest success story as the working tool was produced in a few minutes and gave me the visibility and the control I wanted.

Both stores rebuilt

The TUI gave me the visibility I needed to get the v1 heading to completion. I started v1.1 in parallel, but this time I deleted one year of embedding data at a time and refilled it with spot instances. As v1.1 is a much smaller dataset, processing the entire year took only about 4 hours. It finished on Sunday.

The header image on this post is v1, read key by key from the store’s own objects. It is 561,754 shards, each one 41 km square:

	shards		shards
2017	63,681	2022	61,794
2018	62,308	2023	61,551
2019	63,284	2024	97,355
2020	62,759	2025	26,919
2021	62,103

Tessera tools

I packaged up my Tessera tools into mtelvers/ocaml-tessera-tools, which includes the updated dpixel for generating v1.1 and v2, along with some new tools for working with Zarr data.

These are packaged in a Dockerfile which also runs under Apptainer on the HPCs. Apptainer essentially allows docker run to be invoked as the user with things like $HOME automatically mounted.

`tessera-dpixel`

tessera-dpixel is an updated port of my dpixel.py, which downloads a year of Sentinel-2 and Sentinel-1 for a region and writes the seven .npy files the encoder consumes.

`tessera-shard`

tessera-shard reads region polygons and lists every shard a selection touches, with the row-major indices of the sub-windows that have land in them. A shard is 4096 pixels square, which is 41 km, and the unit of work; a sub-window is a quarter of that each way and the unit of memory. I tried whole-shard downloads first and measured them: one month of one shard took 4m45s and 9.4 GB of RSS, which extrapolates to 40 to 60 GB for a full year. Fine on a big machine, but useless on an Azure worker, or an HPC machine running multiple instances. 1024 pixels is also about the same unit size as the 0.1 degree tiles produced to date.

`tessera-zarr-upload`

tessera-zarr-upload uploads a complete shard to S3 using mtelvers/ocaml-s3 and mtelvers/ocaml-zarr.

The Zarr library needed three things before it could write to the production store.

The sharding encoder encoded every in-bounds inner chunk, whereas zarr-python skips any inner chunk equal to the fill value. Without that, a coastal shard that is mostly sea carries 16,384 compressed chunks of zeros. The test for this is to re-encode a shard that zarr-python wrote and compare the bytes, but it failed in a way I had not asked about: zarr-python lays inner chunks out in Morton order within a shard, not C order. With both fixes, re-encoded shards are identical.

The encoder was pulling each inner chunk out, element by element, via boxed polymorphic variants. An embedding shard is 2.1 billion elements, so that path is many minutes per shard. Ndarray is now built on flat Bigarray views with kind-specialised loops, with row-wise blits and an equals_fill that checks about a nanosecond per element, and inner chunks are encoded across domains.

32 MiB shard	1 domain	8 domains
encode	146 ms	39 ms
decode	121 ms	50 ms

At the same rate, a real 2 GiB shard is about 10 seconds sequentially and a few seconds on 8 domains.

Zarr_s3.Store implements STORE_WITH_ERRORS over S3.Client. get now returns bytes option, so a missing key is distinguishable from a failure, which it previously was not. Integration tests run against a MinIO container.

S3.Client.put_flow uploads whatever an Eio.Flow.source yields, choosing single PUT or multipart the same way put_file does, and reads parts as concurrency slots free up, so memory is bounded by part size times concurrency rather than by the object. A 2 GiB shard becomes 32 parts of 64 MiB with four running concurrently, which also means each part retries independently. And ?delimiter on list_page and fold_pages turns list_dir into one request per thousand children instead of a walk over every object under the prefix.

Mangroves from ten examples

A question I keep coming back to is what a Tessera parallel to ESA WorldCover would look like, and whether you could ask it something like “find all the power stations, and sort them into coal, solar and nuclear”. The framing I settled on is that “find all X” is not a similarity search, it is classification.

Mangroves make a good test case, because Global Mangrove Watch publishes a worldwide answer key to mark against. The regional experiment took 45 seconds across 7 million pixels, including classifier fits. The global one pulled about 11,000 shards, 66 GB, with no retries.

Scored against GMW on held-out world:

Setup	Precision	Recall	F1
10 positive examples	0.20	0.91	0.33
plus one active-learning round	0.35		0.49
plus a 20 km coastal prior	0.50	0.76	0.60

Aardvark pipeline

mtelvers/ferry is running the aardvark tile pipeline. Over the last 8 days it has completed the following:

stage	count
download on pima	5,874
encoded on monteverde, an L4	3,045
encoded on daintree, an A30	2,028
shipped and uploaded	5,073
failures	105

All 105 failures are because pima was moved from one network to another on Tuesday lunchtime. The encode stage failed to copy the NPY files from pima and simply moved on to the next tile. This is easy to recover later by submitting the work queue again, which will pick up the missing ones.

pima finished downloading on Sunday morning, so the GPUs now set the pace alone. 11,024 tiles in the run, 7,577 uploaded, 3,447 to go.

A benchmark that takes the host with it

bench.ci.dev was down on Monday morning. The service was still running, but ARP failed from a machine on the same segment. I added bench.ci.dev to my Slack alert.

systemd-networkd had failed to configure eno1, so there was no DHCP renewal, and dhcpd deleted the A, TXT and PTR records three minutes later. Only the static Gandi records for bench.ci.dev still resolved. A networkctl reconfigure on the serial console fixed it, and dhcpd put all three records straight back with the identical DHCID.

That evening, my Slack alert reported it as down again: ping fine, TCP connecting, SSH never sending its banner. So I installed a small watcher that samples /proc/stat’s process counter every ten seconds and, during a spike, captures the top process names, the parents with the most children and the newest processes, syncing after each capture so the evidence survives a hard reset.

Overnight, there were three OOM kills, all js_of_ocaml, the largest at 92,874,216 kB, about 97% of the machine’s 92 GiB. Eight minutes after the last of them, dhcpd deleted the DNS records again. That is the chain of events: an OOM storm stops networkd from renewing the lease, the lease expires, and the host drops off the network, which is almost certainly what took it away the first time as well. The journal goes back to 27 September, with no OOM kills at all until 1 October, which aligns with the idea that something had just tipped the balance.

The worker run phase had --cpuset-cpus and no memory limit, and the build phase had neither. These Dockerfiles compile the OCaml compiler. ocurrent/current-bench#503 adds a --docker-memory flag applied to both phases and gives the build the same cpuset the run gets, defaulting to off so it is inert upstream until configured. It also moves CPU slot acquisition earlier, so it is held for the whole job, which limits concurrency at some cost to throughput. Deployed to the host at 32g before the PR was opened, because the deployment was the thing that mattered.

On the same day as those OOM kills, js_of_ocaml also produced 72 rows across nine commits quite happily, so this is a runaway on particular commits rather than a creeping requirement.

Jérôme then fixed it properly upstream the same morning in ocsigen/js_of_ocaml#2495, exponential code growth in inline_recursively. The day after, the host had zero OOM kills and a maximum load of 12.34 against 93.16.

A cached clone without the commit

All the Windows base image builds started failing at step 41 of 79 with fatal: unable to read tree (dd88d077...), across 2019, 2022 and 2025, for a SHA that is perfectly available on every other platform.

The Docker layer holding the git clone step is keyed on the line containing the master pin, which has not moved since 16 September, so the clone layer built on 24 September has been reused ever since. This week, a brand-new 2.6 stanza was added, pointing at a commit made on 1 October, which the cached clone predates. Normally, a pin bump moves master too and busts the layer cache, but this was not the case this time.

Linux was immune for two reasons. It copies the checkout byte-for-byte, including .git/config, using cp, so origin in the copy still points to GitHub, and it runs git fetch before each checkout, so a stale layer heals itself. On Windows, there is no git fetch, and even if there were, it would not help, as the copy is created using git clone, so it would fetch from the file on disk, not the upstream repository.

ocurrent/ocaml-dockerfile#275 sets the remote URL back to GitHub and fetches before the checkout on Windows. Followed by ocurrent/docker-base-images#361, which raises the lower bound to 8.4.5.

Revdeps through `day10`, built but not turned on

Last week, I said I would finish the opam-repo-ci side of the day10 reverse dependencies work. I have, and it is not deployed.

ocluster-worker now brackets the revdeps verb’s output with @@@OUTPUT markers exactly as the OBuilder spec already did, so the existing client-side cutter works unchanged. opam-repo-ci computes the list through that verb, with a three-way split in the day10 detection, a tolerant parse because day10’s own warnings can interleave inside the markers, and the list computed against the merged tree. Revdeps are enabled on the debian-13 x86_64 leaf alone, because OBuilder only runs them on master_distro and x86_64 anyway.

The opam-repo-ci commits are on local branches waiting to be pushed to origin/day10 and upstream/live.

Default ACLs, not setgid

The UID renumbering across the EEG machines made some more progress. /maps had been left out of the whole renumber because it used to be a congo NFS with every user squashed to 400:400, and there was no real ownership to renumber. It has since moved to CephFS, which stores real UIDs.

At the start of the migration, I created a group maps with gid 400. Under congo every file was already owned by that group, so everyone in maps could read everything without anyone having to arrange it. On CephFS, files arrive as 30037:30037, the user’s own personal group, and nobody else in maps can get at them.

The obvious fix is setgid on the directories, but it is only half a fix: it makes new files inherit the group, and with the usual umask 022, they still come out group-readable and not group-writable. Default POSIX ACLs are umask-proof, need no setgid, and go in one place:

setfacl -R -m  g:maps:rwX /maps/
setfacl -R -dm g:maps:rwX /maps/

One thing got in the way immediately. acl was not installed on any of the three machines, though CephFS /maps turned out to be mounted with ACL support once it was.

.plan-26-41: Hello, is it Part II you're looking for?

Anil Madhavapeddy's Blog — 2026-10-11T00:00:00-00:00

Start of term is truly crazy around here, so I'll save longer form thoughts for later in the month when things calm down a little.

Most of my focus was on teaching and administration to prepare for the term. I finished the Cambridge 1A Foundations of Computer Science course notes as a Forest, complete with live in-browser OCaml toplevels. These are now on the course webpage and ready for lectures next week. This may well be the last time this course (now about three decades old) runs in its current form, so I'm a little nostalgic heading into delivering it!

CST Part II projects this year

There is a traditional crush of emails this week every year since the returning third year CST students all have to find projects and propose them in the first week of term. There's no 'clearing service' coordination here, so I usually get around forty requests for meetings. This year will see some particularly diverse and fun projects; I'm glad to see the spirit of computing exploration is still alive and kicking with our undergrads!

Anuraag Datta is designing Scour, a language for finding patterns and rule violations in musical scores, supervised by Richard Sharp. This one wins the 'most original' award, as we'll be "debugging Bach"!
Eden Cheung is building an io_uring IO backend for Lean, taking advantage of the executable nature of code written in Lean4. This is really to explore how we might specify shared memory protocols more mechanically, as I believe they're the future of OS interfaces.
Arya Golkari is porting sPyTial to OCaml as a programmatic diagramming library for ML-style data structures (including module signatures and functors). I must ping Shriram Krishnamurthi about this to get the latest developments on their end!
Rohan Wadhawan is writing a compiler from OCaml to GLSL and WGSL shaders, supervised by Patrick Ferris.
Boris Hall is building an OCaml eDSL and runtime for incremental computation with traceable data types, also supervised by Patrick Ferris.
Matt Occleshaw is writing an embedded OCaml DSL for OpenStreetMap whose queries can separate well known results from probable ones, also supervised by Patrick Ferris. This is a good example of layering a language over a folksonomy that has somewhat shifting semantics as the users decide where to steer it.
Ethan Smith is building an omniscient debugger for beginner Python programmers, supervised by Laurie Gale. We break the OCaml monopoly with some micro-Python here!
Minkyum Kim is building an inverted spatial index for Tessera embedding search, supervised by Sadiq Jaffer. We're aiming for sub-second searches over global embeddings on a single GPU, so there's some optimisation to do here.
Finley Stirk is measuring and handling GPS uncertainty in GBIF point labels for Tessera, supervised by Michael Dales and Aneesh Naik. GBIF is full of 'elephants in the ocean' due to the GPS uncertainty inherent in any observations, so this will hopefully greatly improve our habitat mapping efforts.
Soham Shah is continuing work on autoscaling geospatial computation with Python and Yirgacheffe, supervised by Michael Dales, to make it easier for ecologists to run global-scale analyses without becoming systems experts.

Next week is all about responding to the MPhil students and Part IIIs for their (longer form) projects. Fun times!

Back to the Cambridge community

We had the annual Discover the Conservation Research Institute event, with a completely packed room of new faces and familiar ones. Absolutely delightful atmosphere all around.

This was then followed by the usual three-minute group pitches by everyone, so I grabbed a couple of candid camera shots.

I also had the pleasure of the matriculation dinner for the Pembroke College undergraduates, welcoming our four new arrivals with the traditional selfie. This was followed by a day of Director of Studies meetings with the full cohort to make sure all their projects and supervisions were in order.

And after that, it was over to the Computer Science department for our faculty meetings, which I am now attending again as sabbatical is over!

OCaml's module language

I also snatched a few minutes to jot down my thoughts on why OCaml's module language is a great fit for agentic programming. This is really just an explainer of the interface/implementation split that OCaml has supported for 25+ years, but the post was quite widely read by a new audience who are now using coding assistants and so switching between languages at a rapid pace.

References

Madhavapeddy (2025). Foundations of Computer Science. 10.59350/qms3q-ymn65
Madhavapeddy (2026). OCaml's module language is the perfect fit for agentic programming. 10.59350/g6hbj-qmc76
Madhavapeddy (2025). It's time to go post-POSIX at ICFP/SPLASH 2025. 10.59350/mch1m-8a030
Madhavapeddy (2026). Using Forester to turn Foundations of CS into interactive evergreen lectures. 10.59350/sjcvd-hb857

OCaml's module language is the perfect fit for agentic programming

Anil Madhavapeddy's Blog — 2026-10-09T00:00:00-00:00

I've been doing a fair bit of agentic OxCaml programming in the past year, and consistently finding that OCaml is a perfect fit across the spectrum of languages I've been developing in recently. If you're not familiar with OCaml, this post is a quick guide as to why I think this.

As codebases get larger, a coding agent is better at filling in a 'well-specified hole' than cramming in a million-line codebase into its context window. OCaml has a clean separation between .mli files (an interface definition for a module) and the .ml implementations. I'll follow with with a toy project that has two modules, but the technique scales to million-line codebases. We'll start by writing only the interfaces, then exercise them with a test client, then fill in the implementations. After that we get more exotic and refine with OxCaml modes and finish by speculating where formal proofs could go.

First build up only the module interfaces

We have a module Money that tracks our cash (i.e it shoudl never be negative). If you need help with the syntax then the RWO guided tour may be helpful.

(* lib/money.mli *)

type t

val add : t -> t -> t

val sub : t -> t -> t option
(** [sub a b] is [None] if [b > a]. *)

val to_string : t -> string
(** [to_string m] is e.g. ["£3.05"]. *)

The Book module is a list of deposits and withdrawals.

(* lib/book.mli *)

type t

val empty : t
val deposit : Money.t -> t -> t

val withdraw : Money.t -> t -> (t, [ `Insufficient of Money.t ]) result
(** [`Insufficient bal] carries the balance that was too small. *)

val balance : t -> Money.t

Notice that we don't have any implementations yet, but that the types we have defined can reference each other's module despite this. This OCaml project can be made to compile via a dune directive to suppress the need for an implementation as well.

(library
 (name ledger)
 (modules_without_implementation money book))

Now the magic begins, as we can typecheck our project from just this descrption of how they should work together.

$ dune build @check

This interface compiles without warnings, so it's time to exercise our fledgling interface with a test binary!

Building a test client for this interface

By writing a binary next, we can test if the interface we just defined is sufficiently precise to actually use externally.

(* bin/main.ml *)
open Ledger

let () =
  let ( let* ) = Option.bind in
  let r =
    let* ten = Money.of_pence 1000 in
    let* three = Money.of_pence 305 in
    let b = Book.deposit ten Book.empty in
    match Book.withdraw three b with
    | Ok b -> Some (Money.to_string (Book.balance b))
    | Error _ -> None
  in
  print_endline (Option.value r ~default:"failed")

The first realisation when we compile it is that our interface was too abstract, so we have no way to make a Money.t! This makes the build fail:

$ dune build @check
File "bin/main.ml", line 5, characters 15-29:
5 |     let* ten = Money.of_pence 1000 in
                   ^^^^^^^^^^^^^^
Error: Unbound value "Money.of_pence"

We therefore edit our interface to add the constructor function:

(* lib/money.mli *)
type t

val of_pence : int -> t option
(** [of_pence n] is [None] if [n < 0]. *)

Now the binary typechecks, and fails at linking time complaining that there's no implementation. But because the type checker has passed it, we know that the interface is good enough to be worth implementing!

$ dune build ./bin/main.exe
Error: No implementations provided for the following modules:
         "Ledger__Money" referenced from bin/.main.eobjs/native/dune__exe__Main.cmx
         "Ledger__Book" referenced from bin/.main.eobjs/native/dune__exe__Main.cmx

Fill in the module implemenations one at a time

We can now hand Money to the coding agent with an instruction to read the interface (.mli) files, and start implementing the module implementations in dependency order with tests per module (such as expect tests, which keep the expected output next to the code).

(* lib/money.ml *)
type t = int

let of_pence n = if n < 0 then None else Some n
let add = ( + )
let sub a b = if b > a then None else Some (a - b)
let to_string m = Printf.sprintf "£%d.%02d" (m / 100) (m mod 100)

The dune link error now only complains about Ledger__Book. The agent then moves onto writing the Book module, with similar instructions to only read the interface files.

This then brings up another problem with the interface, as writing Book.empty shows needs a starting balance. At this point the agent uses its context-driven discretion to either use of_pence, or add a helper function to Money:

File "lib/book.ml", line 3, characters 12-22:
3 | let empty = Money.zero
                ^^^^^^^^^^
Error: Unbound value "Money.zero"

If the user (or agent goal) agrees to reassess the interface design, the Book interface gains a zero function.

Stopping agents from taking shortcuts

A tempting shortcut for an agent in Book is to treat money as a plain integer and do the operations directly within that implementation:

(* lib/book.ml *)
type t = Money.t

let empty = Money.zero
let deposit m b = Money.add m b
let withdraw m b = if m > b then Error (`Insufficient b) else Ok (b - m)
let balance b = b

This results in a type error in OCaml though:

Error: The implementation "lib/book.ml"
       does not match the interface "lib/.ledger.objs/byte/ledger__Book.cmi":
       Values do not match:
         val withdraw : int -> int -> (int, [> `Insufficient of int ]) result
       is not included in
         val withdraw : t -> t -> (t, [ `Insufficient of t ]) result
       Type "int" is not compatible with type "t"

The agent can't subtract pence directly, since the OCaml interfaces enforce that only the Money module can perform this operation over a value of that type. Conveniently, the compiler rejects it with a message that's helpful enough for the agent to write the correct implementation from the Money interface.

(* lib/book.ml *)
type t = Money.t

let empty = Money.zero
let deposit m b = Money.add m b

let withdraw m b =
  match Money.sub b m with
  | Some b' -> Ok b'
  | None -> Error (`Insufficient b)

let balance b = b

This now fully builds end-to-end, yay!

$ dune build ./bin/main.exe && ./_build/default/bin/main.exe
£6.95

The great thing about this technique is that it scales to enormous projects with hundreds of mli files, and this agentic workflows allows for a cheap definition of a complex set of interfaces before embarking on the expensive implementations. OCaml's separate compilation keeps build times very fast so we have a quick edit/compile loop.

Refining even more with OxCaml modes

We don't have to stop at just OCaml interfaces though! OxCaml is a language extension from Jane Street that provides mode annotations that can extend this workflow. (If you want to learn more, we ran an OxCaml tutorial at ICFP 2025.)

We can now run an agentic pass to refine our interfaces to have even more checks:

(* lib/money.mli *)
type t : immutable_data

val add : t @ local -> t @ local -> t
val to_string : t @ local -> string

(* lib/book.mli *)
type t : immutable_data

The immutable_data annotations promises that a Money.t or Book.t contains no mutable state in its implementation, so it can (e.g.) be shared freely between parallel threads.

The @ local ensures that a function doesn't hold onto its argument, so the caller can pass in a stack-allocated value and not have to have heap allocations.

An agent that decides to "optimise" Book with a mutable balance now fails:

type t = { mutable bal : Money.t }

Error: The implementation "lib/book.ml"
       does not match the interface "lib/.ledger.objs/byte/ledger__Book.cmi":
       Type declarations do not match:
         type t = { mutable bal : Money.t; }
       is not included in
         type t : immutable_data
       The kind of the first is
           mutable_data with Money/2.t @@ forkable unyielding many
         because of the definition of t at file "lib/book.ml", line 1, characters 0-34.
       But the kind of the first must be a subkind of immutable_data.
       The first mode-crosses less than the second along:
         contention: mod uncontended ≰ mod contended
         visibility: mod read_write ≰ mod immutable

This error is admittedly a little opaque to a human user (something that's being worked on in OxCaml), but it's fine for an agent with an OxCaml skill. (I run these agents in a sandboxed devcontainer.) Crucially, these mode annotations helped to stop an agent introducing a subtle error that may have corrupted data when used across multiple processsors.

Going deeper down the refinment rabbithole

All the OxCaml modes earliy are statically defined by the compiler, which is getting increasingly capable. The ICFP 2026 mode crossings paper this summer shows how the compiler automatically strengthens modes for values of certain types, which is how our Money.t can declare immutable_data succinctly. But wouldn't it be cool if we could also express arbitrary logical conditions in the interfaces?!

Here's a sketch of Money using a Gospel-style specification. I've not actually compiled this one, but you'll get the idea:

(* lib/money.mli *)
type t
(*@ model pence : integer
    invariant pence >= 0 *)

val sub : t -> t -> t option
(*@ r = sub a b
    ensures match r with
            | None -> a.pence < b.pence
            | Some c -> c.pence = a.pence - b.pence *)

The comment is now a machine-checked contract, so every implementation must guarantee it satisfies these pre- and post-conditions.

About two decades ago, Patrick Rondon and Ranjit Jhala worked on a liquid OCaml that had these features (PLDI 2008 paper). I'm really excited that it's heading back into modern OxCaml as I've been jealous of Liquid Haskell for a long time :-)

The beautiful thing about using OCaml's module system as a basis for these formal extensions is that separate compilation architecture I sketched above means that the edit/feedback loop is fast even on million-line codebases. The layering of annotations also lets us make code progressively more specified without piling on huge numbers of unit tests.

This is context efficient for agents and preserves human sanity as code gets more complex. We're also only beginning to investigate how to visualise such constraints in our user interfaces, like the work ongoing in Hazel and our own work on bidirectional type slicing to debug type errors (also this last paper just got conditionally accepted into POPL 2027, which I'm super excited about and will wrote more on later!)

References

Madhavapeddy (2025). Cresting the OCaml AI humps. 10.59350/nn1d6-xgt62
Madhavapeddy (2025). Holding an OxCaml tutorial at ICFP/SPLASH 2025. 10.59350/55bc5-x4p75
Carroll et al (2026). Bidirectional Type Slicing. arXiv. 10.48550/arXiv.2607.12197
Lorenzen et al (2024). Oxidizing OCaml with Modal Memory Management. 10.1145/3674642
Rondon et al (2008). Liquid types. 10.1145/1379022.1375602

Zostera, our new implementation of WireGuard in OCaml

Robur Cooperative — 2026-10-08T00:00:00-00:00

How to implement a VPN in OCaml

An Overview of OCaml Release Readiness: How Does the OCaml Ecosystem Stay Up-to-Date with the Compiler?

Tarides — 2026-10-07T00:00:00-00:00

What is the OCaml ecosystem doing to make upgrading to the latest version of the language seamless? When the OCaml compiler receives a new update, it has the potential to have wide-ranging effects to the ecosystem of tools that surround it. In the past, users have hesitated to move forward with the latest release due to proper tooling support lagging behind. To improve the user experience and make transitions smoother, OCaml developers and maintainers agreed to work together to ensure that the core tools were always ready near release time.

This effort is referred to as ‘OCaml Release Readiness’. It is a complex task spread across many actors in the ecosystem and supervised by Florian Angeletti (Inria) and Kate Deplaix (OCSF).

In this post, we will give you a behind-the-scenes look at the work that goes into getting the OCaml ecosystem ‘release ready’.

What are the ‘Core Tools’?

There is no definitive rule for which tools are part of the release readiness effort, but, as a general guideline, it includes the kind of tools that provide key support in projects. If an update is incompatible with a core tool, it will be too disruptive for the user to migrate to the latest compiler version.

Let’s look at the tools that are currently part of the release readiness cycle and what the maintainers do to get them up-to-date with new releases.

Ppxlib

Ppxlib is a library for building pre-processors in OCaml. Pre-processors are called at compile time and alter the program before compilation. They can perform a whole host of helpful functions, such as including a file, generating boilerplate, or extending the language. More or less half of the OCaml ecosystem depends (transitively) on Ppxlib.

Internally, ppxlib has a copy of every abstract syntax tree (AST) of every supported compiler. Currently, that is every compiler version from 4.08 to 5.5. This is necessary because ppxlib has to migrate the ASTs of every user to a common AST (5.2 at the time of writing) to perform the pre-processing, and then migrate back to the AST of the user for the rest of the compilation process.

For each new OCaml release, a new AST corresponding to the latest version is added to the migration chain, and transformations are written to and from that AST. If this AST includes new syntactic features or new representations of existing features, then the complexity of the task grows. The maintainers need to write encodings that preserve the new features and representations, particularly in the migrations to and from older ASTs.

New Ppxlib releases are prepared by Nathan Rebours, via the OCaml Software Foundation (OCSF) and OCamlPro, and by Patrick Ferris at Tarides.

Merlin

Merlin is an editor service for OCaml that supports advanced IDE features. It is an analysis tool that provides semantic information about OCaml code, communicated to editors via a protocol such as LSP.

A cornerstone of its architecture is the ability to recover from parsing and typing errors, so it can keep providing useful information on lines below the error rather than stopping. Achieving this requires patching large parts of the compiler's parser and typechecker in non-trivial ways, which makes upgrading Merlin's vendored parser and typer to each new compiler version a complex, time-consuming task.

Several ongoing upstreaming efforts should make future updates much easier, like type-recovery which is scheduled to be upstreamed in the 5.6 version of the compiler, and the open proposal to upstream Merlin's new -short-paths implementation directly into the compiler as well, removing a long-standing source of drift between the two.

The release readiness work for Merlin is undertaken by a team at Tarides.

OCaml-LSP

OCaml-LSP is the LSP frontend for Merlin. It has only minor dependencies on the compiler, so once Merlin's upgrade to a new OCaml version is done, upgrading OCaml-LSP is usually straightforward.

The release readiness work for OCaml-LSP is done at Tarides.

Dune

Dune is a composable build system for OCaml. Its support for both monorepos and multi-package repositories has made it popular among OCaml developers. Its main branch tracks the latest compiler versions, and because it sits so early in the dependency graph, failures are quick to surface. In practice, the problems that come up are about compiling Dune itself with a newer compiler and not about the projects it builds. Strictly speaking this isn't even required, since a Dune binary built with an older compiler can still build projects with a newer one. But opam constrains a switch to a single compiler version for all packages in it, so in practice Dune has to stay compatible with the latest compiler releases.

The release readiness work for Dune is done by a team at Tarides.

OCamlFormat

OCamlFormat parses and pretty-prints OCaml code. We need to update it every time new syntax is added to the language, which surprisingly happens in almost every OCaml minor release. OCamlFormat vendors two versions of the OCaml parser, so upgrading requires backporting patches with Git and resolving conflicts. We then update and test the printing code. The first parser is used purely for verification: OCamlformat parses the code before and after formatting, compares the two ASTs, and refuses to format if they diverge. The second parser is extended to keep more information about the concrete syntax. For example, these two snippets would have the same AST with OCaml's parser: [ 1; 2 ] and 1 :: 2 :: []. They are encoded differently with our extended parser.

The release readiness work for OCamlFormat is done at Tarides.

Odoc

Odoc reads compiler-produced artifacts containing the abstract syntax tree of compilation units in order to generate module documentation. Since the type of those ASTs evolves between OCaml versions, odoc defines its own internal AST to be independent of the compiler version. Odoc uses cppo to adapt how the compiler's AST is loaded into odoc's internal representation at compile time.

When a new OCaml version introduces changes to the typedtree, odoc fails to compile until it's adapted. At minimum, this means updating the loading logic to handle the new types while staying compatible with older versions via cppo conditionals. Beyond that, odoc's internal AST may need to be extended to represent new features, and add the logic to support rendering the changes in the generated API documentation when applicable.

The release readiness work on odoc is done at Tarides.

Stay in Touch

Did you know how much work goes on behind-the-scenes to keep the OCaml ecosystem’s tools up-to-date with new compiler versions? Connect with us on Bluesky, Mastodon, Threads, and LinkedIn or sign up for our mailing list to stay updated on our latest projects. We look forward to hearing from you!

Terragrunt Alternatives: 6 Tools Compared for 2026

Terrateam Blog — 2026-10-07T00:00:00-00:00

Looking for Terragrunt alternatives? Six tools compared with honest pros and cons, plus how Stategraph works alongside Terragrunt or replaces it.

OCaml Weekly News, 06 Oct 2026

Caml Weekly News — 2026-10-06T12:00:00-00:00

Building and testing a config management orchestrator for Windows

Jane Street Tech Blog — 2026-10-06T00:00:00-00:00

The following is part of a series of posts about 2026 summer intern projects—for more, see “What the interns have wrought, special jumbo 2026 edition”

Notes from week 40

Marc Elvers — 2026-10-05T09:00:00-00:00

Fast Tessera inference that brings one Xeon node level with a GPU; reverse dependencies in day10; Windows, macOS and FreeBSD worker updates and a new ad hoc pipeline engine.

Tessera inference on Endeavour

Intel gave us time on Endeavour to see whether AMX can compete with a GPU for Tessera inference. Last week, I was pleased with 56 tiles per hour per node against around 150 for one MI355X, but Sadiq said it should be better.

Building uxlfoundation/oneDNN and running benchdnn directly against the model’s actual shapes showed torch was passing weights in plain ba layout rather than the AMX-blocked BA16a32b2a, costing 1.7 to 1.8 times per call. The GEMMs were running at the PyTorch ceiling, where the real ceiling is about 203 TF rather than 121. This showed that, while I had calculated the best-case as roughly 97 tiles per hour, it is actually around 160.

From there, it was a sequence of measured steps on the 6980P node:

step	tiles/h
eager PyTorch autocast	56.2
native bf16	60.3
oneDNN-prepacked weights, fused ReLU and add	81.9
heap-only malloc	96.6
C++ thread-per-block library	122.3
per-chip S2 block size	128.3
standalone `tessera_encode`, no Python	136.5

I had originally ruled out the C++ rewrite, but I wanted the absolute control that this would give. I wanted to control the block size to ensure that, at each stage, the full width of the AMX unit was used, while also keeping the working set small enough to fit in one core’s cache rather than running each layer across all pixels. This yielded a 41% improvement.

The standalone encoder is a 190 KB binary plus oneDNN library and the weights. It produces byte-identical output to the Python path, and 93.3% of its int8 values exactly match the published beta1 tiles with a cosine of 0.99997.

Then we need to race the CPUs at Endeavour. One process on every physical core, for the same Manchester tile:

machine	cores	matrix unit	tile
Xeon 6980P	256	AMX	32.2 s
Xeon 6972P DDR	192	AMX	34.0 s
Xeon 6966P-C	192	AMX	35.0 s
Xeon 6787P	172	AMX	46.6 s
Xeon 8592+	128	AMX	50.2 s
NVIDIA A30			60.1 s
Xeon 6745P	64	AMX	60.9 s
2 × AMD EPYC 9965	384	AVX-512 bf16	66.9 s
Xeon 8480+	112	AMX	81.0 s
NVIDIA L4			118.5 s

The EPYC line is a useful control. It has 1.5 times as many cores as the 6980P and takes twice as long, which puts AMX at about 3 times AVX-512 bf16 per core on this model. One AVX-512 bf16 instruction does 32 multiply-adds; one AMX instruction does 8,192.

It was interesting to see the higher-density chips throttle once every AMX unit is busy. On the 6980P a single 16-thread process runs at 3.39 GHz and the full node at 2.47 GHz, a 27% drop, which on its own accounts for most of the per-process slowdown. It is a per-socket effect: the floor is reached as soon as one socket is full and does not drop further when the second fills. It also tracks density. The 2 x 32-core 6745P loses 6%, the 2 x 96-core parts about 20%, and the 2 x 64-core 6767P does not throttle at all, sitting at a fixed 2.91 GHz whether one process is running or sixteen. The older Emerald Rapids and Sapphire Rapids parts are the worst at 33% and 30%, and they are also the two whose power counters are clean enough to show why: both are at the package power limit under load.

The upshot is that the 6980P’s lead is much smaller than its core count suggests. The 2 x 96-core 6972P comes within 6% of it per node, and per core it is ahead.

On the GPU side, I wrote a CUDA forward pass with the same C API, so the same encoder binary links against either. That was not strictly necessary, but spending a day optimising the CPU and then comparing it against an unoptimised GPU would not have been a fair test. On the L4: eager PyTorch 227 s, torch.compile 170 s, TensorRT 166 s, hand-written CUDA 118.5 s, against a cuBLASLt best case estimate of 83 s. So TensorRT buys almost nothing over torch.compile, and costs accuracy, and the hand-written version roughly doubles the best PyTorch route.

A surprising result was that the chunk size needs to be completely different per device. The A30 is best at 1024 pixels and the L4 at 64, a 16-fold difference, and the A30’s sweep spans 1.76 difference depending upon the chunk size. The A30 has 24 MB of L2 but about 933 GB/s of HBM2, so DRAM round trips are cheap and bigger GEMMs win; the L4 has 50 MB of L2 and 300 GB/s of GDDR6, so cache-sized chunks win. The same effect shows up in the bare cuBLASLt benchmark, where L2-sized chunks run about 40% faster than full batches.

Tessera one pixel fixed is merged

Kristian Bodolai reported nodata seams in the Cambridge Zarr variants over Ecuador a couple of weeks ago. It is the one pixel offset from week 38, wearing a different hat: _tile_pixel_offset rounds where the producer floors, so a tile whose fractional offset from the zone origin reaches one half lands a pixel out. The fix is round to math.floor on two lines.

What is new is the second symptom. The displacement is invisible when you look at a single tile, because everything in it moves together. A seam appears only where a shifted tile abuts an unshifted one, which is why he saw it on some boundaries and not others. His example is two adjacent tiles with fractional offsets of 0.354 and 0.508, and the second is barely over the line:

grid_-79.35_-0.55  frac 0.354  round=50838  floor=50838  width 1113
grid_-79.25_-0.55  frac 0.508  round=51952  floor=51951  width 1114

The west tile starts at column 50838 and is 1113 wide, so it ends at 51951. Under floor, the east tile starts at 51951, and they abut exactly; under round, it starts at 51952, and there is a one-pixel gap.

I reproduced it from the tile geometry rather than from the data, added a regression test and checked that it fails on unmodified code before fixing anything, then ran a real conversion on Fargate against a scratch store and re-ran his own script:

SEAMCHECK utm17 lon -79.2999134 crs EPSG:32617 year 2024 shape (30, 30)
nodata_posinf 0
VERDICT: PASS_NO_SEAM

The old code gave 30: a full +inf column.

The seams are the small part. This affects the published v1.0, v1.1 and v2 Zarr stores where the .npy tiles themselves are correct. v1.1-dclimate is clean, because it came through the icechunk transcoder, which never calls this function at all, and where that path does use round it checks first and raises if the value is not already an integer.

ucam-eo/geotessera#433 is merged.

The Zarr stores will now be regenerated.

Tessera inference of Henan, Anhui and Jiangsu

Cissy asked for three provinces in China: Henan, Anhui and Jiangsu, which is 4,204 tiles. Coverage for 2019 showed only 2,901 missing, so I thought that this was sufficiently small that I could run it on a single GPU with the new oneDNN binary. See later for how this was orchestrated. With this ROI processed, I am now working on Pedro’s weather stations ROI, which has around 10 times as many tiles.

day10 and reverse dependencies

After last week’s flurry of issues to achieve parity with obuilder, this week was quiet, with only two actual issues being found. This gave some breathing room to extend mtelvers/day10 to the reverse dependency section of ocurrent/opam-repo-ci. The reverse dependencies are calculated via an ocurrent/obuilder job which runs three opam list commands and scrapes the output back out between markers.

I considered the same approach in day10, I could submit a day10 exec job to run any opam command, but this means that I would need to materialise the entirety of opam-repository to disk to pass to the container. day10 solves against the git database rather than materialising the files and only writes the files to disk for the exact solution it is going to build.

However, opam has a dedicated solver for reverse dependency lookup, whereas day10 was only using 0install-solver. Before writing anything, I measured the current performance:

query	time
baseline repository load, no query	0.89 s
`--depends-on` only	0.96 s
`--coinstallable-with` only	264 s
both together, as actually used	8.0 s

--coinstallable-with on its own keeps 16,955 of 19,143 packages, which is to say almost everything, and takes four minutes to say it. All of its value is in composition with the other flag.

Next, I needed a test harness to compare a yet-to-be-written day10 version with the results produced by opam. It runs the three opam list commands as the reference and day10 revdeps as the candidate, on two compilers, and diffs them. My test packages were coq and dune as the deliberate worst case, with 14,049 dependents, 73% of the available packages.

The day10 version runs in two phases, and only the second one solves.

The first phase is a single pass over the repository, building the candidate set from metadata alone. For each package it resolves depends and depopts against the variant, and a package is a direct dependent if either formula is satisfied by the target. At the same time it records, for every name a package depends on, the formula it depends on it by, which gives a reverse-dependency table. A closure over that table starting from the target then picks up everything that reaches it indirectly, which is the recursive query. Crucially the closure follows hard dependency edges only: running --recursive together with --depopts and --with-test, as a single combined query would, walks the closure through optional and test edges as well and answers a wider question, which is what turns coq’s 11 dependents into 20.

Matching is by version throughout, not by name. A package wanting dune < 3.0 is not a reverse dependency of dune.3.20.2, and on a target that most of the repository mentions, those are most of what a name-only match sweeps in: ignoring the constraints added 1,306 packages that opam excludes.

So no solving at all so far, just a fold and a graph walk. The second phase is the brute force: for each candidate, solve for the target and that candidate together at their exact versions, and keep the candidate if the solve succeeds. That is what --coinstallable-with asks, and asking it any cheaper way does not work. Solving for the candidate on its own does not ask it, because a package that depends on the target only through depopts solves perfectly well without the target present, and so passes for the wrong reason. zenon.0.8.5 is the example: it has coq in depopts and coq-core in conflicts, so a solution that omits coq never meets the conflict. Restricting the repository to a single version of the target has the same hole, since nothing then compels the target to be installed either.

That is one solve per candidate, so thousands of them on a target like dune, which is why the verb takes --fork.

Final agreement:

target	opam	day10	diff
coq.9.2.0 / 5.5	2	2	-0 / +0
coq.9.2.0 / 4.14	10	11	-0 / +1
dune.3.20.2 / 4.14	13,302	13,378	-11 / +87
dune.3.20.2 / 5.5	9,664	9,696	-6 / +38

Wall clock, against the three opam list commands it replaces:

target	opam	day10
coq.9.2.0	17.7 s	4.0 s
dune.3.20.2 / 4.14	112.0 s	265.1 s
dune.3.20.2 / 5.5	92.0 s	294.6 s

So day10 is about four times faster on the ordinary case and two to three times slower on the worst one. That is the shape you would expect from the two designs: opam pays a fixed cost to load and index the repository and then answers cheaply, while day10 pays per candidate. For coq, with ten dependents, the fixed cost dominates and day10 wins. For dune, with 14,049, it does not. Nine minutes to decide what to build is noise against a work list of 13,000 builds, so this is not worth optimising yet.

That is 99.4% and 99.7% on dune across 16,664 candidates, and every remaining difference is understood. The eleven that day10 omits are all cases where it is right: opam includes the target in its own reverse dependencies, and day10 removes it; seven packages are macOS-only; and three are capped to an opam version the harness happens to be running. The 87 extras are depexts. opam consults the scheduler’s own apt and discovers that conf-gtksourceview needs a system package that is not available; day10 solves the opam universe and leaves depexts to the container.

The two other day10 fixes came out of real job logs. A layer was being renamed into place before its layer.json was written, and the “is this layer finished” check tested for the directory outside the lock, so a job arriving in that window found a directory with no manifest and crashed on utimes. Writing the JSON to the temp directory before the rename was the easy fix. And OBuilder sets CI=true where day10 did not, which matters for test frameworks that behave differently under CI. It went into day10 rather than opam-repo-ci because it is constant for every day10 container, and the layer hash deliberately does not include the ambient environment, so anything that varies between invocations sharing a cache would produce colliding hashes.

I’ll complete the opam-repo-ci side this week.

ocaml.org outage

ocaml.org runs two instances as Docker containers behind a Varnish cache. The container logs were showing:

03.10.26 00:14:32.046  dream.logger ERROR Async exception: Unix.Unix_error(Unix.EMFILE, "accept", "")

Each server process held 65,490 open socket file descriptors against a limit of 65,536, with about 131,000 TCP allocations in a closed state against 113 live ones. Leaked, not busy.

The first explanation was DNS, and it was well evidenced. Docker’s embedded resolver was timing out against Scaleway’s upstream servers, about 38,100 errors in the 23:00 hour alone, and every documentation page fetch resolves through a blocking getaddrinfo in Lwt’s thread pool, so the 1,000-thread pool saturated as requests ran for hundreds of seconds.

I remembered seeing a similar issue on ocurrent/ocaml-ci when it was being scraped by bots. At the time, the upstream logs fetch was slow due to a SQL key issue resulting in abandoned connections. The solution at the time was two commits: the first caught the write loop never resolving its exit promise on an exception, and then production strace showed the read path was the dominant leak, so the second made Io.close shut down best-effort and always close. That pin has been running ocaml-ci, which is why the shape of this one was familiar.

The defect is in anmonteiro/gluten, and it is the same two bugs: Io.close calls shutdown and then close inside a single exception handler; on a socket the peer has already reset, shutdown raises ENOTCONN, the handler swallows it, and close is never reached. And the IO loop only closes its socket once both the read and write loops have resolved their exit promises, which only happens on the clean path, so a loop that terminates by raising leaves the join waiting forever. strace shows it exactly:

read(6, "", 4096)       = 0
shutdown(6, SHUT_RDWR)  = -1 ENOTCONN (Transport endpoint is not connected)

and no close(6).

Every request in which the client leaves before the response is fully written leaks one descriptor. Normally, completed requests do not, which is why the site runs for days until something slows responses enough that clients give up on it. A harness of twenty clients that disconnect after 50 ms:

	baseline	20 FIN	20 RST	20 normal
as shipped	6	26	46	46
with the fix	6	6	6	6

and in Docker, 100 reset connections against a documentation page take the old image from 7 descriptors to 94 and the new one from 7 to 6.

Currently, the site is being built from a patched branch on mtelvers/gluten, pinned into ocaml/ocaml.org through pin-depends so that CI and the devcontainer get it too. I have also sent the fix upstream as anmonteiro/gluten#90, where it joins #88 fixing the same two-part bug in the Async backend.

A Windows worker that can use its whole disk

ltsc2025-3 is the last Windows worker without a separate Docker volume, which makes it the last one that can still hit the shared-volume prune livelock. The others were straightforward to split, because they have sub-2TB SSDs where MBR reaches the whole disk. This one has an 8TB spinning disk, and MBR can’t reach beyond 2 TiB, so the remaining 5.3 TB is not space I could claim: it is unaddressable. Splitting it means UEFI and GPT, and I had been putting that off, as QEMU with UEFI and GPT typically takes a good number of iterations before it’s working as I want. In practice, my reservation was misplaced: ovmf was already installed, the change is a q35 machine type, a pflash drive and a per-VM copy of the variables file, and an answer file with EFI and MSR partitions and InstallTo pointing at partition 3 instead of 2. The result is C: at 5,120 GiB for obuilder/hcs and D: at 2,332 GiB for Docker, using the whole 7.3 TB. It is built alongside the old machine rather than over it, and now that it is taking jobs, ltsc2025-3 can be retired.

macOS workers

Three arm64 machines were in a launchd crash loop because the OpenZFS kext was not loaded, which was in turn because ZFS spacemap corruption was panicking the kext on auto-import at boot, and macOS had then evicted ZFS from the auxiliary kernel collection. The error message is actively misleading: it says the extension is not approved even though it is. What was missing was the kext in the booted collection, and a reboot did not restore it. I’ve seen this before, and the fix is to wipe the ZFS partition with zeros so ZFS doesn’t try to read the corrupt data. Once done, the kext can be loaded again, and the Ansible playbooks can be used to rebuild the machines. I also took the opportunity to update to the latest openZFS driver.

/maps moves to CephFS

/maps is the shared research filesystem consisting of about 120 TB on an NFS server congo with all users squashed to uid:gid 400:400. It had reached 100% with 17 GB free. Users deleting files freed nothing because the ZFS snapshots hold it.

I had taken a CephFS mirror using rsync back in April. I ran incremental updates but since writes were blocked only the first one actually did anything. The cutover to CephFS took seconds across all three client machines, using a lazy unmount. This allows running jobs to keep working against the detached mount for days afterwards. 120T at 100% became 402T at 82%.

Every file on /maps is owned by uid 400, because congo’s NFS export all-squashes, and kernel CephFS does not remap. A straight switchover would have locked everyone out of their own data. That was handled with a maps group at gid 400 on all three machines, which in turn required an audit that found 63 colliding uids, which in turn became a renumbering of 74 accounts into a 30000 block across three machines. Everyone rather than just the clashes, because then nobody feels singled out.

The only real user-visible breakage was permissions rather than downtime, and only for people whose login predated the group change, since process group lists are fixed at login.

occ reaches parity on RISC-V

Last week, I left off with mtelvers/occ having just lazy3.ml failing, as it was too slow on the bytecode interpreter. The diagnosis was that caml_bytecode_interpreter is one enormous function whose dominant loop spans the whole body, so every variable looks live everywhere, and with far more of them than there are registers, so key values get reloaded from the frame on every opcode.

The register allocator was approximating live ranges by extending them over whole loops. That rule is ruinous for a function which is one loop with a hundred arms. Taking live intervals off the flow graph from proper live-variable analysis instead:

	original	threaded only	with live intervals	gcc -O2
wall clock	59.4 s	63.4 s	15.0 s	9.5 s
instructions		10,490	6,950	3,118
frame accesses		5,139	681	335

Four times faster, and from 6.2 times slower than gcc to 1.57. Both changes were needed; neither does much alone. lazy3.ml, which had been timing out at about 630 seconds against a 600 second limit, now passes.

That leaves RISC-V at 1,618 passed, 60 skipped, 1 failed, against a gcc-built tree on the same board on the same day at 1,621, 57 and 1. Both fail the same test, and for an upstream reason: OCaml ships a reference file under one name, and ocamltest looks for another. The three tests have a gap: two require a C++ compiler, which is out of scope by design, and one requires -ffunction-sections, which occ does not emit.

ferry

My ad hoc Tessera pipeline was four terminal windows: xargs running GPU inference, another downloading tiles, another uploading to S3, another pulling them down on a second machine, with manual restarts between batches and dispersed over several machines. As an ocurrent/ocurrent maintainer, it is clearly the best pipeline tool available, but it’s not the thing I reach for when the work is ad hoc.

So mtelvers/ferry: stages that each run a shell command on a named host over ssh with their own concurrency limit, where an item moves on as soon as its job exits. It has a notty TUI, runs jobs in their own process group with remote work under ssh -tt, so killing ferry kills the remote side too, and journals finished stages so that a restart resumes correctly.

Here’s what a pipeline might look like. Say you have a pile of scanned PDFs on the NAS and want them searchable. Splitting a PDF into page images is quick; running OCR over those pages is not, and it wants the fast machine.

open Ferry

let nas  = host "nas.local"     (* where the scans live *)
let fast = host "fast.local"    (* has the fast CPU *)

let () =
  items nas "ls ~/scans | sed 's/[.]pdf$//'"
  |> stage "split" nas ~jobs:4 ~out:"~/pages/{id}/"
       "pdftoppm -r 300 -png ~/scans/{id}.pdf {out}/page"
  |> limit 8
       (move "fetch" (at fast "/scratch/{id}/")
        >>> stage "ocr" fast ~jobs:2 ~consume:true ~out:"~/text/{id}/"
              "tesseract /scratch/{id}/page-1.png {out}/text"
        >>> move "file" (at nas "~/searchable/{id}/"))
  |> run

Four jobs concurrently run the splitting, with two doing OCR, and limit 8 keeps at most eight documents on fast.local ready for processing, so the splitter cannot run a thousand PDFs ahead of the OCR and fill the scratch disk. That gate is the thing that the four terminal windows could not give me. My download ran unchecked and much faster than my GPU could run the inference.

Everything else follows from ~out. Because each stage says what it produces, move needs only a destination and finds its source wherever the previous stage left it; a restart lists the outputs once and starts each document after the furthest stage that already has one, so work done before a crash is never repeated; a command that mentions {out} writes into a temporary directory and is renamed into place only on success, so a half-made result cannot look finished; and ~consume:true deletes the staged pages once the OCR has succeeded. None of that is written in the script.

The script is OCaml, and it is not compiled by anything you have to install. ferry links compiler-libs.toplevel and is built (modes byte_complete) with -linkall, so the bytecode runtime is inside the executable. The only other thing a toplevel needs to typecheck a script is the compiled interfaces to typecheck it against, so a dune rule globs the 69 .cmi files of the stdlib together with Ferry’s own and runs them through a twelve-line generator that emits

let files = [
  ("camlinternalFormat.cmi", "\132\149\166\190...");
  ...
]

This is then just another module in the link. The result is a single self-contained 6.6 MB binary with the OCaml compiler inside it. There is nothing to install on the machine you run it from, and nothing to build before a pipeline runs, and a type error arrives before any host is contacted.

The interface has changed with each new commit: it started as a line-based spec file, and became OCaml scripts within the hour. Then commands became functions of the item, and host settings became typed records rather than string variables. Declared outputs came last, and removed the skip checks, the done-lists, the environment exports and the line format in one go.

The real thing: 11,024 tiles, built on pima and encoded on two GPUs. The either gate sends each tile to whichever branch has room, so daintree and monteverde pull 28 and 32 tiles an hour without being told how to share the work.

ocurrent 0.8.0

The Prometheus 2 work from the last two weeks is finished and released: ocurrent/ocurrent#479 for the synchronous collector and #477 for the git argument handling. 0.8.0 rather than 0.7.6 because requiring Prometheus 2.0 raises the minimum for everyone downstream.

The order then matters, because nothing downstream can pin a repository sha containing 0.8.0 until it is actually there. So: the release into opam-repository as ocaml/opam-repository#30827, then the Dockerfile sha in ocurrent/solver-service#84, then ocurrent/ocaml-ci#1075 and ocurrent/opam-repo-ci#488 picking up the new bound. All merged.

32-bit OCaml

The 32-bit compiler fork needed a real fix rather than just a rebase. Upstream now converts between floats and boxed integers without calling into C, but the direct path uses native-word integers, so on a 32-bit target Int64 boxing leaves the upper half undefined and Int64.of_float 0. returns 8796093022208. Int32 and Nativeint are word-sized there and keep the fast path. That fix then broke three nightly builds in a row, because I had declared the C calls alloc=false when both of them do allocate, which skips the wrapper that makes the OCaml stack walkable.

FreeBSD worker

rosemary has now been healthy for ten days. On Friday, the no-solution job time roughly doubled, and resident vnodes increased from 140k to 825k. However, this didn’t appear significant as vmstat -z shows the vnode zone has never once failed to allocate in 2.5 billion requests. The machine had 162 open files and no jails, so the average time increase is probably just down to a busy machine for those few samples.

Scaling and benchmarking a critical message bus using a new indexing strategy

Jane Street Tech Blog — 2026-10-05T00:00:00-00:00

The following is part of a series of posts about 2026 summer intern projects—for more, see “What the interns have wrought, special jumbo 2026 edition”

.plan-26-40: A whole new world (of wall-to-wall Tessera embeddings)

Anil Madhavapeddy's Blog — 2026-10-04T00:00:00-00:00

I'm almost set for the start of teaching term now and meeting our newest intake and catching up with everyone after a year away! My 1A Foundations of CS starts next week, and I've decided to lock in the use of Forester for this. For the first time, we won't be giving out the traditional printed notes. I'm a little worried about not giving away since I've seen students scribbling on them well into their second years in the past! I'm hoping that the cross-referenced Forester will be more friendly to the tablets and notebooks that students are all using these days.

GeoTessera 0.11.0 now has wall-to-wall embeddings!

As its start-of-term here, David Coomes Sadiq Jaffer and I will give an intro to Tessera talk at the CRI next week, so pop along if you're in town!

I also shipped geotessera v0.11.0, which now provides wall-to-wall embeddings for anywhere in the world from 2017-2025, a major first for the project! Yay! See the release notes above for details.

There are now two separate full copies of the v1.1 embeddings available, in either Icechunk or Zarr format, with the choice depending on your particular usecase. This has been a (validly) confusing point for early users, so here's an explanation of the innards.

Icechunk for bulk streaming

First, the dClimate team built a custom inference stack that uses Icechunk, and publishes this in the s3://tessera-embeddings bucket on AWS. This has been optimised for 'streaming access' with relatively large shard and chunk sizes to minimise round-trip traffic.

However, it can only be accessed using Icechunk-compatible bindings since although Icechunk has a Zarr-compatible store interface, its on-disk (via HTTP) format completely differs from standard Zarr. Therefore, a client in something like OCaml that speaks the Zarr standard will not be able to interoperate with the Icechunk store.

Zarr v3 for smaller mobile-friendly chunks

This motivated us to transcode the Icechunk store to a separate pure Zarr v3 format. While we lost some cool management features available with Icechunk (like versioning and snapshots), we gained compatibility with clients that implement pure Zarr (like my own OCaml code for example). While we were here, I also changed the size of the chunking to be lower to make it more friendly for mobile streaming. This is what allows mobile JavaScript frontends like TZE to work.

Therefore as a user, with GeoTessera 0.11 you now have a choice between the two:

The Zarr store on Source Cooperative is the default for streaming reads.
Pass an .icechunk URL to GeoTesseraZarr() instead if you want the Icechunk repository directly. The production store is at s3://tessera-embeddings/v1.1/dclimate.icechunk. I'll expose this as a builtin option in a future geotessera.

Bypassing the Source Coop proxy for direct S3

One reason that many users saw bad performance initially is that the Source Coop has an additional caching layer over Amazon via a Rust proxy (data.source.coop), and this appears to be throttling rapid connections in a way that directly going to S3 isn't. GeoTessera 0.11 now reads straight from the S3 bucket by default to avoid putting any undue pressure on the poor sole proxy server that Source Coop is kindly running for us, with the gateway still available as an opt-in in the geotessera CLI via --via-gateway if you want its Cloudflare edge caching.

Phew! There's a lot going on here with where the embeddings are stored, but I hope we're hitting the right notes as to how to make them as maximally accessible as possible. My next step here is to publish the HTTP perma proxy I wrote a few months ago for caching Zarr tiles, in order to provide a gateway proxy in Cambridge that stashes commonly used years locally to avoid lots of network traffic.

Going to the Other Place to talk at the Intelligent Earth CDT

I was invited by the enthusiastic crew to give a talk on on "Technology for Living Evidence" (my overall name for the work we're doing on Tessera, habitat and species mapping and evidence ) to the new student cohort at the Intelligent Earth CDT in Oxford.

The venue was the incredibly impressive new Centre for Life and Mind, which opened about a year ago and still had that 'new building' smell. I was told it cost a cool ~£200m and is Oxford's largest-ever building project, except for certain other hilariously bad bridges. I'm told it supports 1,400 researchers, >1,000 undergraduates a year, and has shielded EEG rooms, eye-tracking labs, and even a sleep laboratory tucked away somewhere!

The Intelligent Earth CDT is a UKRI-funded initiative that'll train 100 PhD students years across climate, biodiversity, natural hazards, and environmental solutions. This is temporally a follow-on to our own AI4ER program here that's drawing to a close now, so it was also excellent to have Yihang She along from our group to act as a bridge between generations of students!

The director, Philip Stier, was a brilliant host and also tutored me on his expert topic of the physics of clouds. He recently built a flow-matching model that downscales high-resolution (~1km) cloud structure from the coarse (~25km) fields that climate models calculate.

The Oxford-Cambridge rail certainly isnt moving very fast

Getting to Oxford was a nightmare as usual, as the only useful options were trains via London (~3 hours but the line was down), a bus via Bedford (~2.5h), or motorbiking down the single-carriageway A421 (~2h "on a good day"). As others have noted the East-West railway can't come soon enough:

Today, travelling between Oxford and Cambridge by public transport is a joke. [...] With East West Rail, that journey time drops to ninety minutes. [...] This means a researcher in Cambridge can collaborate with a lab in Oxford without losing a day to the journey

Despite the Varsity line dating back well into the 19th century, the campaign to reinstate has been going on for 30 years (!), and under East West Railway Company's revised April 2026 proposals the full end-to-end Oxford–Cambridge service isn't scheduled until the mid-to-late 2030s. Absolute snail's pace progress here...

Scrutineer fixes keep rolling

InfoQ picked up my 'rumour is the exploit' post, on how AI agents can basically find working exploit faster than traditional embargoed disclosure can keep up. In the meanwhile, Scrutineer continues to be extremely successful at finding issues, and Hannes Mehnert and Edwin Torok continue to fix many of them (thank you!!). I'm going to catch up on my Patch the Planet planning backlog later as I've been extremely busy preparing for the start of term.

In terms of where activity has been focussed, mirage/awa-ssh joins the list of repos being fixed, with a nonce fix, a CPU/memory fix and a required-authenticator fix. Elsewhere I spotted ocaml-tls's signature-algorithm fix, happy-eyeballs's empty-address fix, more Solo5 bounds checks (#682/#684 closed, #683 carried over, plus new dev-type and note-size checks).

The LIFE tour, and offsetting ourselves

I built an interactive LIFE tour using an ensemble of agents narrating and animating the LIFE metric over the live Zarr store. This was worryingly quick and good, and with several reactions on the socials like LinkedIn.

This ranged from Alison Eyres being very impressed, to Chris Sandbrook asked how the process actually worked and worrying about the implications. It's fair to say that I am worried as well; I'm feeling awe at the one-shot quality, intrigue at the educational potential for personalisation, and a little shock at the implications for jobs down the line. Will we end up needing to 'offset' our use of AI to support human endeavour, the way we do for carbon credits?

Released a new OCaml mdx to unblock Eio dev

I released mdx 2.7.0 to add support by Thomas Leonard for a new mdx_skip directive and shonfeder's dune 3.24 support fix. Thanks to a Windows VM that Mark Elvers setup for me I no longer need a physical Windows desktop, which has made me much less grumpy. I've resumed working on Eio's Windows process spawning and nt_path normalisation, as well as dusting off my changes to the IOCP bindings. This is the 'relaxing' hacking of the week, but may need to be put on ice as term starts!

References

Madhavapeddy (2026). .plan-26-31: Sorting out Tessera and Evidence TAP infrastructure. 10.59350/30e5y-n8p97
Madhavapeddy (2025). Foundations of Computer Science. 10.59350/qms3q-ymn65
Madhavapeddy (2026). Just a rumour of a bug is enough to find a security exploit these days. 10.59350/tngsm-6rx23
Madhavapeddy (2026). Using Forester to turn Foundations of CS into interactive evergreen lectures. 10.59350/sjcvd-hb857
Madhavapeddy (2025). Disentangling carbon credits and offsets with contributions. 10.59350/g4ch1-64343

How recurring network maintenance exposed 6 bugs

Jane Street Tech Blog — 2026-10-02T00:00:00-00:00

Jane Street does a lot of network maintenance over the weekends. In each maintenance, a cabinet may lose connectivity for 15 to 25 minutes. Our internal Kafka infrastructure is supposed to be resilient to these partitions, and reconnect when connectivity resumes. For several months this year, it didn’t: processes segfaulted, leaked hundreds of thousands of sockets, or crashed in a surge of half a million TCP connections.

Results from the ASIC puzzle

Jane Street Tech Blog — 2026-10-02T00:00:00-00:00

The Era of Programming Languages Exploration is upon Us.

Kiran Gopinathan — 2026-10-02T00:00:00-00:00

There was a recent thread on the TYPES mailing list with researchers speculating about the future of our field given the disruptive effects of AI tools on programming in general.

Combobulate Now Supports OCaml & OxCaml

Tarides — 2026-10-01T00:00:00-00:00

Code navigation is one of those aspects of programming that can either make your experience significantly better, or be such a pain. Most of the time we navigate code as text, i.e, searching with regexp, jumping by lines or moving word by word. But code isn't text. It has syntactic structure, and being aware of that structure when moving and editing opens up a different way of working. This is what structural navigation and editing means: operating on the actual constructs of a program; expressions, bindings, match arms, module definitions, etc, instead of characters and lines.

We have recently improved navigation in OCaml/Oxcaml with Combobulate support, and this post will get you up-to-speed on what’s new, how it works, and where to try it out!

What Has Structural Navigation in OCaml Looked Like Until Now?

OCaml already has a substrate of structural navigation through Merlin (and by extension OCaml-LSP). The jump command gives you a limited form of structural movement such as jumping to the next let, match, module, and a few other constructs. It is useful, but it's a small subset of what structural navigation could be. Previously, this limitation motivated the GopCaml project, which took a more ambitious approach to structural editing for OCaml by working directly with the compiler's AST.

More recently, tree-sitter has introduced a generic abstraction over syntax. Given a tree-sitter grammar for a language, you get an incremental parser that produces a concrete syntax tree you can query and traverse. OCaml has a tree-sitter grammar which is already used in, for example, neocaml-mode where it provides syntax highlighting.

Tree-sitter can be seen as the syntactic counterpart to LSP: where LSP standardizes semantic features, Tree-sitter provides a common protocol for syntax. Much like TextMate grammars provided a generic way to handle syntax highlighting across editors, Tree-sitter gives editors syntactic tools such as highlighting and navigation.

Combobulate by Mickey Petersen takes tree-sitter in a different direction: it uses the syntax tree for structural navigation and editing. It's a minor mode for Emacs that supports many languages, and it now supports OCaml.

Why Does Combobulate Matter for OCaml?

OCaml code nests very deeply. Modules contain structures, structures contain let bindings, let bindings contain match expressions, and match cases can contain further match expressions. Type declarations can define records, variants, and GADTs in a single type ... and ... block. Many of these constructs can recurse into each other with no fixed limit; this is part of what makes OCaml expressive, but it also means that even a small OCaml file produces a deep and wide tree-sitter parse tree.

Implementing structural navigation for OCaml is harder than for most languages precisely because of this: the procedures that tell Combobulate how to pick the right node at any point have to account for potentially infinite nesting at every level. This is also why line-based movement becomes incredibly slow and unreliable. Jumping to the next let with an incremental search won’t help when there are six of them nested within each other. This is why structural navigation, which helps us move by the structure of the code, and the relationships between different nodes in the tree, feels natural and makes a real difference.

Combobulate is an important addition to the OCaml ecosystem because it perfectly complements tools like Merlin and OCaml-LSP. While Merlin is great for semantic intelligence, type checking, autocomplete, and jumping to definitions, its structural navigation features (like the jump command) are limited. By letting Combobulate handle the purely syntactic, structural movement and editing, the two tools work together to provide a comprehensive editing experience: Merlin understands what your code means, while Combobulate understands its shape.

Navigating OCaml with Combobulate

Once Combobulate is active in your OCaml buffer, you should see a © in the mode line. There is a Magit-style transient UI bound to C-c o o that lists every binding, which is handy while you're learning. To inspect the full keymap directly, run M-x describe-keymap RET combobulate-key-map.

With Combobulate, you have different commands to navigate your code in a variety of ways: jumping between siblings, jumping between occurrences of words, traversing the node tree sequentially, and more.

Navigation Commands

Binding	Summary	What it does
`C-M-u` / `C-M-d`	Up/Down into list	Move in/out to the parent/child node.
`C-M-n` / `C-M-p`	Forward/Backward sibling	Move to the next/previous sibling at the current level.
`M-e` / `M-a`	Logical next/previous	Jump to the next/previous logical node, regardless of nesting.
`M-n` / `M-p`	Sequence navigation	Move between paired sequence points (e.g., jumping from the word `let` to the next occurrence of `let`).
`C-M-a` / `C-M-e`	Move to the start/end of defun	Move to the beginning/end of defun. This is based on best-effort. In nested let bindings, it doesn't work very well.

Navigation Examples

Combobulate primarily handles code navigation in terms of two axes:

Vertical/Hierarchical (Parents and Children): Moving "up" (C-M-u) leaves the current node for its enclosing parent, while moving "down" (C-M-d) descends into the child node at the cursor.
Horizontal (Siblings): Moving forward (C-M-n) or backward (C-M-p) hops between sibling nodes at the same syntactic level, such as adjacent match cases, list elements, or record fields.

When hierarchical or sibling navigation isn't enough, Combobulate also offers logical navigation (M-e / M-a). Rather than being constrained to direct parent-child or sibling relationships, logical navigation moves sequentially across nodes in their logical reading order—allowing you to cross operator boundaries or escape deeply nested subtrees.

Simple Examples

Navigating down into a body (C-M-d)

"Down" means entering whatever node the cursor is sitting on. The clearest case is descending from a module declaration into its contents:
```
module Counter = struct
  let value = 0
  let bump x = x + 1
end
```
Place the cursor on module. Press C-M-d thrice and the cursor moves to let value = 0. Press C-M-d again and you descend further, into the binding itself.
Navigating up to the parent (C-M-u)

"Up" is the inverse: leave the current node and land on its enclosing parent. Suppose the cursor is on the number 100 inside a record:
```
let player = { name = "Ada"; score = 100 }
```
C-M-u jumps to the whole field score = 100. Press it again to land on the record { ... }. To move from 100 directly to the let keyword, use C-M-a.
Navigating siblings (C-M-n / C-M-p)

Siblings are nodes at the same level, like match cases, tuple components, record fields, and array elements. Take a match expression:
```
match shape with
| Circle r    -> pi *. r *. r
| Square s    -> s *. s
| Triangle (b, h) -> 0.5 *. b *. h
```
Place the cursor on the first match arm (Circle r -> ...). C-M-n moves to Square s -> .... Again to Triangle .... C-M-p walks back.

Complex Examples

Using only parent-child or sibling navigation is not always sufficient to navigate OCaml code efficiently. Because OCaml's deep nesting can lead to highly nested concrete syntax trees, you need a few more tools in your belt to avoid getting stuck.

Example 1: Using next-sequent (M-n) and prev-sequent (M-p)

In subsequent let...in bindings, parent-child/sibling navigation is insufficient and unreliable due to how let...in is represented as deeply nested subtrees in the tree-sitter grammar. Each successive binding is actually a child of the one before it, meaning C-M-p won't walk backwards up the chain. Instead, use sequence navigation to hop directly from one let to the next and back.
```
let emit_string_table_section fmt section_name
  (table : Dwarf_write.string_table) =
  let buf = Buffer.create 64 in
  let contents = Buffer.contents buf in
  let i = ref 0 in
  let len = String.length contents in
  while !i < len do
    let start = !i in
    while !i < len && contents.[!i] <> '\x00' do
      incr i
    done;
    let s = String.sub contents start (!i - start) in
    emit_asciz fmt s;
    if !i < len then incr i
  done
```
If we want to move from the let-binding on line 3 to the let-binding on line 6, sequence commands M-n and M-p let you jump forward and backward easily.
Example 2: Using logical-next (M-e) and logical-prev (M-a)
```
if (x = 1) then true else false
```
When the cursor is on if, you can do C-M-d to go to the parenthesis (, then C-M-d again to enter x, or C-M-n to go to then and else.

However, if we have the same code without the parenthesis:
```
if x = 1 then true else false
```
There is no direct sibling relationship to go from x to then using C-M-d or C-M-n. In this case, we use logical-next (M-e) to cross the operator boundary and jump directly to the then branch.

Logical next/prev allows you to move to the next node in the tree irrespective of their parent/sibling relationships. It is also incredibly helpful for passing over ->, =, and other operators.
Example 3: Escaping Deep Subtrees

If you are at the end of a long top-level item and want to navigate to the beginning of the next top-level item, use logical-next (M-e). If you try to use forward sibling navigation (C-M-n) from the end of the item, the cursor won't move at all since you are deep inside a nested subtree with no siblings to your right. Using M-e lets you jump out of the subtree instantly to the next top-level construct.

Editing Commands

Because Combobulate's editing commands are built on top of its navigation primitives, particularly sibling navigation, they all work in OCaml without any extra configuration. If you can navigate between two nodes, you can edit them.

Binding	Summary	What it does
`C-c o e`	Envelope prefix	Apply a code template (envelope) at the cursor. Press `C-h` after to see what's available in this context.
`M-h`	Expand region	Mark the current node. Repeat to expand the region to the parent iteratively.
`C-M-h`	Mark defun	Mark the current enclosing defun. Repeat to expand to the next enclosing defun iteratively.
`M-N` or `M-S-n`	Drag forward	Swap the current node with its next sibling, preserving formatting.
`M-P` or `M-S-p`	Drag backward	Swap the current node with its previous sibling.
`C-c o c`	Clone node dwim	Duplicate the node at cursor. If ambiguous, you cycle through candidates with a live preview (the carousel).
`C-c o t`	Place cursors	Place multiple cursors (or field-editor fields) at every related sibling; e.g. each element of an array, each field in a record.

Editing Examples

Expanding the region (M-h)

Each press grows the selection to the next syntactic unit. Starting on r inside a function call:
```
let area = pi *. r *. r
```
- M-h once → selects r.
- M-h again → selects pi *. r *. r.
- M-h again → selects the whole let binding.
M-h displays numbers indicating where the next enclosing region starts, helping you visualize where the cursor will move if you perform a hierarchy-up navigation. Unlike Merlin's type-enclosing (which operates on typed AST expressions and requires code to typecheck), Combobulate's expansion is purely syntactic: it operates on any concrete syntax node (including patterns, type declarations, and comments) even when the code is incomplete or doesn't compile.
Expanding an envelope (C-c o e)

Envelopes are context-aware templates. Press C-c o e then C-h to see what's available.

For example, to add a module template:
- Place your cursor where you want to add the template.
- Press C-c o e to list all available templates.
- Press M to activate the modules template.
- The template will be added with name as an editable hole:
```
module name = struct

end
```
- Press TAB to jump between holes.
Adding multiple cursors (C-c o t)

Cursors land on every sibling at the current level. This is perfect for bulk-editing collections. Place the cursor on any element of an array:
```
let primes = [| 2; 3; 5; 7; 11 |]
```
Press C-c o t t and a cursor is placed on each element. Anything you type happens to all five at once!
Swapping siblings — drag forward / backward (M-N / M-P)

Drag transposes the node at the cursor with its neighbor, preserving formatting. Useful for reordering elements or record fields:
```
let primes = [| 2; 3; 5; 7; 11 |]
```
With the cursor on 2, press M-N (or M-S-N) to swap them:
```
let primes = [| 3; 2; 5; 7; 11 |]
```
Cloning a node (C-c o c)

Duplicates the node at the cursor. On a record field:
```
type user = {
  name : string;
  age  : int;
}
```
Place the cursor on name : string and press C-c o c to duplicate it seamlessly.

Inspection & Search

Binding	Summary	What it does
`C-c o B q`	Query builder	Open the interactive tree-sitter query builder, with completion and highlighting, for ad-hoc searches and bulk edits.

Query Builder Example

Open a live tree-sitter query builder with C-c o B q. If you have value_definitions in your file, you can underline all of them with a blue line using the query:

(value_definition) @hl.blue.underline

Setup

Since Combobulate is built on tree-sitter you will need Emacs 29 or later, as that's when built-in tree-sitter support landed. Install Combobulate from the master branch and add the OCaml grammars to your config file.

To get started with OCaml, add the OCaml grammars to your config file:

(setq treesit-language-source-alist
      '((ocaml . ("https://github.com/tree-sitter/tree-sitter-ocaml"
                  "v0.26.0" "grammars/ocaml/src"))
        (ocaml_interface ("https://github.com/tree-sitter/tree-sitter-ocaml"
                            "v0.26.0" "grammars/interface/src"))))

Run M-x treesit-install-language-grammar for each.

Combobulate can be used with either neocaml-mode or tuareg-mode or tuareg-interface-mode as your major mode. When it's working you'll see © in the mode line, and C-c o o opens the full command palette.

Try it out

Open up a project you are working on. Place your cursor on a case in a match expression and try to teleport to the next sibling.

You can check out the PR adding OCaml support and the PR adding OxCaml support in the Combobulate repo to explore the implementation process in more detail.

Feedback Welcome

OCaml's syntax is flexible enough that there isn't always one obvious answer to "what should the next sibling be?" or "what counts as descending one level?". We had to make judgment calls on a number of corner cases, like what sibling navigation does inside a type ... and ... block, how hierarchy behaves around functors, where sibling navigation should land in deeply nested expressions. We're happy with the choices we made, but we know they won't match everyone's expectations perfectly. If something feels off in your workflow, or you think a particular movement should behave differently, we'd like to hear about it. Open an issue on the Combobulate repo, make a post on Discuss, or contact us to let us know.

Stay in touch with us on Bluesky, Mastodon, and LinkedIn or sign up to our mailing list to stay updated on our latest projects. We look forward to hearing from you!

A resilient kernel log reporter

Jane Street Tech Blog — 2026-10-01T00:00:00-00:00

The following is part of a series of posts about 2026 summer intern projects—for more, see “What the interns have wrought, special jumbo 2026 edition”

A FUSE server for our in-house storage backend

Jane Street Tech Blog — 2026-10-01T00:00:00-00:00

The following is part of a series of posts about 2026 summer intern projects—for more, see “What the interns have wrought, special jumbo 2026 edition”

Sabbatical Diary: Every End Had a Start

Richard Mortier — 2026-10-01T00:00:00-00:00

Having set off with a song in my heart I finish with a rather different song playing. No matter. Variety is the spice of life after all.

It was, in the end, more productive than I initially thought while also leaving me feeling like I didn’t get more than about 10% of what I set out to do, done. Pretty average for me really.

Notwithstanding that, it was certainly necessary: the few years leading up to it had been hard for a range of reasons. A year’s sabbatical was a very welcome reset.

I’d started out with three main targets, to whit,

Actually get on with a substantial piece of writing I’ve been meaning to do for a while.
Write research proposal #1 about Idea #1.
Write research proposal #2 about Idea #2.

How far did I get? Well, in addition to making very considerable progress resolving a personal situation that I’ve alluded to a few times but don’t feel like going into more detail about here,^[1]

I wrote one big chunk of it, and sketched several others. The chunk that’s been written has already proved useful. Hopefully I’ll get to the rest of it in the coming months.
I did nothing at all other than have a couple of conversations while at the UK Systems workshop. C’est la vie. On the backburner for now.
I wrote a talk which has now been given three times, the last of which has resulted in starting to look at relevant funding calls. Watch this space.

So nothing completed, one thing not even started, but progress on the other two that I can hopefully take forwards.

In addition to the other background work (a pile of sysadmin-y things, fiddling with some scripts and code and stuff^[2], supervising Ph.D. students, participating in a handful of workshops, University Council^[3], two other unanticipated research things came up:

With Rupen Mitra and Theo Aslanidis we’ve started trying to work out how to realise the “AI-RAN” vision, whereby AI workloads can be run using spare capacity in the 5G/6G RAN. This builds on Amjad’s excellent work, and seems quite promising.
With colleagues from Queens’ College I’ve started looking into the challenges of trying to make the William Gates Building simultaneously more energy efficient and more comfortable, building on work already done at Queens’ College. Still at the discussion stage so far, but one funding proposal is 85% written and I’ve started exploring with the University what might be possible to push the work forward.

Still, onwards and upwards as they say. New academic year, new term, new students, new roles: I remain on University Council until the end of my term coinciding with the end of the calendar year, and now I’m back from sabbatical I pick up my positions on the General Board of the Faculties^[4] and Information Services Committee. In the Department I also pick up a new role for me, Director of Undergraduate Education. I have no doubt that, in the fullness of time, we shall all learn a lot.

It has eaten up an awful lot of time and energy though, which is my excuse for not having got more done and I’m sticking to it. ↩
Notably pramblr which I’ve used to extract myself from Ramblr and generate pages like this one describing a short walk from Bowlees past Low Force to High Force and back, including an interactive GPX trace map and photo gallery. ↩
Work for which got busy in recent months as a result of several significant matters arising. ↩
AKA the General Board. ↩

What the interns have wrought, special jumbo 2026 edition