Starting the Self-Directed Research Podcast
James Munns and I have launched a new podcast called the Self-Directed Research Podcast. You can find it on Spotify, Apple Podcasts, or via the plain RSS feed.
The weekly format is driven by concrete work we're each doing in the open-source software, Rust, and embedded hardware spaces. The goal: bounce ideas off each other publicly, gather eyeballs, and eventually attract funding. It's less interview show and more structured journaling — every week forces me to take one of my many drafts or side projects from the pile and get it to a state where I can talk about it for twenty minutes. James's tangents, meanwhile, are a free embedded-systems education.
James has been expanding the postcard ecosystem, with particular focus on the RPC layer, in service of larger projects he's holding back on for now. My side includes fluke, an HTTP/1 and HTTP/2 implementation in Rust built on io_uring and kTLS, sponsored first by fly.io and then by Shopify.
The first episode covers a topic I got wrong in my earlier article on Rust build times: the advice to break everything into small crates didn't hold up. Some of the other advice in that piece has aged fine, and there have been meaningful developments with linkers and rustc backends that we get to review in the episode.
The podcast is the only time-sensitive launch here, and it works even if nobody listens — but you're welcome to eavesdrop.
We're actively seeking corporate sponsors. If you'd like a 30-second ad read for your company or product, send me an email and I'll send over a media kit.
Open-source updates
Beyond the podcast, I've open-sourced several projects that have been in the works:
merde_json, a lightweight alternative to
serde_jsonrubicon, a library for a particular dynamic linking pattern in Rust
rc-zip, a sans-io Rust zip implementation that I maintain
Infrastructure-wise, I moved off fly.io onto my own setup over the past 18 months. I built a CDN on Kubernetes, which works surprisingly well for roughly 85EUR/month — there are several write-ups worth doing from that work.
Personal news
I got divorced last year. It was amicable — we went for drinks after signing and posted a "just divorced" selfie. I'm proud of how reasonable we both were through it, even though it hurt badly. Neither of us was pretending otherwise.
The accompanying life changes — new place, living alone for the first time in a long while, financial stress — pushed me past what I could manage on my own. I'd previously been able to handle my depression without medication, compelled to stick around even when it wasn't working for me. That approach stopped being viable.
With encouragement, I sought help and got on antidepressants (Escitalopram 10mg). After a rocky start, things became significantly easier — in a "fuck, how many years have I wasted" kind of way. Grief set in before acceptance: I could have been emotionally stable, energetic, and productive this entire time. Eight months in, as I slowly lower the dosage, I can feel that being all those things without medication is harder. But I know there's something that works for me now.
With more time and space this past year, I've also been thinking about identity. I never felt particularly attached to male descriptors, and I've decided to start using they/them pronouns. I'm a fan of conventions, so you can query my pronouns over DNS:
lith on main via 🦀 v1.80.0
❯ doggo pronouns.fasterthanli.me TXT
NAME TYPE CLASS TTL ADDRESS NAMESERVER
pronouns.fasterthanli.me. TXT IN 21600s "they/them" 100.100.100.100:53
This isn't a huge deal for me; he/him felt wrong, so I'm switching to something neutral. I don't care much about gender — my own, at least. The bigger thing is being comfortable saying this in public. I haven't figured out other languages: I don't like "iel" in French, so I'll use "il/lui" there for now.
If you use he/him out of habit, nobody will spontaneously combust. Just correct yourself and move on — as long as the mistake isn't deliberate or persistent, we're golden.
I also gave a talk at P99 CONF about the Rust/io_uring/HTTP work behind fluke.
If you'd like to support this work directly, the sponsor list at the top of this page is current — with special thanks to Dominik Wagner, Colin VanDervoort, John VanEnk, and everyone else (including 241 more readers) who helps keep things running.
Rethinking the stack
One reason for the publishing slowdown was growing dissatisfaction with the website itself. The blog has migrated through jekyll, nanoc, hugo, zola, and finally, since 2019, its own Rust codebase — at a time when that was still a rather inadvisable choice. Over the years features accumulated: a pluggable sponsor backend (adding GitHub Sponsors as a hedge against Patreon's troubles), an ever-improving image pipeline, video encoding, and more. At one point a custom YouTube player clone was built, serving H.264, VP9 and AV1 up to 4K@60 — it was later delisted, but there are plans to bring it back.
The predictable consequence of a large Rust project set in: compile times skyrocketed, as did link times. Even in debug mode, with debug = 1, all LTO disabled, incremental builds on, and the codebase split across crates, pulling in certain dependencies made performance unacceptable. Fixing this has been the focus for many months.
The infrastructure journey involved moving from a plain Dockerfile to nix, and then to Earthly. Nomad was tried briefly before committing to Kubernetes via k3s, along with helm. Artifact Hub supplied a Postgres controller and backup solutions for local volumes to object storage. Terraform handled what it does well, with Ansible covering the rest — resisting the urge to rewrite everything in Rust.
A key enabler for a proper CDN-like setup was the trafficDistribution field, alpha as of Kubernetes 1.30. Route53 routes traffic to a geographically close edge node, which proxies to a topologically close service — ideally local, falling back to a neighbor if unhealthy or overloaded.
The result: zero-downtime deployments (no automatic rollbacks; manifests are scp'd to a directory k3s watches). A code change ships around the world in 58 seconds, covering building in CI, packaging and pushing a container image, updating the k8s manifest, pulling the image on all nodes, creating new containers, waiting for health checks, and destroying old ones. All of this runs for roughly 85EUR per month, most of which is a dedicated server for the control plane and standby video encodes. It's a hard cap on monthly costs — a middle ground between unlimited serverless spending and a single small VPS.
The site layout was also redone with a system font stack, leaving Berkeley Mono as a variable font for code, weighing about 32KB. (A patched Iosevka font with NerdFont icons is still referenced in some places.) Diagrams are done in draw.io and rendered to SVG via Node.js, avoiding headless Chrome. The SVG pipeline now keeps text selectable, pulling fonts from the website via @font-face instead of converting text to paths. A custom minifier was written when svgo broke files and svgcleaner proved abandoned.
Asset processing is now fully on-demand. No more committing PNG, JPG, WebP, or AVIF to the content repository. Everything is served from a cookie-less subdomain, transcoded from JPEG-XL to AVIF or WebP, with in-memory, on-disk, and S3 caching. Safari will use JPEG-XL when available, so that's served directly too. There's no need for JPEG or PNG in 2024 — lossless compression is rarely required here.
This setup might collapse under load, as it has in the past, but that's an acceptable risk. Even if the site goes down, the content lives on YouTube, mirrors, and rogue translations. The site is best effort, and some pages still execute SQL queries when they shouldn't.
The entire rebuild now enables a 3-second cycle from hitting Cmd+S to seeing changes in the browser — thanks to dynamic linking.
Dynamic linking in Rust
The rubicon project enables a form of dynamic linking for Rust that addresses the compile-time and link-time bloat. It lets crates export their thread-locals and "process-locals" (statics) so other copies of the same code in separate shared objects can import them.
This makes crates like tokio, parking_lot, eyre, and tracing work across shared objects despite code duplication. The project no longer takes half a minute to link regardless of how many modules are added — the dynamic boundary is chosen explicitly. Trait object safety work has led to a particular appreciation of current Rust limitations, but the approach works. There's substantial layer reuse across container images, keeping deploys fast.
A new content model
The previous early-access system — two-week sponsor exclusivity for articles — proved insufficient. The 10EUR/month tier didn't justify itself every two weeks. Instead, content now falls into two categories:
- Articles: free for everyone from day one
- Features: free in video form, sponsor-exclusive as text for 6 months
Articles cover time-sensitive topics — new crates, stabilized Rust features, items with news value. Features are longer reads, often 60+ minutes, with more visual elements. Videos make sense to keep free: they're an enjoyable medium, they pay for themselves through ads and YouTube Premium, and there's no hosting cost. The text format is substantial additional work, presented as a premium feature. Everything still becomes free eventually, six months later — enough to build a backlog that justifies support.
Corporate backing
Since becoming fully independent, the work has been supported by corporate sponsors: first fly.io, and then Shopify. That support enabled sustained work on rubicon, which could transform the Rust development experience. The long-term goal is to make rubicon unnecessary and support this workflow at the cargo/rustc level — requiring further funding. Presentations on related work, like fluke, are scheduled for P99 CONF later this year. Additional sponsorships are being sought, along with renewing past ones.
When Your Pretty-Printed JSON Disappears
Picture this: you’ve got a JSON file you need to inspect, so you reach for jq to get some color-coded output and readable formatting. The command seems straightforward — pipe the file in, get the pretty result out.
But sometimes nothing comes out. Or worse, the output is truncated, garbled, or the terminal locks up entirely. The culprit isn’t always your data or your jq filter — sometimes it’s the terminal itself, doing something you didn’t ask for.
Cool Bear’s hot tip: that’s a useless use of cat
No warm-up today, Cool Bear. Fine — let’s read the jq man page. It takes a “filter” followed by some files. The filter we want is ., which — just like in filesystem paths — means “the current thing.”
The Filter Is Only Half the Story
Once you’ve got the right filter, the next question is what happens between jq writing its output and your eyes reading it on screen. That pipeline isn’t as transparent as it seems. Data flows through a series of buffers, line disciplines, and terminal emulator states — and any one of them can inject delays, mangle bytes, or swallow output entirely.
The terminal’s job is to interpret the byte stream it receives. Control characters like \n move the cursor, \r returns it to the column zero, and escape sequences can clear the screen, set colors, or reposition the cursor arbitrarily. When a program like jq prints output meant for a human, it relies on the terminal to handle those sequences correctly. But when the output is redirected to a file or another program, those same sequences become noise — or worse, they can trigger unintended behavior.
A Matter of Buffering
One classic source of confusion is buffering policy. When a program’s stdout is a terminal, it’s typically line-buffered: output is flushed on each newline. When stdout is a pipe or a file, it becomes block-buffered: the program accumulates output until a buffer fills up (usually 4 or 8 KB) before writing anything. That’s why you might see nothing for a long time when piping a slow data source — the output sits in memory until the buffer fills.
The fix is to force line buffering or disable buffering entirely, depending on the tool. Some programs have flags for this — stdbuf is a general utility on Linux that can change the buffering of any command. Other tools, like Python, take an environment variable (PYTHONUNBUFFERED) or a command-line option (-u).
When the Terminal Bites Back
Terminal emulators don’t just display text — they parse it. Some control sequences are harmless cursor movements; others are requests to change the terminal’s title, scroll buffer, or even set the window size. A malformed or malicious byte sequence can send the terminal into a state that leaves it unusable.
That’s why piping a file of unknown content into cat or less is risky. Binary data may contain escape sequences that the terminal interprets as commands — clearing the screen, switching character sets, or enabling bracketed paste mode. If you’ve ever had your prompt mysteriously “eat” input, bracketed paste is often the cause: the terminal notifies the program when the user pastes text, and if the program doesn’t handle the notification properly, the pasted text vanishes.
The safer path is to be deliberate about what you send to a terminal. For inspecting potentially hostile files, use a pager like less with the -R flag to pass through ANSI colors only, or use hexdump/xxd to view raw bytes. And for your own programs, take the time to detect whether stdout is a terminal (isatty()) and adjust your output accordingly — no colors and no control sequences when it isn’t.
Buffering and TTY Detection: Two Sides of the Same Coin
Both buffering and TTY detection come down to the same underlying question: what kind of device is on the other end of stdout? A terminal can handle interactive niceties; a file or a pipe cannot. Tools that ignore this distinction produce output that’s either painfully slow (block buffering on a long-running process) or painful to read (raw ANSI codes in a log file).
For Rust code, the standard library’s println! always goes through a Stdout handle that is line-buffered when connected to a terminal and block-buffered otherwise. Libraries like atty exist to detect a TTY explicitly. The pattern is always the same: check the destination, then decide how to write and what to write.
Getting this wrong isn’t just a convenience issue. If a program writes escape sequences to a non-terminal destination, downstream tools that parse the data may be confused. If it doesn’t write them to a terminal, the user loses colors, progress bars, or cursor manipulation. The dirty dozen of terminal handling — line discipline, signal-driven I/O (SIGWINCH for resize), pseudo-terminal pairing, and all the rest — are worth keeping in mind every time you string together two commands with |.
And if your nicely piped jq output ever just doesn’t show up, start with the buffer.
