Why `streaming` Is My Favourite Haskell Streaming Library

It’s really easy to misuse lazy I/O (e.g., hGetContents) in nontrivial Haskell programs. You can accidentally close a Handle before the computation which reads from it has been forced, and it’s hard to predict exactly when data will be produced or consumed by IO actions. Streaming libraries in Haskell avoid these problems by explicitly interleaving the yielding of data and execution of effects, as well as helping control the memory usage of a program by limiting the amount of data “in flight”.

A number of veteran Haskellers have built streaming libraries, and off the top of my head I’m aware of conduit, io-streams, iteratee, machines, pipes, streaming, and streamly. Of those, I think conduit, pipes, streaming, and streamly are the most commonly used ones today. It can be hard to know which library to choose when there’s so many options, so here is my heuristic:

1. If you’re doing simple streaming (e.g., from a network connection straight into a file), use whatever your library uses (usually conduit); or
2. If you’re doing anything more complicated, or you’re doing greenfield work, use streaming.

I’ll explain why after the jump.

Which Build Tool For A Bootstrappable Project?

I have a nascent side project which is intended to participate in a bootstrap chain. This means it shouldn’t depend on too many things, that the transitive closure of its build dependencies must also be small, and at no point in the process should any build depend on an opaque binary blob.

Choices on the language side are pretty constrained. Zig is currently not a candidate (despite the language itself being rather promising), because it has removed its C++-based bootstrap in favour of keeping a WASM-based build of a previous compiler version. It’s great that their compiler output is reproducible — Zig-built-by-Zig is byte-for-byte identical with Zig-built-via-WASM — but for now, it’s not truly bootstrappable. (Andrew Kelley says he hopes someone writes a Zig compiler in C when Zig stabilises. I sincerely hope this happens.)

Rust is right out, for reasons described in the Zig article:

Use a prior build of the compiler - This is the approach taken by Rust as well as many other languages.

One big downside is losing the ability to build any commit from source without meta-complexity creeping in. For example, let’s say that you are trying to do git bisect. At some point, git checks out an older commit, but the script fails to build from source because the binary that is being used to build the compiler is now the wrong version. Sure, this can be addressed, but this introduces unwanted complexity that contributors would rather not deal with.

Additionally, building the compiler is limited by what targets prior binaries are available for. For example, if there is not a riscv64 build of the compiler available, then you can’t build from source on riscv64 hardware.

The bottom line here is that it does not adequately support the use case of being able to build any commit on any system.

As far as I can see, the best choice for writing bootstrap-related software in 2024 is still C99, with as few dependencies as possible. Any (hopefully few) necessary dependencies should also be bootstrappable, written in C99 and ideally provide pkg-config-style .pc files to describe the necessary compiler/linker flags. But at least there are several C compilers as well as several implementations of pkg-config (the FreeDesktop one, pkgconf, u-config, etc.).

Since we are compiling C, what should we use for the build system? Autotools is under scrutiny again in the wake of the xz-utils compromise, as code to trigger the payload was smuggled into the dist tarball as “autotools junk” that nobody looks at. Should bootstrappable projects still use autotools, or is there something better in 2024?

The Road to Amazonka 2.0

Last month, Brendan Hay and I released the 2.0 version of Amazonka, the de facto but unofficial AWS SDK for Haskell. Before that, Amazonka had seen intermittent commits and some pretty major improvements, but hadn’t managed an actual release in about four years. Because of the lack of visible progress, more serious industrial users maintained private forks instead of contributing to the main repository. It took about two years of work to pick up what was left behind, triage all the open issues, make several necessary major improvements, and get the whole project back into a shippable state.

I believe that many open source ecosystems have projects like Amazonka: projects which are large, important to their ecosystem, and stuck. These are my notes about how to unstick such a project, using Amazonka as a case study. It’s a fair amount of work, but a surprising amount of help can come out of the woodwork once someone makes the first move. That person could be you, and if it’s not you, then who’s it gonna be?

The Maddest My Code Made Anyone

I was lucky enough to grow up through the early 2000’s, during the golden age of Half-Life mods. Industry classics like Counter-Strike (CS) had just been invented, every month brought new mods to try, and files were too big to download on a dial-up connection so you’d leech them off a server at a big local LAN party. One of my personal favourites was Natural Selection (NS), a sci-fi marines-vs-aliens deal where one of the marine players had to command the others, RTS-style, from a command chair. If you’ve never heard of it before, this 2022 video review of NS will give you a sense of what it was like.

As Half-Life modding matured, some really interesting inventions appeared. MetaMod was a C++ framework that interposed itself between the server binary and the actual mod DLL, allowing you to inject custom behaviour into an existing mod. I didn’t understand enough C++ to write MetaMod plugins, but that didn’t matter: AMX Mod and later AMX Mod X let you write custom plugins using a simpler C-style language called Pawn (known back then as “Small”). This enabled an explosion of ways for operators to tweak their game servers: quality-of-life improvements for players, reserved player slots for members, and delightfully bonkers gameplay changes. I remember having my mind blown the first time I stumbled upon a game of CS with a class-based perks system, inspired by Warcraft 3, and that was just one instance of the creativity that came from the AMX(X) modding scenes.

And with the Half-Life-specific background covered, we are now ready to talk about NS: Combat and my gloriously dumb contribution to the AMXX world.

Monoids in the Category of...

The unfortunate meme phrase “a monad is just a monoid in the category of endofunctors, what’s the problem?” comes from two sources:

• The fact and most of the phrasing comes from Mac Lane’s Categories for the Working Mathematician, but
• “What’s the problem?” is a cheeky addition from a funny 2009 blog post: A Brief, Incomplete, and Mostly Wrong History of Programming Languages

The meme words have become an annoying blot on the fringes of the Haskell universe. Learning resources don’t mention it, the core Haskell community doesn’t like it because it adds little and spooks newcomers, and it’s completely unnecessary to understand it if you just want to write Haskell code. But it is interesting, and it pops up in enough cross-language programming communities that there’s still a lot of curiosity about the meme words. I wrote an explanation on reddit recently, it became my highest-voted comment overnight, and someone said that it deserved its own blog post. This is that post.

This is not a monad tutorial. You do not need to read this, especially if you’re new to Haskell. Do something more useful with your time. But if you will not be satisfied until you understand the meme words, let’s proceed. I’ll assume knowledge of categories, functors, and natural transformations.