Biff

Biff 2.0 sneak peak

2026-04-20T18:25:00.000Z

I have for the past year or two been working on some large Biff changes, such as those discussed in Structuring large Clojure codebases with Biff and Biff support for XTDB v2 is in pre-release. Now that coding agents have gone mainstream (and in particular, now that I personally have started using them heavily), I've had a few more ideas for changes I'd like to make to Biff. And also thanks to coding agents, I've actually been able to make consistent progress instead of my Biff development time being bottlenecked by how late I can stay awake on weekend nights after my kids are sleeping. So we, fingers crossed, are getting close to some major Biff updates, and I figure I may as well slap a 2.0 label on it.

Here's what I've got in the works.

SQLite will be the default database

This is the biggest change. Biff will retain first-class support for XTDB, but it'll also have first-class support for SQLite, and I'll update the starter project to use SQLite by default. There will still be a (non-default) starter project that uses XTDB.

Biff has used XTDB since its (Biff's) initial release in 2020, back when the database was still called Crux. About a year ago I started working on migrating Biff from XTDB v1 to XTDB v2, which brings a whole new architecture, including column-oriented indexes that make analytical queries faster. Besides writing some Biff-specific helper code for XTDB v2, I migrated Yakread (a 10k-LOC article recommender system) to v2 and did a bunch of benchmarking for Yakread's queries. (A big thank you to the XTDB team who responded to lots of my questions during this time and also made a bunch of query optimizations!)

Long-story short: despite the optimizations, I had trouble getting Yakread's page load times to be as quick as I wanted. For the particular queries Yakread runs—which are mostly row-oriented—I've generally found v2's performance to be slower than v1. There is also a larger per-query latency overhead, perhaps another design tradeoff of the new architecture (you can still run v2 as an embedded node within your application process, but it’s designed primarily to be run on a separate machine like more traditional networked databases).

I also will admit that before this benchmarking exercise I had not actually used SQLite much, and I was unaware of how ridiculously fast it is. And one of the main downsides of SQLite when compared to XTDB—that SQLite is a mutable database—is mitigated by Litestream, which streams changes to object storage and lets you restore from (and even run ad-hoc queries on) historical snapshots saved with 30-second granularity.

I could see myself switching back to XTDB at some point in the future. It's still the early days for v2 and the XTDB team is doing lots of work, including on query performance. And SQLite's speed comes with tradeoffs:

Scaling beyond one machine is an unsolved problem. Litefs can let you put SQLite nodes in a cluster where writes get forwarded to a single leader and changes are streamed to the other nodes. However, to use it with Litestream, you have to disable automatic leader failover. So you basically have to choose between HA or PITR.
SQLite only supports a few basic datatypes: ints, floats, strings, and blobs (byte arrays). A large part of my work in integrating SQLite into Biff has been to set up automatic data type coercion so you can use richer types (UUID, boolean, instant, enum, map/set/vector) in your schema without having to do manual coercion when reading and writing.
Litestream's snapshots-at-30-second-granularity is fine for recovering from bad transactions like a DELETE FROM without the WHERE, but it's less helpful than XTDB/Datomic for the debugging-weird-production-issues use case: you can't include a transaction ID or similar in your production logs and then re-run queries with 100% confidence that the results you're seeing are what the application saw when it e.g. threw an unexpected exception.

I was chatting with Jeremy from the XTDB team last week, and he mentioned they've been working on having XTDB ingest changes directly from Postgres. It sounds like it shouldn't be much work to make that work with SQLite too, which means that you could stick an XTDB node alongside your SQLite-powered Biff app and then get more granular historical queries. Maybe XTDB could be a replacement for Litestream?

That could get even more interesting if eventually we can do the inverse as well, where data from our immutable XTDB log could be sent both to a bitemporal index for historical queries and also to SQLite "indexes"/databases for the application servers to use. That would solve the HA problem too.

Anyway. However it happens, I'm looking forward to the glorious future when we finally have an inside-out database that's fast for all query shapes, highly available, models time correctly, and can even do advanced things like let you put a UUID in it. In the meantime, I think SQLite is a reasonable default given Biff's focus on solo developers, and I would absolutely consider XTDB today for situations in which modeling time correctly is a top concern.

Alternate starter projects will get easier

Biff consists of a starter project, a bunch of helper code exposed through a single com.biffweb namespace, tooling for CLI tasks and deployment, and a big pile of documentation. The com.biffweb namespace is on its way out: I'll be publishing Biff helper code as individual libraries like com.biffweb.sqlite (and com.biffweb.xtdb), com.biffweb.authentication, com.biffweb.middleware, com.biffweb.config, etc.

Part of the motivation for this change is that Biff is more mature than it was five years ago and it's become more clear what the different cohesive parts of Biff should actually be. I started out with a single kitchen-sink library because splitting it up felt premature; I didn't think it would realistically make sense to use one of them outside a standard Biff project that would already be depending on all the Biff libraries anyway.

But over the past few months, I've been developing a couple new side projects from scratch without even using Biff. As I've done this, I've started extracting various things into standalone libraries, and this time I do see them as useful libraries in their own right. For example, the new biff.authentication library will be an easy way to add email-based authentication to any Clojure web app that uses Reitit—it even comes with a default sign-in page.

The other factor behind this change is agent-driven development. The difficulty of mixing-and-matching different libraries is dramatically easier now to the point where I wondered briefly if Biff was even needed anymore. Developing those new side projects via agent has disabused me of that notion: agents still need a lot of structure (e.g. in the form of these Biff libraries) to guide them. Even for starting new projects, why have everyone generate a different starter project via some prompt when you could have a single person generate the starter project, make sure it actually works, and then publish that?

That's still a meaningful change though: the effort required to create and maintain new project templates has decreased significantly. So I think it makes more sense for Biff to be split up into multiple libraries that can themselves be mixed-and-matched. I will myself provide Biff starter projects for SQLite and XTDB, respectively. If anyone else wants to make a Biff starter project variant with different library choices, they'll similarly be able to do that without much effort.

For vanity reasons, I'll need to continue having a single "main" Biff repo of some sort (did I mention Biff hit 1,000 github stars recently?). Maybe I'll have that repo be the default starter project.

New approaches for structuring application logic

Two of these Biff libraries that happen to contain some new stuff—instead of being a splitting-out of code that was already in Biff—are biff.graph, which lets you structure your domain model as a queryable graph, inspired by Pathom; and biff.fx, which helps you remove effectful code from your application logic via state machines.

Both libraries help you write purer code (and thus code that's easier to understand and test). biff.graph is a higher-level abstraction that helps with code that reads data. biff.fx is a lower-level thing that I mostly use when writing data. However they're also useful together: e.g. my GET request handlers are typically biff.fx machines that run a biff.graph query and pass the results to the (now pure) rendering code:

(def some-route
  ["/some-page/:id"
   {:get
    (fx/machine ::some-page

      :start
      (fn [{:keys [path-params] :as request}]
        {:stuff [:biff.fx/graph
                 {:stuff/id (parse-uuid (:id path-params))}
                 [:stuff/foo :stuff/bar]]
         :biff.fx/next :render-stuff})

      :render-stuff
      (fn [{:keys [stuff] :as request}]
        {:status 200
         :headers {"content/type" "text/html"}
         :body (render-html
                [:div "foo: " (:stuff/foo stuff)
                 ", bar: " (:stuff/bar stuff)])}))}])

biff.fx provides a defroute macro to make this kind of thing more concise, so the code I actually write looks more like this:

(fx/defroute some-page "/some-page/:id"
  [:biff.fx/graph
   {:params/stuff [:stuff/foo :stuff/bar]}]

  :get
  (fn [request stuff]
    [:div
     "foo: " (:stuff/foo stuff)
     ", bar: " (:stuff/bar stuff)]))

I'll save a fuller explanation for later; hopefully that gives you the flavor of what these libs do.

I've been using Pathom heavily over the past few years, both for work and pleasure. I've started referring to the code structure it enables as “data-oriented dependency injection.” It helps you structure your application in small easy-to-understand chunks that declare exactly what data they need as input and what data they provide as output. The main downside in my experience is that it can be difficult to understand exactly what Pathom is doing and debug when things go wrong.

For “serious” projects, that's a price worth paying. For the kinds of solo projects that Biff is aimed at, I've felt apprehensive about foisting another layer of abstraction on people for code structure benefits that they may or may not notice.

However, my own experience is that even for small apps, the benefit is real. So biff.graph is an attempt to provide the same graph computational model / “data-oriented dependency injection” with as small of an implementation as possible: biff.graph is about 400 lines of code currently, whereas Pathom is closer to 10k.

The main tradeoff I've made in service of that goal is to omit the query planning step that Pathom uses. biff.graph traverses directly over your input query, looking up which resolver(s) to call for each attribute as it goes. For each resolver, biff.graph runs what is more-or-less a separate query to get that resolver's inputs. This hopefully makes biff.graph easier to trace and understand what it's doing, but it also means biff.graph isn't able to optimize the query plan the way Pathom does. (biff.graph does support batch resolvers and caching at least).

biff.fx is more of an original creation. Instead of a single function, you have a map of functions, one for each state. Effects happen in the transitions. You define global “fx handlers” that do things like HTTP requests, database queries/transactions, etc, represented by keywords (e.g. :biff.fx/graph in the example). I’ve changed up the format for describing effects a few times; I think I've finally landed on something that feels ergonomic ([:do-something arg1 arg2] as a replacement for (do-something! ctx arg1 arg2)).

Authorization rules are so back

Biff entered this world as a replacement for Firebase, which I had enjoyed using but left me with the desire for a regular long-lived clojure backend. Firebase lets your frontend submit arbitrary transactions from the frontend, and then they're checked against some centralized authorization rules you define (e.g. “documents in the stuff table can only be edited if the current user's ID is the same as stuff.user_id”). I implemented a similar thing where you would submit transactions in a format similar to Firebase's, then I would translate them to XTDB's transaction format and pass a diff of the database changes to your authorization functions.

I ended up abandoning the SPA approach altogether for server-side rendering (with htmx), and that made authorization rules unnecessary since transactions were originating from the backend: I no longer needed to validate completely arbitrary transactions.

Once again, coding agents have changed the game. When working on mature codebases, of course we all read our generated code carefully before submitting a pull request. But when I've got a new app idea, I want to mostly just vibe code it until I get to the MVP. I'd like to be able to do a light review just to make sure the structure of the code is reasonable. With authorization rules, you can carefully review those central rules in the same way you'd carefully review the database schema, and then you can have confidence that the feature code isn't missing an authorization check. (Of course you still have to make sure the agent didn't bypass the authorization rules...)

This is only for writing data. For reading data, I typically have a few Pathom/biff.graph resolvers that e.g. read an entity ID from the incoming request's path parameters and ensure the user has access to that entity (like the :param/stuff resolver alluded to in the example above). Other related entities are queried as joins against that root entity, so if the authorization check fails, the rest of the query will fail too. So once again you have a way to put authorization logic in a central place that can be reused by your feature-specific code.

oh yeah and datastar

As mentioned above, Biff uses htmx. I like server-side rendering and I think it's a particularly good fit for Biff's solo developer focus. htmx however has a critical flaw: it's too popular. It has 47k github stars—that's half of what Tailwind has.

Datastar fixes this problem by being a much younger project—a niche of a niche. There is a much smaller chance that your colleagues will have heard of it. Datastar also has some smaller but still tangible benefits:

It has some frontend reactivity built in. With htmx, you typically use another tool like _hyperscript or Alpine.js to provide interactivity in cases where you really don't want to wait for a server roundtrip (e.g. a dropdown menu). Datastar has a concept of "signals" baked in so you don't need a second tool.
It has a smaller API surface; much of what htmx offers is replaced by "just use out-of-band swaps." So it might be easier to learn?
It works well for fancy CQRS stuff (still on my list of things to try out).

Of the changes I've mentioned, this one is the most experimental. I actually haven't even made an official decision if I really will switch Biff from htmx to Datastar; at this point I'm just making a prediction that I probably will.

More broadly I would like to explore how far I can push the server-side rendering model before I feel it breaking down. e.g. what approach would I use with it to handle forms with 50+ fields and lots of conditional logic, complex validation logic etc? How about charts? (What I'm getting at: would I regret asking an LLM to migrate our large codebase at work over to htmx/Datastar?).

I’d like to give an honorable mention to inline snapshot testing which I’ve been excited about for a year and a half but now find unnecessary—counterproductive, even—with coding agents. I had started working on some updates to my test code so you could do inline snapshot tests in plain .clj files instead of in .edn files (turns out that tooling support is best when you put your code in files meant for code). But with coding agents, I’ve found that I don’t want tests that auto-update when the actual result changes: it’s too easy for agents to ignore new results that are obviously incorrect. And of course I don’t care if my coding agent finds updating unit tests to be tedious. So the test-related stuff that Biff does will be limited to making your application code more pure so you (your agent) can write dumb (is (= (f x) y)) tests. I might add some structure/patterns for integration tests, though.

Another change driven by coding agents, not a change to the code but a change to my philosophy: I'm more interested in smaller projects. As mentioned, my time for working on personal projects has been extremely limited until a few months ago. I've only ever had a single Biff project at a time that I have attempted to work on regularly; new projects started after the old one failed. So the primary use case I designed Biff for was “serious side projects,” applications that may be solo projects now but will definitely be bringing in a 6-figure income and fulfilling all your entrepreneurial desires at... some point. That one project is the only thing I've ever had a chance of having time for.

Now I can code up an MVP for something over a weekend without ever sitting down at my desk. I built an app that helps me find good Star Wars: Unlimited decks to play. I'm building a blogging platform next. After that maybe I'll build a music recommender system. Or a state legislation tracker/summarizer.

I'm having a blast. Maybe that will affect design decisions I make down the road? I certainly am interested in the use case of doing agent-driven development from a mobile device, so maybe expect something in that area.

Biff support for XTDB v2 is in pre-release

2025-11-09T11:00:00.000Z

I've been working on/preparing for migrating Biff to XTDB v2 since that became generally available in June. After investigating the deployment options and performance characteristics, I've added some XTDB v2 helper functions to the Biff library (under a new com.biffweb.experimental namespace) and I've made a version of the starter project that uses XTDB v2.

You can create a new XTDB v2 Biff project by running clj -M -e '(load-string (slurp "https://biffweb.com/new.clj"))' -M xtdb2. See this gist for a diff between the old/main starter project and this new one.

To give you a quick overview of what Biff provides:

There are use-xtdb2 and use-xtdb2-listener components, roughly the same as we have already for XTDB v1.
The ctx map will have a :biff/conn key in it (a Hikari connection pool object) which you can pass to xtdb.api/q to do queries.
There is no longer a custom Biff transaction format. There is still a lightweight wrapper function, com.biffweb.experimental/submit-tx, which will apply Malli validation to any :put-docs / :patch-docs operations.

There's still plenty of work to do before XTDB v2 support in Biff is officially released and becomes the default:

Next up, I'm migrating Yakread to XTDB v2. This will help me find any more issues that need to be addressed/make sure that Biff is indeed ready for XTDB v2.
After that I need to update a bunch of documentation, including the tutorial project.

Since those next two steps will take a while, I wanted to do this "pre-release" for anyone who would like to get a head start on trying out Biff with XTDB v2. If you do so, let me know whatever questions/comments you have. Just note that the new functions in Biff's API are still experimental and might have breaking changes before I do the official release.

And for anyone who would rather not deal with migrating an existing app, Biff will still support XTDB v1. It's totally fine to stay on that.

Finally: I'll be at Clojure/Conj next week, at least if my flight doesn't get canceled. Come say hi.

Relaunching Yakread: an algorithmic reading app

2025-09-06T12:30:00.000Z

I've recently finished a year-long rewrite of the Yakread codebase and have released it under a source-available license. Yakread is a reading app that makes heavy use of algorithmic recommendation/filtering. I originally launched it in 2022 during the last leg of my time as a full-time entrepreneur. It's written with Biff, a Clojure web framework that I also created during that time.

I'm publishing Yakread's source code mainly so that it can serve as a non-toy example project for Biff users. It's about 10K lines of code as of writing. I'm also using Yakread to experiment with potential framework features before adding them into Biff.

The app

I like reading stuff on the internet. Social media tends to be pretty shallow, though. Long-form content (articles, blogs, newsletters) is better on average but can take more work to manage: RSS readers and email inboxes tend to get filled up pretty easily, and sorting things chronologically benefits the most frequent publishers. I've found there's a certain amount of mental overhead associated with long-form content that, when you have only a few minutes to read something, often makes it easier to just pull up Reddit.

Yakread is my attempt to make reading long-form content as frictionless as reading social media. It's structured as a daily email with links to articles. New users start out getting five links a day to articles that were liked by other Yakread users. You can also add your own newsletter/RSS subscriptions to Yakread, and there's support for bookmarking individual articles to read later a la Pocket or Instapaper. Posts from these content sources are also compiled algorithmically so that:

Blogs that publish a few times per year don't get buried by daily newsletters and other frequent publishers.
Subscriptions that you interact with the most don't get buried by dozens of other subscriptions that you signed up for on a whim.
Articles you miss get resurfaced repeatedly, so you don't feel like you have to "keep up" with anything.

The web app also features a "for you" feed, similar to the daily emails, which lets you read on-demand. There are pages that list your content chronologically in case you want to read something specific: the recommendation algorithm is there as a default, but it's not the only way to read. There's a "favorites" page which lists articles that you've thumbs-upped.

Yakread is monetized through native ads (mostly for newsletters) and a "premium" subscription which removes ads.

The code

The README has a section describing the parts of Yakread's code structure that differ from regular Biff projects. Here are a few high-level points, written without assuming any prior knowledge of Biff.

"The algorithm":

For recommending new articles (i.e. not ones from your own subscriptions or bookmarks), Yakread uses Spark MLlib's collaboritive filtering implementation. There are controls layered on top that bias the results toward articles that have been recommended a fewer number of times across all users (exploration vs. exploitation).
Ads are selected via the same collaborative filtering model. The predicted rating for each ad is treated as a probability that the user will click on that ad, then we calculate the expected value of showing each ad (i.e. probability of a click multiplied by how much the advertiser is bidding for each click) and charge the advertiser (in the case of a click) via a second-price auction.
The algorithm for selecting articles from your subscriptions and bookmarks is a few hundred lines of custom code, which e.g.: computes a pair of "affinity" scores (lower bound and upper bound) for each of your subscriptions based on your previous interactions; ranks subscriptions based on affinity score, again with controls for exploration vs. exploitation; ranks articles based on how recently they were recommended and how recently they were published; figures out the right balance between recommending subscription posts and recommending bookmarks.

Everything else:

Yakread is a server-side rendered app that uses htmx. There's nothing too fancy going on in the UI (the biggest form in the app has 8 fields), so I'm keeping it simple.
It uses XTDB for the database. You could think of XTDB's immutable architecture kind of like "distributed SQLite." Queries operate on a local point-in-time snapshot of the data, so you can run multiple queries while handling a given request without worrying about network latency (pretty helpful for a recommender system).
The app's data model is organized via Pathom. I sometimes think of Pathom as "data-oriented dependency injection." It gives you the benefit of ORM model objects from OOP languages but without having to pass around a database connection. You specify up front what entities and fields you want, then Pathom wires up the data in the correct shape for you.
I use state machines to separate pure application logic from effectful code. Application logic returns data describing the effects it needs to perform (Pathom queries, network requests, database transactions, etc), then the machine transitions to other states that perform the effects, then results are passed to the next pure logic state, and so on. This makes unit tests easy to write since you never have to mock anything or check the results of side effects.
The tests are largely inline snapshot tests.
Yakread currently deployed as a monolith to a single DigitalOcean droplet which handles both web requests and background jobs. If/when the time comes to deploy a separate worker (which probably needs to happen soon...), the same deployment artifact can be ran with a BIFF_PROFILE=worker env var set.
I deploy Yakread with rsync. Even though there's only a single web server, most deploys can be done without downtime via the REPL: after rsync finishes, new code is evaluated while the application runs. Full restarts typically only happen when new dependencies are added.
Yakread does a lot of email: I use SubethaSMTP to receive email newsletters (set an MX record and open up port 25) and MailerSend for sending the daily digests.

The theory

I've been interested in recommender systems for a long time, starting with music and then moving into written content. In both domains I've been attracted to the idea of a system that can handle most of the tedious organizational work while only requiring you to do the part that only you, as the human, can do: give feedback on which things you like and don't like. I think there's plenty of unrealized potential for recommender systems to help people learn, enjoy life, and coordinate.

Recommender systems often get a bad rap, and for good reason: most people's exposure to them comes in the form of large companies trying to shove the digital equivalent of potato chips down their throat. However I see that as a problem of business incentives rather than a problem with algorithmic recommendation in general; there aren't any technical barriers to writing algorithms that serve up salad instead of potato chips.

So I don't think a mass return to reverse-chronological feeds is the answer: competition is. Ideally we'd have a larger distribution of companies offering recommendation-powered services that had to compete based on the quality of their results. Instead we mostly have a few behemoths that are optimized to squeeze every bit of interaction from you that they can, the long-forgotten sanctity of your notifications tab be damned.

I hope that Yakread makes the internet a little bit better in that regard. Although I've pretty much given up on trying to take over the world, I still like the idea of being part of a movement. And there is interesting stuff happening: Bluesky, for instance, has had far more success than I thought it would when it was announced back in 2019. The popularity of email newsletters is encouraging.

The internet is still young. Maybe the next decade can be a phase of building digital public infrastructure.

Writing your tests in EDN files

2025-07-19T10:45:00.000Z

I've previously written about my latest approach to unit tests (which I have been informed is called "snapshot testing"):

[Y]ou define only the input data for your function, and then the expected return value is generated by calling your function. The expected value is saved to an EDN file and checked into source, at which point you ensure the expected value is, in fact, what you expect. Then going forward, the unit test simply checks that what the function returns still matches what’s in the EDN file. If it’s supposed to change, you regenerate the EDN file and inspect it before committing.

I still like that general approach, however my previous implementation of it ended up being a little too heavy-weight/too reliant on inversion of control. The test runner code had all these things built into it for dealing with fixtures, providing a seeded database value, and other concerns. Writing new tests ended up requiring a little too much cognitive overhead, and I reverted back to manual testing (via a mix of the REPL and the browser).

I have now simplified the approach so that writing tests is basically the same as running code in the REPL, and there's barely anything baked into the test runner itself that you have to remember. I put all my tests in EDN files like this (named with the pattern my_namespace_test.edn):

{:require
 [[com.yakread.model.recommend :refer :all]
  [com.yakread.lib.test :as t]
  [clojure.data.generators :as gen]],
 :tests
 [{:eval (weight 0), :result 1.0}
  _
  {:eval (weight 1), :result 0.9355069850316178}
  _
  {:eval (weight 5), :result 0.7165313105737893}
  ...]}

(weight is a simple function for the forgetting curve, which I'm using in Yakread's recommendation algorithm.)

I only write the :eval part of each test case. The test runner evaluates that code, adds in the :result part, and pprints it all back to the test file. Right now there isn't a concept of "passing" or "failing" tests. Instead, when the tests are right, you check them into git; if any test results change, you'll see it in the diff. Then you can decide whether to commit the new results (if the change is expected) or go fix the bug (if it wasn't). If I had CI tests for my personal projects, I'd probably add a flag to have the test runner report any test cases with changed results as failed.

In my lib.test namespace I've added a couple helper functions, such as a t/with-db function that populates an in-memory XTDB database value:

{:require
 [[com.yakread.work.digest :refer :all]
  [com.yakread.lib.test :as t]
  [clojure.data.generators :as gen]],
 :tests
 [{:eval
   (t/with-db
    [db
     [{:xt/id "user1", :user/email "user1@example.com"}
      {:xt/id "user2",
       :user/email "user2@example.com",
       :user/digest-last-sent #time/instant "2000-01-01T00:00:00Z"}]]
    (queue-send-digest
     {:biff/db db,
      :biff/now #time/instant "2000-01-01T16:00:01Z",
      :biff/queues
      {:work.digest/send-digest
       (java.util.concurrent.PriorityBlockingQueue. 11 (fn [a b]))}}
     :start)),
   :result
   {:biff.pipe/next
    ({:biff.pipe/current :biff.pipe/queue,
      :biff.pipe.queue/id :work.digest/send-digest,
      :biff.pipe.queue/job
      {:user/email "user1@example.com", :xt/id "user1"}})}}
  ...]}

(queue-send-digest returns a list of users who need to be sent an email digest of their RSS subscriptions and other content.)

I like this approach a lot more than the old one: you just write regular code, with test helper functions for seeded databases or whatever if you need them. It's been pretty convenient to write my "REPL" code in these _test.edn files and then have the results auto-update as I develop the function under test.

There are a couple other doodads: if the code in :eval throws an exception, the test runner writes the exception as data into the test case, albeit under an :ex key instead of under :results:

{:require
 [[com.yakread.model.recommend :refer :all]
  [com.yakread.lib.test :as t]
  [clojure.data.generators :as gen]],
 :tests
 [{:eval (weight 0)
   :ex
   {:cause "oh no",
    :data {:it's "sluggo"},
    :via
    [{:type clojure.lang.ExceptionInfo,
      :message "oh no",
      :data {:it's "sluggo"},
      :at
      [com.yakread.model.recommend$eval75461$weight__75462
       invoke
       "recommend.clj"
       60]}],
    :trace
    [[com.yakread.model.recommend$eval75461$weight__75462
      invoke
      "recommend.clj"
      60]
     [tmp418706$eval83727 invokeStatic "NO_SOURCE_FILE" 0]
     [tmp418706$eval83727 invoke "NO_SOURCE_FILE" -1]
     [clojure.lang.Compiler eval "Compiler.java" 7700]
     [clojure.lang.Compiler eval "Compiler.java" 7655]
     [clojure.core$eval invokeStatic "core.clj" 3232]
     [clojure.core$eval invoke "core.clj" 3228]]}}
  ...]}

The stack trace gets truncated so it only contains frames from your :eval code (mostly—I could truncate it a little more).

I also capture any tap>'d values and insert those into the test case, whether or not there was an exception. It's handy for inspecting intermediate values:

:tests
[{:eval (weight 1),
  :result 0.9355069850316178,
  :tapped ["hello there" "exponent: -1/15"]}
  ...

And that's it. If you want to try this out, you can copy run-examples! (the test runner function) into your own project. It searches your classpath for any files ending in _test.edn and runs the tests therein. I call it from a file watcher (Biff's on-save function) so your test results get updated whenever you save any file in the project.

EDN-infused plain html forms

2025-06-20T19:36:00.000Z

Merry solstice. After about a year, I'm roughly 80% done with the Yakread rewrite. Now all that's left is the remaining 80%. My last post is still a good explanation of the new Biff things I've been hacking on as part of that. Over the past couple weeks I've also been thinking about how to do forms.

So far Biff hasn't provided anything special for forms: if you need an email address, you do [:input {:name "email"} ...], you get the value as a string from (-> request :params :email), you parse it if needed (not needed in this case), then you stick it in a map like {:user/email email, ...} or whatever. Works fine for small forms; no need to over-complicate things.

But what if you have form with 50 fields? It would be nice if we could get EDN from the frontend, e.g. {:user/email "abc@example.com", :user/age 666} instead of {:email "abc@example.com", :age "666"}. Same as you get if you're doing a cljs frontend instead of htmx. htmx users deserve nice things too!

I've started rendering my form fields like [:input {:name (pr-str :user/email)} ...] (turns out :name will accept just about anything) and then using a wrap-parse-form middleware to parse the requests. That function attempts to parse each key in the form params with edn/read-string (fast-edn, actually), skipping keys that fail. For each parsed key, we then check your Biff app's Malli schema to see if that key is defined and what its type is. We use the type to figure out how to parse the form value. There are default parse functions for a few common types (int is Long/parseLong, :uuid is parse-uuid, etc). For other types, you can define a custom form parser in your schema, for example:

(def schema
  {::cents [:int {:biff.form/parser
                  #(-> %
                       (Float/parseFloat)
                       (* 100)
                       (Math/Round))}]
   :ad [:map {:closed true}
        [:ad/budget ::cents]
        ...

Now if I have a form field like [:input {:name (pr-str :ad/budget)} ...] and the user types in 12.34, on the backend I'll get {:ad/budget 1234, ...} automagically.

The form data isn't quite self-describing like EDN is: it relies on schema defined somewhere outside the form. I started out doing stuff like [:input {:name (pr-str {:field :user/favorite-number, :type :int})} ...] (seriously, you really can put anything in :name), but since I'm writing this middleware for Biff apps specifically, I didn't feel like that approach was adding much value. And I'm all about value.

What about forms with multiple entities? If your :name value is a vector like (pr-str [:user :user/email]), then wrap-parse-form will do an (assoc-in params [:user :user/email] ...). I don't at the moment have any special support for arrays of things, but you can do :name (pr-str [:users 3 :user/email]) and then you'll get {:users {3 {:user/email ...}}} in the request.

Other Biff news

Remaining things in the Yakread TODO list include finishing the ad system, adding premium plans, precomputing some recommendation models so that page loads are faster, and setting up email digests of your subscriptions. How long could that take? Surely not long! Oh, and then I just need to migrate all the users over from the currently-in-production Yakread as well as another similar app that stopped being profitable last year... but yes, certainly not long.

Once that's humming along and my monthly side project operational costs are back in the double digits, it'll be time for a much needed Biff release. I'll extract some of the stuff from Yakread and package it up real nice, and then go through some maintenance tasks that have been... festering, shall we say. And then it's time for...

xᴛᴅʙ ᴠᴇʀsɪᴏɴ 2: at last. Everyone's favorite 4-letter immutable database is out of beta. Which means it's really time to get Biff on it. I figure Yakread, once the rewrite is done, will make a nice open-source example of porting a nontrivial app from XTDB v1 to v2. So expect a big Biff release with migration guide and all that. Hopefully by the end of the year 😬. Maybe I could even look into integrating XTDB with Rama.

Until we meet again, perhaps at the equinox. Or at the conj. I've got my ticket already.

Two free t-shirt ideas:

"(got? :lisp)" -- styled to look like those "got milk?" fridge magnets.
"Breaking changes are for the weak" -- not sure how to style it, but this t-shirt definitely needs to exist.

Structuring large Clojure codebases with Biff

2025-01-28T20:15:00.000Z

I've been making some progress on rewriting Yakread (a fancy reading app) from ~scratch and open-sourcing it in the process. Along the way I'm experimenting with potential new features for Biff, my Clojure web framework, which Yakread is built with. In particular I'm working on approaches for keeping Biff apps more manageable as the codebase grows: the original Yakread codebase was about 10k lines and was already getting pretty crufty. I've also learned some things from contributing to our ~85k-line Clojure codebase at work.

I thought it'd be worth going over the main new architectural approaches in Yakread for anyone interested in poking around the code/as a preview of what to expect in Biff later on. The open-source repo has only a sliver of the production app's functionality so far, but it has examples of all the approaches described below.

Materialized views

"Old Yakread" has a lot of slow queries. For example, loading the subscriptions page on my account takes more than 10 seconds: for each of my hundreds of subscriptions, it has to run a query to figure out how many unread posts there are and when the most recent post was published. This is currently done the dumb way, i.e. Yakread queries for every single post and then computes the aggregate data.

The traditional way to solve this would be to denormalize the data model (add fields for “# unread items” and “last published at” to the subscription model) and keep it up to date manually (update those fields whenever a new post is published, whenever the user reads a post, etc). However, this approach can get out of hand.

I’ve addressed this in a cleaner way by implementing materialized views for XTDB. I store them in a dedicated RocksDB instance. For each piece of denormalized data you need, you define a pure "denormalizer" function* which takes in an item from XTDB’s transaction log along with the current RocksDB state and returns a map of key-value pairs that will be written back to RocksDB. Biff handles everything else: setting up RocksDB, running your denormalizer functions whenever there’s a new XTDB transaction, and providing a RocksDB snapshot for querying that’s consistent with your current XTDB snapshot (we retain XTDB's database-as-a-value semantics).

*Still deciding on the name... the codebase calls them "indexer" functions currently, but I decided "materialized views" are a clearer/more accurate term than "indexes."

This is a lower-level approach than something like Materialize, which lets you write regular SQL queries instead of defining these denormalizer functions (i.e. you’re defining a function of current DB state -> materialized view instead of new transaction, current materialized view -> new materialized view). However, when I experimented with Materialize several years ago I found that its memory overhead made it untenable for my use case. I’m sure it’s much better for, say, aggregating metrics from large real-time systems, even if it sadly didn’t work out for the simplify-random-guy’s-RSS-reader use case. (I’d also like to look into other things in this space like Rama and Feldera).

Writing the incremental view maintenance logic by hand is somewhat tedious, but the testing approach I'm using makes it really not bad. I’ve written code with Fugato that can take the database schema for your app and generate test data for use with test.check (Clojure’s property-based testing library). All you have to do is write an “oracle” function that takes a database snapshot and computes what the materialized view should look like for that snapshot. e.g. for the “subscription last published at” materialized view, the oracle function simply gets all the posts for each subscription and then finds the one with the latest published-at date. Then the testing code ensures that the materialized view computed by your denormalizer function matches.

Separating application logic from effects

100% of the application code in "New Yakread" is pure. The unit tests never have to set up mocks or check the results of side effects. Every unit test has the form “pass some data to this pure function and make sure the output matches this constant.”

I accomplish this by turning each function with application logic into a little state machine, where each state is either a unit of pure computation (app logic) or an effect handler. Whenever the pure app logic needs to do something effectful, like run an HTTP request or save something to the database, it returns some data describing the effect, which causes the machine to transition to an effectful state.

These effectful states perform the effects and then transition back to your pure application logic. They're centralized in one spot in your app, and they’re kept as dumb as possible: take some input data, do the effect, return the output.

For testing, each pure app logic state can be called individually without running the whole machine. This facilitates an approach to unit testing that I’ve been enjoying quite a bit: you define only the input data for your function, and then the expected return value is generated by calling your function. The expected value is saved to an EDN file and checked into source, at which point you ensure the expected value is, in fact, what you expect. Then going forward, the unit test simply checks that what the function returns still matches what’s in the EDN file. If it’s supposed to change, you regenerate the EDN file and inspect it before committing.

This is so convenient that for once I’ve actually been writing unit tests as I develop code instead of testing manually e.g. via the browser and then (maybe) writing tests after the fact. Thanks to Chris Badahdah for telling me about this approach.

This also has benefits for observability: the state-machine-executor code is a convenient place to put tracing and exception handling code. Whenever your app logic throws an exception, you'll be guaranteed to have the necessary input data to reproduce it, without having to add logging yourself.

Pathom

The final big change I’ve made is that the codebase is now split up into a bunch of Pathom resolvers. I've also written a few Pathom/Biff helper functions, such as this one which auto-generates resolvers for your XTDB entities.

It took me a while to “get” Pathom, but now I’m a big fan. My latest attempt at explaining it is to make a comparison with OOP. It's standard to have classes for your domain entities (User, Product, whatever) which have some fields that correspond to columns in the corresponding database record and some fields/methods that return derived data. A common example would be having columns for first_name and last_name in the database, and then the class could also have a derived full_name method which combines the two.

Pathom lets you do a similar thing: you define "resolvers" for each of the fields in your domain; some resolvers fetch stuff from the database, while other resolvers return derived data built on top of those values.

The big difference between Pathom and OOP-style "model code" (not sure what to call it) is that In the OOP style, your model code and app code gets interleaved. With Pathom, you first submit a query and get the results as a plain data structure, then you pass the data structure to your application code. Like the effects stuff in the previous section, this helps your application code stay pure.

I use Pathom heavily to separate my model and view code. Each GET request handler defines a Pathom query, and I have middleware which runs the query and passes the results to the handler function. Those handler functions thus never need to include code at the top level for querying the database, fetching things from S3, or augmenting database records with custom business logic: that’s all done by Pathom resolvers in other namespaces. This makes the request handlers easier to write and understand.

I also have resolvers that return UI fragments. For example, the subscriptions page needs to render a list of all your RSS and newsletter subscriptions. I have a resolver which renders an individual subscription card (which includes things like the subscription title and the number of unread posts). The parent resolver—the one that renders the entire list of subscriptions—doesn’t need to specify all the data needed to render each card. It just queries for a list of “subscription cards,” which again helps to keep each individual resolver small and easy to understand.

This approach is conceptually similar to what Fulcro does: each UI component declares its own query. The difference is that Fulcro is a SPA framework; Fulcro includes a bunch of plumbing so that you can extend this programming model to the frontend. Since Yakread is server-side rendered with htmx, it just uses plain backend Pathom resolvers for everything.

The Yakread rewrite has a long ways to go. However I think the experimentation phase may be largely over: the framework features I’ve described above are all in place; now I just need to bang out the rest of the app. Once that's done, I'll start extracting various parts out of Yakread and releasing them as part of Biff.

Yakread + Pathom [video]

2024-12-21T13:30:00.000Z

An explanation of how I'm using Pathom to keep Yakread's code organized. Also some slight schema modifications.

Code

Yakread schema [video]

2024-12-07T10:30:00.000Z

I've started open-sourcing Yakread. I'm going to try publishing videos about it along the way. This first video describes Yakread's schema. The production quality here is a little low... next time I'll try disabling dark mode and making the text larger.

Removing effects from business logic

2024-10-05T19:15:00.000Z

This is a thing (state machine? effects-processing pipeline?) I'm trying out to keep effectful code separate from business logic, to make testing easier. It's partially inspired by re-frame's event handling. The idea isn't new, but I'd never actually done it for my own code until now. I'd be curious to hear what other things people have done in this vein. I like this more than the typical approach of just mocking out the effectful bits.

I was working on the following function, which is the HTTP request handler for a "subscribe to RSS feed" button. You enter a URL into a form, then this handler:

Fetches the URL and figures out if it goes directly to an RSS feed or if it goes to a page that has an RSS feed(s) in its metadata.
If there aren't any feeds, show an error message. Otherwise, save the feeds to the database and redirect to the user's subscriptions page.

(def rss-route
  ["/dev/subscriptions/add/rss"
   {:name :app.subscriptions.add/rss
    :post
    (fn [{:keys [session
                 params]
          :as ctx}]
      (let [url (lib.rss/fix-url (:url params))
            http-response (http/get url {"User-Agent" "https://yakread.com"})
            feed-urls (->> (lib.rss/parse-urls (assoc http-response :url url))
                           (mapv :url)
                           (take 20)
                           vec)]
        (if (empty? feed-urls)
          {:status                     303
           :biff.response/route-name   :app.subscriptions.add/page
           :biff.response/route-params {:error "invalid-rss-feed"}}
          (do
            (biff/submit-tx ctx
              (for [url feed-urls]
                {:db/doc-type :conn/rss
                 :db.op/upsert {:conn/user (:uid session)
                                :conn.rss/url url}
                 :conn.rss/subscribed-at :db/now}))
            {:status                     303
             :biff.response/route-name   :app.subscriptions/page
             :biff.response/route-params {:added-feeds (count feed-urls)}}))))}])

(I have other middleware that takes the :biff.response/* values and converts them to :headers {"location" ...}.)

This handler has two effects: first it it calls clj-http.client/get on the URL that the user submitted; second, it calls com.biffweb/submit-tx if it gets a valid RSS url(s).

And here is the purified version of that handler, which I'll explain in a moment:

(def rss-route
  ["/dev/subscriptions/add/rss"
   {:name :app.subscriptions.add/rss
    :post
    (fn [{:keys [session
                 params
                 biff.chain/state]
          :as ctx}]
      (case state
        nil
        {:biff.chain/queue      [:biff.chain/http ::add-urls]
         :biff.chain.http/input {:url     (lib.rss/fix-url (:url params))
                                 :method  :get
                                 :headers {"User-Agent" "https://yakread.com/"}}}

        ::add-urls
        (let [feed-urls (->> (lib.rss/parse-urls (:biff.chain.http/output ctx))
                             (mapv :url)
                             (take 20)
                             vec)]
          (if (empty? feed-urls)
            {:status                     303
             :biff.response/route-name   :app.subscriptions.add/page
             :biff.response/route-params {:error "invalid-rss-feed"}}
            {:biff.chain/queue    [:biff.chain/tx ::success]
             :biff.chain.tx/input (vec
                                   (for [url feed-urls]
                                     {:db/doc-type :conn/rss
                                      :db.op/upsert {:conn/user (:uid session)
                                                     :conn.rss/url url}
                                      :conn.rss/subscribed-at :db/now}))
             ::feed-urls feed-urls}))

        ::success
        {:status                     303
         :biff.response/route-name   :app.subscriptions/page
         :biff.response/route-params {:added-feeds (count (::feed-urls ctx))}}))}])

The handler is meant to be called multiple times by some sort of orchestrator (I'll get to that in a minute). Every time the original function would have performed an effect, this function instead returns some data that describes the effect. The orchestrator runs the effect and then passes the effect's output back to the handler function.

The handler uses the value of :biff.chain/state to know which "segment" of the business logic is being executed currently, and :biff.chain/queue tells the orchestrator which segments/effects should happen next. The first time the function is called, state is nil; i.e. we use that as the start state (I could've set it to something like :biff.chain/start, but eh).

Our effects code is stored in a lib.chain/globals map which is shared across the codebase:

(ns com.yakread.lib.chain
  (:require [clj-http.client :as http]
            [com.biffweb :as biff]))

...

(def globals
  {:biff.chain/http
   (fn [{:keys [biff.chain.http/input] :as ctx}]
     (assoc ctx :biff.chain.http/output (-> (http/request input)
                                            (assoc :url (:url input))
                                            (dissoc :http-client))))

   :biff.chain/tx
   (fn [{:keys [biff.chain.tx/input] :as ctx}]
     (assoc ctx :biff.chain.tx/output (biff/submit-tx ctx input)))})

It's as dumb as possible: take some input, do the effect, return the output. I wrap the functions so that the input and output goes under namespaced keys.

Going back to the handler function, the chain of states will look like this in the success case:

nil -> :biff.chain/http -> ::add-urls -> :biff.chain/tx -> ::success

And it'll look like this in the failure case:

nil -> :biff.chain/http -> ::add-urls

The orchestrator gets the function associated with each state and calls them in order, taking the return value of each function and passing it to the next.

(defn orchestrate [{:keys [biff.chain/globals] :as ctx} f]
  (loop [{[state & remaining] :biff.chain/queue :as result} (f ctx)]
    (if-not state
      result
      (recur ((get globals state f)
              (merge ctx result {:biff.chain/state state
                                 :biff.chain/queue remaining}))))))

(defn wrap-chain [handler]
  (fn [ctx]
    (orchestrate ctx handler)))

I apply the wrap-chain middleware to all my handlers. Since the "non-chain" handlers don't return :biff.chain/queue, they work the same as if they were called directly.

Testing is now really easy. Just a bunch of plain function calls; no need to set up e.g. a mock database or anything. For example:

(deftest rss-route
  (let [[_ {:keys [post]}] sut/rss-route]
    (is (= {:biff.chain/queue
            [:biff.chain/http :com.yakread.app.subscriptions.add/add-urls],
            :biff.chain.http/input
            {:url "https://example.com",
             :method :get,
             :headers {"User-Agent" "https://yakread.com/"}}}
           (post {:params {:url "example.com"}})))
    ...))

For the parts of the handler that depend on effect output, I wrote a function that generates the fixtures. I call it manually and check the results into source control. Then tests can use the read-fixtures function:

(defn write-fixtures! []
  (let [{:biff.chain/keys [http]} (:biff.chain/globals main/initial-system)
        http-get (fn [url]
                   (http {:biff.chain.http/input {:url url
                                                  :method :get
                                                  :headers {"User-Agent" "https://yakread.com"}}}))]
    (with-open [o (io/writer "test/com/yakread/app/subscriptions/add_test/fixtures.edn")]
      (pprint
       {:example-com          (http-get "https://example.com")
        :obryant-dev          (http-get "https://obryant.dev")
        :obryant-dev-feed-xml (http-get "https://obryant.dev/feed.xml")}
       o))))

(defn read-fixtures []
  (edn/read-string (slurp (io/resource "com/yakread/app/subscriptions/add_test/fixtures.edn"))))

...

(deftest rss-route
  (let [[_ {:keys [post]}] sut/rss-route
        {:keys [example-com
                obryant-dev
                obryant-dev-feed-xml]} (read-fixtures)]
    ...
    (is (= {:biff.chain/queue
            [:biff.chain/tx :com.yakread.app.subscriptions.add/success],
            :biff.chain.tx/input
            [{:db/doc-type :conn/rss,
              :db.op/upsert
              {:conn/user 1, :conn.rss/url "https://obryant.dev/feed.xml"},
              :conn.rss/subscribed-at :db/now}],
            :com.yakread.app.subscriptions.add/feed-urls
            ["https://obryant.dev/feed.xml"]}
           (post (merge {:biff.chain/state ::sut/add-urls
                         :session {:uid 1}}
                        obryant-dev))))))

write-fixtures! calls the functions from :biff.chain/globals, e.g. instead of calling clj-http.client/get directly, to ensure the fixtures match what the effect code will produce. Whenever I update the effect code, I can call write-fixtures! again and make sure the tests still pass.

Of course you don't have to write tests that are all of the form (is (= <constant> (my-function ...))); you could do fancier things too like property-based testing. Or you could pass in a :biff.chain/globals map that has mocked effects so you can write integration tests in the imperative style while keeping your unit tests in the functional style.

RocksDB indexes are done, open-sourcing Yakread next

2024-10-05T19:15:00.000Z

Last time on Biff: The Newsletter:

I've just barely started to re-implement the entire index feature by using RocksDB directly for the secondary indexes. RocksDB is a mutable KV store, so we can update documents in place without retaining the history—a useful feature for our main source-of-truth database; less critical for these indexes.

That work is now done. Check out the code and the tests. From users' perspective, the main change is that your indexer functions—which take in your XTDB transactions as input, and create custom indexes/derived data/materialized views as output—now return maps instead of XTDB transactions. The map keys and values are serialized to bytes (via Nippy for the values, and via a custom function for keys since we need the serialization to be stable) and stored in RocksDB.

On the usage side, there's a nifty biff/open-db-with-index function that's similar to xt/open-db, except that the db value it returns also lets you query a snapshot of your RocksDB index:

(with-open [db (biff/open-db-with-index ctx)]
  ;; Query xt
  (xt/q db ...)
  ;; Read from your index
  (biff/index-get db :my-index-id :some-key))

And Biff ensures that the two snapshots are consistent; i.e. the Biff index snapshot was built from all the transactions that the XT snapshot was and none more.

Now that indexes are ready to go, I'm back to working on Yakread. I'm doing a complete rewrite using both the new indexes feature and Pathom to make the code base more maintainable. When the rewrite is done I'll make another release of Biff with all the index + Pathom features, then I'll release Yakread as open-source and use it as a "flagship" example Biff app. I probably won't emphasize indexes or Pathom in e.g. the tutorial, since for a lot of Biff users they would probably just complicate things unnecessarily. But I will likely write some documentation about using Biff for "serious"/commercial projects that are expected to grow large.

So far I'm organizing Yakread's source code into three main folders/namespaces:

com.yakread.model.*: this contains indexes and Pathom resolvers. The resolvers handle most of the database/index access for your application.
com.yakread.app.*: this part also contains a bunch of Pathom resolvers, but instead of returning "regular" data, they return HTML (hiccup) snippets. Each page you navigate to in the app is generated by wiring up a bunch of different resolvers. This is where I also put the non-GET HTTP routes, scheduled tasks, and other application-feature type things.
com.yakread.lib.*: this is where most of the business logic/helper functions go. Each child namespace (e.g. com.yakread.lib.user) is treated as its own independent library, with a public API and a private implementation. model and app code can both require namespaces from lib, but not the other way around.

Once this is all done there a few other things I'd like to explore in the realm of making-Biff-suitable-for-large-apps, like "how do you handle complex forms with dozens of fields and conditional logic etc" and "how do you gradually add more client-side interactivity to your app without rewriting the whole thing in React?"