Rethinking beta reader feedback with a purpose-built site

Back in 2020, I wrote about using beta readers to get feedback on my zines before publication. The process involved emailing PDFs to people and manually collating their replies, one comment at a time. Over the past couple of years, I've worked extensively with Marie Flanagan to turn that process into something far more efficient: a custom site built specifically for collecting, categorizing, and acting on reader feedback.

The site itself isn't open source, but this post covers how we designed it and the thinking behind the feedback categories we settled on. We'll also look at what the admin interface taught us about which pages were working and which needed serious revision.

Where the original process broke down

The original workflow had two major problems. First, managing feedback was awkward: for every comment I received, I had to decide whether to address it, track it, and mark it done once handled. My solution was a Trello board with one card per page of each zine, adding comments to cards manually or with a Python script. It worked, sort of, but the overhead limited me to asking at most ten people for feedback.

The second problem was that feedback arrived without any categorization. A comment like "CSS often seems random to me" could mean several different things: does the reader want more explanation, doubt the example, or think the whole topic is out of place? Without structure, it was hard to know what action to take.

Help This Book as the model

In 2021, I read Rob Fitzpatrick's Write Useful Books, which strongly recommends gathering feedback early and often. Fitzpatrick's approach included a custom website for readers to comment on in-progress books, called Help This Book:

In the Help This Book interface, readers highlight text and select the type of feedback they want to leave — clicking "Confusing" opens a field for their comment. But this interface didn't translate well to my work. Zines are images, not text, so a highlight-the-sentence approach wouldn't work. That's why in 2021 I asked Marie to help me build a custom site inspired by Help This Book but designed for zine pages.

The two hardest parts of the project were selecting the feedback categories and designing the site itself. Writing the code turned out to be the easy part.

Why categories matter

The feedback categories we chose do more than organize comments. They shape what readers write before they start typing, since selecting a category is a required first step. This structure has three main benefits:

  • It filters unwanted feedback. There's deliberately no "typo" category — catching typos is the copy editor's job, not the beta reader's.
  • It produces actionable responses. "I love this" requires no action; "This is confusing" tells us something needs clarifying.
  • It helps us group similar reactions. If several readers flag the same page as confusing, we can review that feedback as a cluster.

We developed these categories by examining feedback from previous zines and finding patterns. That analysis produced five categories.

"I learned something"

The whole point of the zines is to teach, so this is the best signal we can receive. Plenty of lightbulb reactions means the page is doing its job.

"I love this"

Readers often responded positively without claiming to have learned anything specific. I was initially against this category — learning is the goal, after all — but we added it anyway because so much feedback fit that pattern. I'm glad we did: hearts are encouraging and signal that a page should stay as-is.

"I have a question"

Questions reveal confusion about specific points. On early drafts of the "How DNS Works" zine, readers asked things like:

  • what is a "domain"?
  • why do we need to map a domain name to an IP address?
  • do authoritative nameservers push updates to resolvers or do resolvers "check" frequently to update their caches?
  • is the resolver built into the browser or is this a server the browser knows to go and query?

Not every question signals a problem — some reflect a reader who understood the material and wants to explore beyond the zine's scope. But most prompted improvements to the explanations themselves.

"I'm confused"

This category wasn't in our first version, but we kept receiving feedback that effectively meant "I'm confused" without a specific question attached. Sometimes an explanation is so unclear that the reader can't formulate what they don't understand. Examples from "How DNS Works" include:

  • I don't get the last section in the response record, the "glue records"
  • This section is over my head…
  • I didn't see where SOA records were defined. Did I miss something earlier or later?
  • I was confused here by the server word again, due to it being able to refer to a resolver / nameserver.

Readers often note "I was initially confused by X, but then I figured it out," which is also valuable feedback.

"I have a suggestion"

This final category is the catchall for anything that doesn't fit elsewhere — including occasional typo reports, even though we ask readers not to focus on those. We generally treat suggestions as lower priority than confusion or questions.

Listening to learners, not experts

The point of beta readers is feedback from people learning the material, not experts in it. We therefore prioritize "I'm confused" and "I have a question" feedback over suggestions, except when the suggestion comes from someone I know and trust. If learners aren't confused and are picking things up, the content is working. Technical review and copy editing happen later in the process and are outside the scope of this site.

The reader's view of the site

Here's what beta readers see when using the site:

And here are the categories a reader can select from:

Admin view: reading the emoji dashboard

Behind the scenes, the admin interface is where I check off comments once they're handled and add my own notes. The quickest read on how any page is doing comes from scanning the emoji reactions. Both pages shown below are from a very early draft, long since edited.

A page with promise

The comments are blurred, but the emojis tell the story: a few lightbulbs for "I learned something!" and hearts for "I love this!" Without reading a single comment, I know this page has potential — there may be improvements to make, but readers are engaging with it.

Scanning through the zine and seeing "this page has 10 hearts and 7 lightbulbs" is an incredibly efficient way to know what's working. That tells us what doesn't need attention, and it's motivating to see confirmation that pages are landing well.

A page that needs work

This page, by contrast, attracted only question marks and suggestion reactions — along with my own comments in purple, with reader comments in blue. Several notes were about a Rust panel that seemed arbitrary. I'd suspected as much already, but hearing it directly from multiple readers confirmed that fixing it needed to be a priority.

Confusion and suggestion reactions are just as useful as positive ones. When we see an abundance of them, we might rewrite or delete the page entirely and do another feedback round to see whether the problem persists. "I have a question" reactions aren't always negative — they can reflect a reader who understood enough to be curious about follow-up topics outside the zine's scope.

Pages with no feedback

A few pages typically get zero comments. When that happens, our options are to ask a trusted friend for their read, post the page publicly on Mastodon or Twitter, or simply trust our instincts about whether the content belongs.

Reader Feedback and Deployment Notes

At the end of a zine, readers can also submit overall feedback through a separate form in the admin interface. This section has proven especially useful for catching conceptual gaps: readers will sometimes note that they didn’t fully understand a particular topic, which signals that the explanation needs more work.

Under the Hood

Though implementing the code was described as the easiest part of the process, the technical setup is worth noting:

  • A Go backend compiled to a static binary
  • A SQLite database
  • A Vue.js frontend
  • A DigitalOcean server

Some operational details:

  • All zine pages are committed to the Git repository and compiled directly into the Go binary. Storing them in the database might be more flexible, but that change hasn’t been made yet.
  • Pages are password-protected; the password is emailed to each reader when feedback is requested.
  • Deployment involves rebuilding the Go binary locally with tinybuild, copying it to the DigitalOcean server via scp, and restarting the systemd process.

The tool is not open-source. It’s heavily customized to a specific workflow, lacks documentation, and intentionally keeps all zine pages in the Git repo. While there have been occasional thoughts about generalizing it for others, that’s unlikely to happen.

The Impact on Zine Quality

The feedback site was first built in summer 2021 and has since been used for three zines: How DNS Works, The Pocket Guide to Debugging, and an in-progress zine about binary data representation in memory.

The main benefit has been scale. Feedback can now come from dozens of readers rather than just three or four, which has made a tangible improvement in the overall quality of the zines.