Generating sketch-rnn faces, then sorting the interesting from the boring

For generative art week at the Recurse Center, I wanted to generate line drawings of faces. The Google Quickdraw dataset has plenty of face sketches, and while many are forgettable, some have real character. Two examples that stood out to me:

That looked like a reasonable starting point.

Getting sketch-rnn running in the browser

sketch-rnn is an RNN trained on the Quickdraw dataset that produces line drawings. The magic sketchpad demo shows it well: you start a rough drawing, and the model completes it in the style of whatever category you pick—cat, bridge, and so on.

The ml5js library has a tutorial with example code for using sketch-rnn via JavaScript, which I followed initially.

Refactoring the example to async/await

The ml5js example relied on global variables and callbacks, which made it awkward to modify. I spent a bit of time converting it to async/await so the flow was easier to control and experiment with. It worked, though the refactor took longer than expected.

My first idea was an interactive site, but I intentionally went simpler: set the model loose to draw lots of faces and observe the output before doing anything more elaborate.

The result is a rough, non-responsive page that renders faces one by one and collects them in a right-hand gallery: lots of sketch-rnn faces. Watching it draw live is, at least, entertaining.

The set of generated images looks something like this:

Honestly, I do not like any of the faces it produced, but it is a start. The page is sluggish on my 2015 laptop, though better on the iPad. I did not profile thoroughly, but a good chunk of time seems to go into a function with lstm in its name. I don’t know what an LSTM is exactly beyond being part of an RNN, but the amount of JavaScript-side math to compute the next stroke explains the slowness.

Teaching the model taste

For example, this face:

strikes me as considerably more interesting than this one:

Would it be possible to get the neural network to tell “interesting” faces from “boring” ones, and focus generation on the former? That is the current goal. Right now I’m stuck trying to load a pre-trained model into Python, so there’s plenty of work left.

Related prior art exists: someone did a similar exercise separating bad flamingos from good flamingos in the Quickdraw corpus.

I still know close to nothing about RNNs. Maybe answering this question will change that.

Meanwhile: a refrigerator poetry forum

On a side note, here is the refrigerator poetry forum mentioned earlier: refrigerator-poetry-forum.herokuapp.com. It allows composing magnet-poetry-style poems on a fridge, and visitors have posted some charming ones. Building it has made me more willing to reach for Rails in the future. If abuse appears I’ll take it offline, but so far so good.