Counting to three with a recurrent network
A day after failing to get an RNN to imitate Shakespeare's English, I stepped way back and tried something much more modest: teaching the same kind of network to output simple numeric sequences.
0 1 2 0 1 2 0 1 2 0 1 2
And slightly longer ones:
0 1 2 1 0 1 2 1 0 1 2 1 0 1 2 1 0
The architecture was, as far as I can tell, the same one that wouldn't cooperate on the Shakespeare task. Getting it to memorize short number runs was a useful sanity check that the model itself could learn anything at all.
Running the experiment
The training code is in PyTorch. Rather than walk through every line, here is what I used:
- The complete script is available as a GitHub gist.
- You can also run it directly in a Colab notebook.
The gist includes runs with a few different sequence lengths. Not surprisingly, memorizing a 20-element sequence takes noticeably longer to train than a 5-element one.
Why this feels like progress
Getting the RNN to do something I can fully inspect and understand is a good place to be. With this simpler case working, I'm fairly optimistic about going back to the character-level problem of generating English words.
