A First Week of RNN Training, in Lessons

After experimenting with pre-trained models for generating sketch-rnn faces, I wanted to train something myself. Rather than jumping straight to complex images, I started with a character-level RNN that generates vaguely Shakespearean text. My tooling choice landed on PyTorch with the fast.ai helper libraries, largely because a friend was already using them. What followed was a crash course in neural network training, mostly learned the hard way.

Tensors Are Everywhere, and They’re Not What I Expected

All data in this world is tensors. A 1-dimensional tensor is a vector, a 2-dimensional one is a matrix, and beyond that, things get abstract. Manipulating them is straightforward enough — tensor.flatten() turns a multidimensional tensor into a vector — though I’m still unsure about the exact ordering of elements during that flattening.

The terminology threw me off more than the math. I’d normally say an 8x9 matrix has a dimension of 72, since that’s the dimension of its vector space. Torch instead calls that matrix 2-dimensional. That took some getting used to.

Dimension Mismatches: The Real Obstacle Course

A large portion of my last few days was spent staring at errors like

ValueError: Expected target size (77, 64) got torch.Size (77, 70)

The most common causes:

  • Passing a 3-dimensional tensor to a loss function that expected 2 dimensions.
  • Forgetting to embed class-label inputs into a higher-dimensional space, so each number becomes a 64-dimensional vector.
  • Multiplying a vector by an incorrectly sized matrix.

What still confuses me is that many methods accept both 2- and 3-dimensional tensors interchangeably. A 2x3 matrix can be multiplied by a 3x4 tensor, or by a 3x4x89 tensor, or by something with far more dimensions. That matches the tensor notation I remember from quantum computing, but it’s much harder to reason about when you’re thinking in terms of concrete numbers in a grid.

PyTorch’s “Cross Entropy” Isn’t What the Name Says

A loss function measures how similar two vectors are, lower being better. So I was surprised to find the cross entropy loss of two identical vectors wasn’t zero. The catch: PyTorch’s cross_entropy(x, y) actually computes cross_entropy(softmax(x), y), applying a softmax to x before measuring the difference.

GPUs Make a Real Difference

The speedup is dramatic. When I switched training from CPU to a GPU, each batch ran literally 10 times faster.

Colab Works Well, With Caveats

Google’s Colab — basically a fork of Jupyter notebook — has been my environment of choice, mainly because it provides free GPU access. The main annoyance is that idle notebooks get killed fairly aggressively to save resources. That alone would be fine, since you can save state to Google Drive.

The real friction is authentication: every time I want to read files from Google Drive, I have to reauthenticate by clicking a link and pasting an OAuth code. It feels like a one-time authorization should stick forever.

When Training Loss Flatlines

Most of yesterday and today was spent debugging a model whose training loss never improved. The model wasn’t training at all. Predictions looked like this:

eto e  enaih eet codosueonites st tne   esee ob nmnoesnrertieieeu  ooe

That output is odd — all the letters are common in English, which suggests the model learned something. It’s better than gibberish like zxisqqqqxw, which is what I’d expect from an untrained network. But it certainly hasn’t learned much.

Code That Doesn’t Work (Yet)

For anyone curious what an RNN implementation in failure mode looks like, the code I’ve written so far shows it. The core neural network section is

class RNN(nn.Module):
    def __init__(self):
        super().__init__()
        self.i2h = nn.Linear(nv, nh) # Wxh
        self.h2h = nn.Linear(nh, nh) # Whh
        self.h2o = nn.Linear(nh, nv) # Why
        self.hidden = torch.zeros(1, nh).cuda()

    def forward(self, input):
        x = self.i2h(torch.nn.functional.one_hot(input, num_classes=nv).type(torch.FloatTensor).cuda())
        y = self.h2h(self.hidden)
        hidden = torch.tanh(y + x)
        self.hidden = hidden.detach()
        z = self.h2o(hidden)
        return z

which sets up a bunch of matrices mirroring Karpathy’s classic RNN example:

  def step(self, x):
    # update the hidden state
    self.h = np.tanh(np.dot(self.W_hh, self.h) + np.dot(self.W_xh, x))
    # compute the output vector
    y = np.dot(self.W_hy, self.h)
    return y

His code generates semi-coherent Shakespeare after training; mine produces nonsense. That’s the puzzle for tomorrow.

One useful snippet I do have is for sampling from the model’s output probability vector with a “temperature” parameter. At very low temperature, it always picks the most likely letter — which right now is literally always a space, itself a warning sign. Higher temperatures allow less likely options to win:

temperature = 1
prediction_vector = F.softmax(learn.model(x)[0]/temperature)
v.textify(torch.multinomial(prediction_vector, 1).flatten(), sep='')

Learning by Head-Banging

My learning style hasn’t changed: I avoid books and video courses (even though fast.ai has a whole course that looks great), preferring to hit a problem head-on until I’m thoroughly lost, then read up on the specific issue blocking me. So far it’s been fun — I now know far more about loss functions than I did a week ago, and I’ve finally learned what softmax actually does.