Diagnosing a Stalled RNN: BPTT, Gradient Norms, and Long Training Loops

Over the past few days I've been experimenting with back-propagation through time (BPTT) for training RNNs, and the experience has surfaced several debugging challenges that are worth unpacking.

When Loss Goes Up Instead of Plateauing

The most puzzling symptom appeared while training a model: the loss decreased for a while, then climbed dramatically.

This behavior is hard to explain if the optimizer were merely struggling—typically that would manifest as a plateau with some oscillation. A steady climb suggests something else is going wrong, but I don't yet have a clear explanation.

To check whether vanishing or exploding gradients were implicated, I plotted the gradient norm of one weight matrix over the same training period:

The gradient norm does shrink to quite small values toward the end, which looks consistent with a vanishing gradient problem.

BPTT Implementation: Periodic vs. Per-Character Updates

I've been comparing two training strategies. The first processes one character at a time, performing a gradient step after each:

for input, target in training_data:
    output, hidden = model(input, hidden)
    loss = F.cross_entropy(output, target)
    optimizer.zero_grad()
    loss.backward() # calculate the derivative
    optimizer.step() # adjust weights 
    hidden.detach()

The second approach feeds a longer sequence of characters and only runs backpropagation periodically:

for i, (input, target) in enumerate(training_data):
    output, hidden = model(input, hidden)
    # only do the optimizer step 10% of of the time
    if random.randint(0, 10) == 2:
        loss = F.cross_entropy(output, target)
        optimizer.zero_grad()
        loss.backward() # calculate the derivative
        optimizer.step() # adjust weights 
        hidden.detach()

This raises two immediate questions:

  • With BPTT over 40 steps, the model fails to train (showing the improve-then-degrade pattern above), while a 10-step window works fine. Why would the longer window cause this?
  • Training the same data one character at a time taught the model character names (e.g., CYMBELINE), but the BPTT-trained model doesn't seem to have picked those up. Whether this stems from the training data or the model configuration is still unclear.

Slowing Down the Iteration Loop

A significant practical bottleneck is the feedback cycle: each training run takes about 30 minutes before I can judge whether it worked. With many hypotheses to test, this is frustrating—especially when trying to track them all in a Jupyter notebook.

Gradient Clipping: One Line, Uncertain Effect

I recently added gradient clipping to the training function. The idea is that if the gradient norm exceeds a threshold (say, 1), the optimizer scales it down to a smaller norm. It's a single line of code, but whether it's actually helping is not yet clear.

torch.nn.utils.clip_grad_norm_(self.rnn.parameters(), 1)

Open Questions

Several questions remain on my list for the next round of experiments:

  • Is gradient clipping helping?
  • Why does the loss climb so much, and how is that even possible?
  • Why does BPTT fail with a sequence length of 40 but work with a length of 10?
  • Would randomizing the sequence length during BPTT help?
  • Does a gradient norm reaching 12 constitute an "exploding gradient"? (I suspect not.)
  • Is a gradient norm dropping to 0.05 a "vanishing gradient"? (I suspect yes.)
  • Does the seq_len parameter in PyTorch's LSTM class correspond to BPTT—i.e., do I need to implement BPTT manually, or will the LSTM handle it if data is formatted correctly?
  • How can I detect numerical stability issues?

Sample Output from the BPTT Model

The BPTT model produced the following text:

that than their conirot-ula thine not wate) For then in that my shill, And
Time, Wiss envage so love at thes: Time worture women of their cay bu thee
wisted all Tom werthen hear momed. An is perselfed? Bu mundeve teassed for my
sead wherey tood the ob'e, With eres, The ecref of heaven. 25 Levery I t

This looks quite similar to what the non-BPTT model generated:

at soerin, I kanth as jow gill fimes, To metes think our wink we in fatching
and, Drose, How the wit? our arpear War, our in wioken alous, To thigh dies wit
stain! navinge a sput pie, thick done a my wiscian. Hark's king, and Evit night
and find. Woman steed and oppet, I diplifire, and evole witk ud