Hidden State vs. Output in Sequence Autoencoders
A question that came up while working through the translation tutorial was whether the Encoder class's encoding should be its output vector or its hidden vector. The tutorial's encoder produces both, so the ambiguity is understandable.
The answer, found via the PyTorch forums while searching for autoencoder implementations, is straightforward: the hidden vector is the encoding. The output vector can be discarded.
Practical Implication
With that clarified, a working autoencoder becomes much easier to assemble. The encoder's final hidden state carries the compressed representation of the input sequence, and that's what should be passed to the decoder. The per-timestep output vectors from the encoder are not needed for the reconstruction task.
The relu activation's precise role in the encoder remains an open question, but the hidden-state clarification removes the main blocker. Sometimes the most valuable step is simply articulating the confusion clearly enough to search for it.
