Transformer Language Model Tutorial - Incorrect Attention Mask Description

[zoeqevans]

The tutorial Language Modeling With nn.Transformer and Torchtext describes an attention mask that will prevent nn.TransformerEncoder from attending to not-yet-seen tokens:

Along with the input sequence, a square attention mask is required because the self-attention layers in nn.TransformerEncoder are only allowed to attend the earlier positions in the sequence.

I think the mask is actually upper triangular: a square mask would not prevent a token attending to future token.

I may have misunderstood the mask description; if not, happy to write the PR that fixes this.

cc @pytorch/team-text-core @Nayef211

[hudeven]

@montyevans Make sense! Please feel free to send a PR.

[keith-golden]

@montyevans @hudeven To add to this, the transformer (at least as it's implemented in "Attention is all you need") does not mask self-attention in the encoder, rather, the upper triangular mask is for the self-attention blocks in the decoder.

We want the encoder to see the whole sequence, but the decoder should only be able to see the tokens it has predicted up to that point, hence the mask. The tutorial applies the mask to the encoder instead. Hope this helps.

[bjhargrave]

/assigntome