Port new seq2seq tutorial

[chsasank]

From https://github.com/spro/practical-pytorch/tree/master/seq2seq-translation

cc @pytorch/team-text-core @Nayef211

[poweihuang17]

BTW, what's the attention model implemented in official tutorial? The Bahdanau et al. model is said to be implemented, but it's actually not that one. The code used input embedding and hidden state to calculate the attention weight, but the Bahdanau et al. model should not be like this.

[wzpfish]

BTW, what's the attention model implemented in official tutorial? The Bahdanau et al. model is said to be implemented, but it's actually not that one. The code used input embedding and hidden state to calculate the attention weight, but the Bahdanau et al. model should not be like this.

I think it's a bug. Never seen the attention of this form before...

[bbruceyuan]

@wzpfish In my opinion, it's a wrong implementation of using MLP to compute attention scores. All of the encoder hidden states should be used.

[wzpfish]

@hey-bruce Look at the tutorial at https://pytorch.org/tutorials/intermediate/seq2seq_translation_tutorial.html

attn_weights = F.softmax(self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)

This attention weights do not use any of the encoder hidden states. I think there is something wrong.

[rajarsheem]

@wzpfish Exactly, I raised an issue in their discuss forum. https://discuss.pytorch.org/t/possible-bug-in-seq2seq-tutorial/48241

Shortly afterwards, I saw your comment when I was searching for similar issue here if someone have already raised.

[karkirowle]

I was just studying this, and I came to the same realisation. The annotations should be used instead of the embedded output words, which would be the encoder outputs. It's not a one-line change, I think.
Also, Bahdanau uses bidirectional recurrent neural networks.

[karkirowle]

I can make a pull request stating something like "this is being reworked", but the tutorial is not appropriate in this form, and this really needs to be changed as soon as possible. You can even see that the learnt attention is wrong. I can start working on making an updated version of this tutorial eventually if needed.

[QasimKhan5x]

/assigntome