Transformer tutorial multiplying with sqrt(d_model)

[RogerJL]

tutorials/beginner_source/translation_transformer.py

Line 135 in 5e772fa

return self.embedding(tokens.long()) * math.sqrt(self.emb_size)

src = self.embedding(src) * math.sqrt(self.d_model)

shouln't this be

src = self.embedding(src) / math.sqrt(self.d_model)

at least that is the impression I got when reading the "Attention is all you need" paper.
Or is there some new research finding that multiplying is better?

cc @sekyondaMeta @svekars @kit1980 @subramen @albanD

[IvanLauLinTiong]

/assigntome

[kit1980]

Looking online, there are some discussions and looks like * is actually correct.
https://stackoverflow.com/questions/56930821/why-does-embedding-vector-multiplied-by-a-constant-in-transformer-model

I'm closing this, if someone more knowledgeable wants to correct me - please re-open.

[RogerJL]

Noted that this scaling was in a different place than the scaling in "Attention is all you need"
When rereading the paper, I noticed this hidden in 3.4 Embeddings and Softmax text

"In the embedding layers, we multiply those weights by √dmodel."

Closing is correct!