Transformer encoders incorporating word translation for russian-vietnamese machine translation

Thien Nguyen, Trang Nguyen, Huu Nguyen, Phuoc Tran

Research output: Contribution to journalArticlepeer-review


Neural machine translation systems including the latest Transformer models represent translation units in the form of embeddings – vectors of real numbers. Such continuous representations of translation units lead to smoother translation results, but do not always guarantee better results due to wrong word translations, compared to statistical machine translation systems. Moreover, for low-resource language pairs, such as Russian-Vietnamese, the errors of word translations in neural machine translation systems are more aggravated. In order to solve the problem, we try different ways of con-catenating source word embeddings with embeddings of their corresponding word transla-tions, when building a Transformer-based translation system for the Russian-Vietnamese language pair. As a result, we create two novel Transformer models: Transformer with Long Encoder and Transformer with Short Encoder. In the Transformer with Long Encoder source word embedding and translation embedding of single size are concatenated to form a vector of double size. The Long Encoder reduces the size of the concatenated embedding to single size with a linear layer, and then adds it with positional embedding of the source word to create a final embedding. The Short Encoder resembles the Long Encoder except for the linear layer. Instead, the Short Encoder creates word embedding and translation embedding of half-size, and then concatenates them to form a concatenated embedding of single size. The experimental results show that the proposed models provide better translation quality compared to the baseline Transformer model.

Original languageEnglish
Pages (from-to)35-42
Number of pages8
JournalICIC Express Letters, Part B: Applications
Issue number1
Publication statusPublished - Jan 2021
Externally publishedYes


  • Neural machine transla-tion
  • Neural networks
  • Russian-Vietnamese machine translation
  • Transformer
  • Word translation



State classification of scientific and technological information


Cite this