Enriching Neural MT through Multi-Task Training

• Název práce v češtině: Obohacování neuronového strojového překladu technikou sdíleného trénování na více úlohách
• Název v anglickém jazyce: Enriching Neural MT through Multi-Task Training
• Klíčová slova: multi-task neuronový strojový překlad NMT Transformer němčina
• Klíčová slova anglicky: multi-task neural machine translation NMT Transformer German
• Akademický rok vypsání: 2017/2018
• Typ práce: diplomová práce
• Jazyk práce: angličtina
• Ústav: Ústav formální a aplikované lingvistiky (32-UFAL)
• Vedoucí / školitel: doc. RNDr. Ondřej Bojar, Ph.D.
• Řešitel: Mgr. Dominik Macháček - zadáno a potvrzeno stud. odd.
• Datum přihlášení: 18.04.2018
• Datum zadání: 24.04.2018
• Datum potvrzení stud. oddělením: 09.05.2018
• Datum a čas obhajoby: 11.09.2018 09:00
• Datum odevzdání elektronické podoby: 18.07.2018
• Datum odevzdání tištěné podoby: 20.07.2018
• Datum proběhlé obhajoby: 11.09.2018
• Oponenti: Mgr. Jindřich Helcl, Ph.D.

Zásady pro vypracování

Multi-task training has shown promising results in various areas of deep learning application, including neural machine translation (NMT). NMT has also already benefited from enriched information in either the source or the target language (e.g. part-of-speech tags, syntactic tags from CCG and others) for the sequence-to-sequence model with attention.

The goal of the thesis is to experiment with the Transformer architecture of NMT and to employ various types of additional information of the source or target side in a multi-task setup, aiming at a better translation quality. Specifically, the work will carry out a reasonably large set of experiments where the NMT system is trained to perform not only the translation but also one or more additional tasks for the source or target sentence. For example, the system can be trained to translate and label the source for part-of-speech tags, or translate and identify named entities in the source. The motivation for including these additional tasks is to improve the generalization capacity of the NMT model: if the model is capable of determining named entities, it should be able to spot them during translation as well and translate them accordingly.

The thesis will explore primarily a "simplified" multi-task setup which does not need any changes in the network architecture. The multiple tasks are reflected only in the training data, e.g. by interleaving tokens or alternating segments for the tasks considered. More advanced techniques of multi-task training are also possible.

The experiments will be carried out on the German-Czech translation task and possibly other language pairs, too. The translation direction will be chosen depending on the available automatic annotation for each of the languages. The outputs will be evaluated primarily with automatic methods. For the final runs, a small manual evaluation is also desirable.

Seznam odborné literatury

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6000–6010. Curran Associates, Inc., 2017.

Rico Sennrich and Barry Haddow. Linguistic Input Features Improve Neural Machine Translation. In Proceedings of the First Conference on Machine Translation, pages 83–91, Berlin, Germany, August 2016. Association for Computational Linguistics.

Jan Niehues and Eunah Cho. Exploiting linguistic resources for neural machine translation using multi-task learning. In Proceedings of the Second Conference on Machine Translation, pages 80–89. Association for Computational Linguistics, 2017.

Orhan Firat, Kyunghyun Cho, and Yoshua Bengio. Multi-way, multilingual neural machine translation with a shared attention mechanism. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 866–875, San Diego, California, June 2016. Association for Computational Linguistics.