SubjectsSubjects(version: 861)
Course, academic year 2019/2020
  
Compendium of the Neural Machine Translation - NPFL116
Title: Kompendium neuronového strojového překladu
Guaranteed by: Institute of Formal and Applied Linguistics (32-UFAL)
Faculty: Faculty of Mathematics and Physics
Actual: from 2017
Semester: summer
E-Credits: 3
Hours per week, examination: summer s.:0/2 C [hours/week]
Capacity: unlimited
Min. number of students: unlimited
State of the course: taught
Language: Czech, English
Teaching methods: full-time
Guarantor: Mgr. Jindřich Libovický, Ph.D.
Mgr. Jindřich Helcl
Annotation -
Last update: T_UFAL (09.01.2017)
Neural machine translation recently became a new interesting and successful paradigm. The new paradigm brings new theoretical concepts and new ways of seeing the classic problems of machine translation. The goal of this seminar is to familiarize the students with the theoretical framework of neural machine translation in such depth that would allow them to study the most recent academic papers on this topic.
Aim of the course - Czech
Last update: T_UFAL (09.01.2017)

Na prvních šesti seminářích se studenti formou přednášky a diskuse seznámí s následujícími tématy:

Rekurentní neuronové sítě a s nimi spojená matematická intuice

Sequence-to-sequence learning

Attention model a jeho varianty

Řešení problému omezeného slovníku

Pokročilé metody trénování (minimum risk training, zpětnovazební učení)

Ve zbývající části semestru budou formou studentských referátů prezentovány předem vybrané články z odborných konferencí a časopisů. V následující moderované diskuzi se studenti pokusí navrhnout, jak by bylo možné obsah probíraného článku naprogramovat.

Zápočet bude možné získat za:

Včasné písemné zodpovězení otázek k prezentovaným článkům a

Přednesení referátu k vybranému článku nebo fungující implementaci některého ze článků

Course completion requirements -
Last update: doc. RNDr. Vladislav Kuboň, Ph.D. (05.06.2018)

Students pass the practicals by submitting sufficient number of assignments. The assignments are announced regularly the whole semester and are due in several weeks. Considering the rules for completing the practicals, it is not possible to retry passing it. Passing the practicals is not a requirement for going to the exam.

The exam consists of a written part and an optional oral part, where the students can react to queries regarding the written part and also answers additional questions.

The requirements of the exam correspond to the course syllabus, in the level of detail which was presented on the lectures.

Literature -
Last update: T_UFAL (09.01.2017)

Goodfellow, Ian, Yoshua Bengio, Aaron Courville. Deep Learning. MIT Press, 2016. (chapters 10-12).

LeCun, Yann, Yoshua Bengio, and Geoffrey Hinton. "Deep learning." Nature 521.7553 (2015): 436-444.

Sutskever, Ilya, Oriol Vinyals, and Quoc V. Le. "Sequence to sequence learning with neural networks." Advances in neural information processing systems. 2014.

Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate." arXiv preprint arXiv:1409.0473 (2014).

Sennrich, Rico, Barry Haddow, and Alexandra Birch. "Neural machine translation of rare words with subword units." arXiv preprint arXiv:1508.07909 (2015).

Shen, Shiqi, et al. "Minimum risk training for neural machine translation." arXiv preprint arXiv:1512.02433 (2015).

Wu, Yonghui, et al. "Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation." arXiv preprint arXiv:1609.08144 (2016).

Johnson, Melvin, et al. "Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation." arXiv preprint arXiv:1611.04558 (2016).

Syllabus -
Last update: Mgr. Jindřich Helcl (13.05.2019)

Feed-forward neural networks

  • Basic architectures and activation functions
  • Optimization algorithms for training deep models

Regularization of deep learning models

  • Classical regularization based on penalization by parameter norm.
  • Dropout
  • Batch Normalization
  • Multi-task learning

Convolutional neural networks

  • Convolution and pooling layers
  • Very deep convolutional network architectures
  • State-of-the-art models for image recognition, object detection and image segmentation

Recurrent neural networks

  • Recurrent neural networks and their training
  • Long short-term memory
  • Gated recurrent units
  • Bidirectional and deep recurrent neural networks
  • Encoder-decoder architectures

Practical methodology

  • Selection of a suitable architecture
  • Selection of hyperparameters

Natural language processing

  • Distributed word representation
  • Representation of words as sequences of characters
  • State-of-the-art algorithms for morphological tagging, named-entity recognition, machine translation, image captioning

Deep generative models

  • Variational autoencoders
  • Generative adversarial networks

Structured prediction

  • CRF layer
  • CTC loss and its applications in state-of-the-art speech recognition algorithms.

Introduction to reinforcement learning

 
Charles University | Information system of Charles University | http://www.cuni.cz/UKEN-329.html