Predicting Word Importance Using Pre-Trained Language Models
| Název práce v češtině: | Predikcia dôležitosti slov pomocou predtrénovaných jazykových modelov |
|---|---|
| Název v anglickém jazyce: | Predicting Word Importance Using Pre-Trained Language Models |
| Klíčová slova: | dôležitosť slov|jazykové modelovanie |
| Klíčová slova anglicky: | word importance|language modeling |
| Akademický rok vypsání: | 2023/2024 |
| Typ práce: | bakalářská práce |
| Jazyk práce: | angličtina |
| Ústav: | Ústav formální a aplikované lingvistiky (32-UFAL) |
| Vedoucí / školitel: | Mgr. Dávid Javorský |
| Řešitel: | skrytý - zadáno a potvrzeno stud. odd. |
| Datum přihlášení: | 30.10.2023 |
| Datum zadání: | 03.11.2023 |
| Datum potvrzení stud. oddělením: | 03.11.2023 |
| Datum a čas obhajoby: | 05.09.2024 09:00 |
| Datum odevzdání elektronické podoby: | 17.07.2024 |
| Datum odevzdání tištěné podoby: | 17.07.2024 |
| Datum proběhlé obhajoby: | 05.09.2024 |
| Oponenti: | Mgr. Dominik Macháček, Ph.D. |
| Konzultanti: | doc. RNDr. Ondřej Bojar, Ph.D. |
| Zásady pro vypracování |
| Words are the smallest discrete units of a language that have a particular meaning, and their contribution in decision-making processes of neural models, or in human brains, is undoubtedly unequal.
The goal of this thesis is therefore to examine a small set of possible definitions of word importance (with a focus on semantic importance), and to train a neural model capable of assigning these importance scores to each input word. This will be accomplished by leveraging a masked language modeling approach (i.e. finetuning pre-trained language models) and repurposing its paradigm: Instead of predicting which words are missing, our objective will be to predict what words are inserted. As part of the evaluation, the goal of the thesis is to implement an annotation tool for collecting gold labels for importance scores (linked to our suggested definitions of importance) and to compare them to the acquired scores using our proposed method. A valuable extension of this work would be to evaluate the importance scores on a downstream task, e.g. keyword identification. |
| Seznam odborné literatury |
| Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dávid Javorský, Ondřej Bojar, and François Yvon. 2023. Assessing Word Importance Using Models Trained for Semantic Tasks. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8846–8856, Toronto, Canada. Association for Computational Linguistics. Sushant Kafle and Matt Huenerfauth. 2018. A Corpus for Modeling Word Importance in Spoken Dialogue Transcripts. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). Martinc, Matej, Blaž Škrlj, and Senja Pollak. "TNT-KID: Transformer-based neural tagger for keyword identification." Natural Language Engineering 28.4 (2022): 409-448. |