Story Generation Using Dynamic Text Infilling
| Thesis title in Czech: | Generování příběhů pomocí dynamického rozšiřování textu |
|---|---|
| Thesis title in English: | Story Generation Using Dynamic Text Infilling |
| Key words: | generování příběhů|jazykové modely|neuronové sítě |
| English key words: | story generation|language models|neural networks |
| Academic year of topic announcement: | 2022/2023 |
| Thesis type: | Bachelor's thesis |
| Thesis language: | angličtina |
| Department: | Institute of Formal and Applied Linguistics (32-UFAL) |
| Supervisor: | Ing. Zdeněk Kasner, Ph.D. |
| Author: | Bc. Katarína Bucková - assigned and confirmed by the Study Dept. |
| Date of registration: | 01.11.2022 |
| Date of assignment: | 02.11.2022 |
| Confirmed by Study dept. on: | 23.11.2022 |
| Date and time of defence: | 07.09.2023 09:00 |
| Date of electronic submission: | 20.07.2023 |
| Date of submission of printed version: | 20.07.2023 |
| Date of proceeded defence: | 07.09.2023 |
| Opponents: | Mgr. Rudolf Rosa, Ph.D. |
| Guidelines |
| Currently, it is possible to generate long-form stories using Transformer-based pretrained neural language models such as GPT-2 (Radford et al., 2019). However, it is hard to guarantee the logical coherency of the story. Even if the story is generated hierarchically (Schmidtová, 2022), the model may diverge from the outline due to the nature of left-to-right autoregressive decoding, which provides only a weak supervision signal.
An alternative to left-to-right decoding is text infilling, i.e., generating a missing span of variable length given its left and right context (Donahue et al., 2020). As shown by Donahue et al. (2020), this technique can be used to infill masked parts of short stories from the ROCStories dataset (Mostafazadeh et al., 2016). The goal of the work is to examine the potential of text infilling for generating longer stories. The student will (1) finetune a pretrained neural language model for text infilling, (2) implement a system that will recursively apply the model to dynamically expand the story outline, and (3) evaluate the quality of the system outputs using both automatic metrics and human-based ratings. For stopping the recursion, the system should identify which parts of the story can be expanded while keeping the story coherent (Mori et al., 2020). |
| References |
| Donahue, C., Lee, M., & Liang, P. (2020, July). Enabling Language Models to Fill in the Blanks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 2492-2501).
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language Models are Unsupervised Multitask Learners,” Technical Report, OpenAI, Feb. 2019. https://openai.com/blog/better-language-models/ Mostafazadeh, N., Chambers, N., He, X., Parikh, D., Batra, D., Vanderwende, L., ... & Allen, J. (2016, June). A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of NAACL HLT 2016 (pp. 839-849). Mori, Y., Yamane, H., Mukuta, Y., & Harada, T. (2020). Finding and generating a missing part for story completion. In Proceedings of the The 4th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (pp. 156-166). Schmidtová, P. (2022). Theatre play generation. [Master Thesis, Charles University]. |