IMAGINE: Improving Multi-modal lAnguage Generation wIth world kNowledgE is a research project which main goal is to investigate how to incorporate world knowledge into vision & language tasks within natural language generation. I am a Marie Skwodówska-Curie Global Fellow.
I spent ~2 years in New York University’s Courant Institute for Mathematical Sciences where I worked with Kyunghyun Cho. I am physically in Amsterdam, and the project is implemented at the Institute for Logic, Language and Computation (ILLC) in the University of Amsterdam, where I work with Raquel Fernández.
Concretely, I investigate how to:
- Gather world-knowledge (semi-)automatically from publicly available multi-modal knowledge bases.
- Learn representations for a knowledge base that encompasses both text and images.
- Integrate this knowledge into multi-modal language generation tasks, such as multi-modal machine translation, visual question answering and image description generation.
Please get in touch with you would like to collaborate on any of these research topics!