About
I am an assistant professor of artificial intelligence (AI) in the Department of Medical Informatics in the Amsterdam UMC, University of Amsterdam, where I investigate methods based on machine learning (ML) and natural language processing (NLP) for problems in medicine and psychology.
More broadly, my research focuses on machine learning for natural languages in general, and in the complex interaction between language and vision, knowledge graphs, and commonsense knowledge.
Projects
2019–2023 Multi3Generation: Multi-task, Multilingual, Multimodal Language Generation. EU COST Action
2019–2022 IMAGINE: Integrating multimodal, world and commonsense knowledge into language generation. EU H2020 MSCA [finished]
News
Dec/2022 Nishant Mishra officially starts working on the project ‘Responsible ML and NLP for healthcare’, co-supervised with Ameen Abu-Hanna.
Nov/2022 I have been offered a tenured position at the Amsterdam UMC!
Sep/2022 Zeyu Zhang officially starts working on the project ‘NLP and data management for mental health research’, co-supervised with Sebastian Schelter.
May/2022 I am hiring a PhD student for the project Natural Language Processing and Machine Learning for Healthcare. Applications closed!
Mar/2022 I am hiring a PhD student for the project Responsible Natural Language Processing and Data Management for Mental Health. Applications closed!
Mar/2022 Our paper VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomena was accepted for publication at ACL 2022. We investigate to what extent existing pretrained vision-and-language models ground text on vision and vice-versa using a counterfactual approach and focusing on fine-grained linguistic phenomena! (code and data)
Feb/2022 I am tenure-track Assistant Professor of AI in Medicine in the Department of Medical Informatics, Amsterdam UMC, University of Amsterdam!
Nov/2021 Our paper VisualSem: A High-quality Knowledge Graph for Vision & Language was accepted for publication at the ‘Multilingual Representation Learning’ (MRL 2021) Workshop. We created a dataset to support research in multilingual models for vision and language! (data)
Sep/2021 Collaborative article Neural Natural Language Generation: A Survey on Multilinguality, Multimodality, Controllability and Learning accepted for publication in the Journal of Artificial Intelligence Research (JAIR) where we discuss the recent trends in neural approaches to natural language generation.
May/2021 Book chapter Natural language processing for mental disorders: an overview accepted for publication in: Natural Language Processing in Healthcare: A Special Focus on Low Resource Languages (CRC Press).
May/2021 Our paper Seeing past words: Testing the cross-modal capabilities of pretrained V&L models on counting tasks was accepted for publication at the ‘Beyond Language: Multimodal Semantic Representations’ (MMSR 2021) Workshop. We release a counting probe dataset to be used in evaluating pretrained V&L models (data).
Mar/2021 Paper accepted at NAACL 2021: Wikipedia Entities as Rendezvous across Languages: Grounding Multilingual Language Models by Predicting Wikipedia Hyperlinks. (code and data)
Feb/2021 I am guest lecturer in the Erasmus Mundus European Masters on Technology for Translation and Interpreting. I give lectures on Machine Translation and Machine Learning in the University of Wolverhampton.
Jan/2021 We published a pre-print on probing the counting capabilities of pretrained vision & language models: Seeing past words: Testing the cross-modal capabilities of pretrained V&L models?
Sep/2020 We release the beta-version of the VisualSem knowledge graph, a multimodal and multilingual resource to support vision & language research: VisualSem: a high-quality knowledge graph for vision and language (code)
Sep/2020 Two papers accepted at AACL 2020: Are scene graphs good enough to improve Image Captioning? and English intermediate-task training improves zero-shot cross-lingual transfer too.
Aug/2020 I am co-organizer of the Repl4NLP 2021 (Representation Learning for NLP 2021) workshop.
Jul/2020 I am Area Chair for Language Grounding to Vision at EACL 2021.
Aug/2019 I am Short-Term Scientific Mission Coordinator and member of the Management Committee of the Multi3Generation COST Action.
Jun/2019 I have joined Kyunghyun Cho’s group in New York University.
May/2019 Paper accepted at ACL 2019 where we propose a latent variable model for multi-modal machine translation (pre-print available here and code coming soon).
Mar/2019 I have joined Raquel Fernández’s group in the University of Amsterdam.
Feb/2019 I have received a Marie Skłodowska-Curie Global Fellowship! I will be visiting the New York University Centre for Data Science, Facebook Artificial Intelligence Research in Paris, just to return to the University of Amsterdam! My project is called IMAGINE: Improving lAnguage Generation wIth world kNowledgE and more information is available here.
Jun/2018 I gave two lectures at the Summer School in Translation Technologies organised by the Faculty of Arts in Universidade do Porto. The lectures were on statistical and neural machine translation.
Mar/2018 I have joined Khalil Sima’an’s group in the University of Amsterdam!
Oct/2017 Best Long Paper Award at INLG 2017 in a collaboration with Thiago Castro Ferreira, Sander Wubben and Emiel Krahmer with paper Linguistic realisation as machine translation: Comparing different MT models for AMR-to-text generation.
Sep/2017 Paper accepted at EMNLP 2017 where I study different strategies to incorporate global visual features into neural machine translation. (code and paper).
Sep/2017 Paper accepted at RANLP 2017 where I propose a ranking model to learn multilingual and multi-modal sentence embeddings. (code and paper).
Jul/2017 Paper accepted at ACL 2017 where I propose a doubly-attentive decoder for multi-modal machine translation! (code and paper).
Apr/2017 Collaboration with eBay Inc. lead to two papers! One paper accepted at EACL 2017 and another paper at the V&L workshop 2017!
