publications

2024

1. Scaling Laws for Imitation Learning in Single-Agent Games

   Jens Tuyls, Dhruv Madeka, Kari Torkkola, and 3 more authors

   In Transactions on Machine Learning Research, 2024

   Abs arXiv Code

   Imitation Learning (IL) is one of the most widely used methods in machine learning. Yet, many works find it is often unable to fully recover the underlying expert behavior (Wen et al., 2020; Jacob et al., 2022), even in constrained environments like single-agent games (De Haan et al., 2019; Hambro et al., 2022b). However, none of these works deeply investigate the role of scaling up the model and data size. Inspired by recent work in Natural Language Processing (NLP) (Kaplan et al., 2020; Hoffmann et al., 2022) where “scaling up” has resulted in increasingly more capable LLMs, we investigate whether carefully scaling up model and data size can bring simi- lar improvements in the imitation learning setting for single-agent games. We first demonstrate our findings on a variety of Atari games, and there- after focus on the extremely challenging game of NetHack. In all games, we find that IL loss and mean return scale smoothly with the compute bud- get (FLOPs) and are strongly correlated, resulting in power laws for training compute-optimal IL agents. Finally, we forecast and train several NetHack agents with IL and find they outperform prior state-of-the-art by 1.5x in all settings. Our work both demonstrates the scaling behavior of imitation learning in a variety of single-agent games, as well as the viability of scaling up cur- rent approaches for increasingly capable agents in NetHack, a game that remains elusively hard for current AI systems.

2. Language-guided World Models: A Model-based Approach to AI Control

   Alex L Zhang*, Khanh Xuan Nguyen*, Jens Tuyls, and 2 more authors

   In ACL SpLU-RoboNLP Workshop, 2024

   Oral Presentation

   Abs arXiv Code

   This paper introduces the concept of Language- Guided World Models (LWMs)—probabilistic models that can simulate environments by reading texts. Agents equipped with these models provide humans with more extensive and efficient control, allowing them to simultaneously alter agent behaviors in multiple tasks via natural verbal communication. In this work, we take initial steps in developing robust LWMs that can generalize to compositionally novel language descriptions. We design a challenging world modeling benchmark based on the game of MESSENGER (Hanjie et al., 2021), featuring evaluation settings that require varying degrees of compositional generalization. Our exper- iments reveal the lack of generalizability of the state-of-the-art Transformer model, as it offers marginal improvements in simulation qual- ity over a no-text baseline. We devise a more robust model by fusing the Transformer with the EMMA attention mechanism (Hanjie et al., 2021). Our model substantially outperforms the Transformer and approaches the performance of a model with an oracle semantic parsing and grounding capability. To demonstrate the practicality of this model in improving AI safety and transparency, we simulate a scenario in which the model enables an agent to present plans to a human before execution, and to re- vise plans based on their language feedback.

3. Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learning

   Chongyi Zheng*, Jens Tuyls*, Joanne Peng, and 1 more author

   In Intrinsically-Motivated and Open-Ended Learning Workshop @NeurIPS2024, 2024

   Abs arXiv Code

   Self-supervised learning has the potential of lifting several of the key challenges in reinforcement learning today, such as exploration, representation learning, and reward design. Recent work (METRA) has effectively argued that moving away from mutual information and instead optimizing a certain Wasserstein distance is important for good performance. In this paper, we argue that the benefits seen in that paper can largely be explained within the existing framework of mutual information skill learning (MISL). Our analysis suggests a new MISL method (contrastive successor features) that retains the excellent performance of METRA with fewer moving parts, and highlights connections between skill learning, contrastive representation learning, and successor features. Finally, through careful ablation studies, we provide further insight into some of the key ingredients for both our method and METRA.

2022

1. Multi-Stage Episodic Control for Strategic Exploration in Text Games

   Jens Tuyls, Shunyu Yao, Sham Kakade, and 1 more author

   In International Conference on Learning Representations, 2022

   Spotlight

   Abs arXiv Code

   Text adventure games present unique challenges to reinforcement learning methods due to their combinatorially large action spaces and sparse rewards. The interplay of these two factors is particularly demanding because large action spaces require extensive exploration, while sparse rewards provide limited feedback. This work proposes to tackle the explore-vs-exploit dilemma using a multi-stage approach that explicitly disentangles these two strategies within each episode. Our algorithm, called eXploit-Then-eXplore (XTX), begins each episode using an exploitation policy that imitates a set of promising trajectories from the past, and then switches over to an exploration policy aimed at discovering novel actions that lead to unseen state spaces. This policy decomposition allows us to combine global de- cisions about which parts of the game space to return to with curiosity-based local exploration in that space, motivated by how a human may approach these games. Our method significantly outperforms prior approaches by 24% and 10% average normalized score over 12 games from the Jericho benchmark (Hausknecht et al., 2020) in both deterministic and stochastic settings, respectively. On the game of Zork1, in particular, XTX obtains a score of 103, more than a 2x improvement over prior methods, and pushes past several known bottlenecks in the game that have plagued previous state-of-the-art methods.

2020

1. Gradient-based Analysis of NLP Models is Manipulable

   Junlin Wang*, Jens Tuyls*, Eric Wallace, and 1 more author

   In Findings of Empirical Methods in Natural Language Processing, 2020

   Abs arXiv Website

   Gradient-based analysis methods, such as saliency map visualizations and adversarial input perturbations, have found widespread use in interpreting neural NLP models due to their simplicity, flexibility, and most importantly, their faithfulness. In this paper, however, we demonstrate that the gradients of a model are easily manipulable, and thus bring into question the reliability of gradient-based analyses. In particular, we merge the layers of a target model with a Facade that overwhelms the gradients without affecting the predictions. This Facade can be trained to have gradients that are misleading and irrelevant to the task, such as focusing only on the stop words in the input. On a variety of NLP tasks (text classification, NLI, and QA), we show that our method can manipulate numerous gradient-based analysis techniques: saliency maps, input reduction, and adversarial perturbations all identify unimportant or targeted tokens as being highly important.

2. Generative Modeling for Atmospheric Convection

   G. Mooers, Jens Tuyls, S. Mandt, and 2 more authors

   In International Conference on Climate Informatics, 2020

   Abs arXiv

   To improve climate modeling, we need a better understanding of multi-scale atmospheric dynamics – the relationship between large scale environment and small-scale storm formation, morphology and propagation – as well as superior stochastic parameterization of convective organization. We analyze raw output from ∼ 6 million instances of explicitly simulated convection spanning all global geographic regimes of convection in the tropics, focusing on the vertical velocities extracted every 15 minutes from ∼ 4 hundred thousands separate instances of a storm-permitting moist turbulence model embedded within a multi-scale global model of the atmosphere. Generative modeling techniques applied on high-resolution climate data for representation learning hold the potential to drive next-generation parameterization and breakthroughs in understanding of convection and storm development. To that end, we design and implement a specialized Variational Autoencoder (VAE) to perform structural replication, dimensionality reduction and clustering on these cloud-resolving vertical velocity outputs. Our VAE reproduces the structure of disparate classes of convection, successfully capturing both their magnitude and variances. This VAE thus provides a novel way to perform unsupervised grouping of convective organization in multi-scale simulations of the atmosphere in a physically sensible manner. The success of our VAE in structural emulation, learning physical meaning in convective transitions and anomalous vertical velocity field detection may help set the stage for developing generative models for stochastic parameterization that might one day replace explicit convection calculations.

3. Differentially Private Language Models Benefit from Public Pre-training

   Gavin Kerrigan, Dylan Slack, and Jens Tuyls

   In EMNLP PrivateNLP Workshop, 2020

   Abs arXiv

   Language modeling is a keystone task in natu- ral language processing. When training a lan- guage model on sensitive information, differ- ential privacy (DP) allows us to quantify the degree to which our private data is protected. However, training algorithms which enforce differential privacy often lead to degradation in model quality. We study the feasibility of learning a language model which is simultane- ously high-quality and privacy preserving by tuning a public base model on a private cor- pus. We find that DP fine-tuning boosts the performance of language models in the private domain, making the training of such models possible.

2019

1. AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models

   Eric Wallace, Jens Tuyls, Junlin Wang, and 3 more authors

   In Empirical Methods in Natural Language Processing, 2019

   Best Demo Award

   Abs arXiv Poster Website

   Neural NLP models are increasingly accurate but are imperfect and opaque — they break in counterintuitive ways and leave end users puzzled at their behavior. Model interpretation methods ameliorate this opacity by providing explanations for specific model predictions. Unfortunately, existing interpretation codebases make it difficult to apply these methods to new models and tasks, which hinders adoption for practitioners and burdens interpretability researchers. We introduce AllenNLP Interpret, a flexible framework for interpreting NLP models. The toolkit provides interpretation primitives (e.g., input gradients) for any AllenNLP model and task, a suite of built-in interpretation methods, and a library of front-end visualization components. We demonstrate the toolkit’s flexibility and utility by implementing live demos for five interpretation methods (e.g., saliency maps and adversarial attacks) on a variety of models and tasks (e.g., masked language modeling using BERT and reading comprehension us- ing BiDAF). These demos, alongside our code and tutorials, are available at https://allennlp.org/interpret.