Text-based games present a unique challenge for autonomous agents to operate in natural language and handle enormous action spaces. In this paper, we propose the Contextual Action Language Model (CALM) to generate a compact set of action candidates at each game state. Our key insight is to train language models on human gameplay, where people demonstrate linguistic priors and a general game sense for promising actions conditioned on game history. We combine CALM with a reinforcement learning agent which re-ranks the generated action candidates to maximize ingame rewards. We evaluate our approach using the Jericho benchmark (Hausknecht et al., 2019a), on games unseen by CALM during training. Our method obtains a 69% relative improvement in average game score over the previous state-of-the-art model. Surprisingly, on half of these games, CALM is competitive with or better than other models that have access to ground truth admissible actions. * * Code and data are available at https://github. com/princeton-nlp/calm-textgame.Observation: You are in the living room. There is a doorway to the east, a wooden door with strange gothic lettering to the west, which appears to be nailed shut, a trophy case, and a large oriental rug in the center of the room. You are carrying: A brass lantern . . . Random Actions: close door, north a, eat troll with egg, . . . CALM (n-gram) Actions: enter room, leave room, lock room, open door, close door, knock on door, . . . CALM (GPT-2) Actions: east, open case, get rug, turn on lantern, move rug, unlock case with key, . . . Next Observation: With a great effort, the rug is moved to one side of the room, revealing the dusty cover of a closed trap door...Figure 1: Sample gameplay from ZORK1 along with action sets generated by two variants of CALM. The game recognizes a vocabulary size of 697, resulting in more than 697 4 ≈ 200 billion potential 4-word actions.'move rug' is the optimal action to take here and is generated by our method as a candidate.