Pragmatic Image Compression for Human-in-the-Loop Decision-Making

Abstract

Standard lossy image compression algorithms aim to preserve an image's appearance, while minimizing the number of bits needed to transmit it. However, the amount of information actually needed by a user for downstream tasks -e.g., deciding which product to click on in a shopping website -is likely much lower. To achieve this lower bitrate, we would ideally only transmit the visual features that drive user behavior, while discarding details irrelevant to the user's decisions. We approach this problem by training a compression model through human-in-the-loop learning as the user performs tasks with the compressed images. The key insight is to train the model to produce a compressed image that induces the user to take the same action that they would have taken had they seen the original image. To approximate the loss function for this model, we train a discriminator that tries to distinguish whether a user's action was taken in response to the compressed image or the original. We evaluate our method through experiments with human participants on four tasks: reading handwritten digits, verifying photos of faces, browsing an online shopping catalogue, and playing a car racing video game. The results show that our method learns to match the user's actions with and without compression at lower bitrates than baseline methods, and adapts the compression model to the user's behavior: it preserves the digit number and randomizes handwriting style in the digit reading task, preserves hats and eyeglasses while randomizing faces in the photo verification task, preserves the perceived price of an item while randomizing its color and background in the online shopping task, and preserves upcoming bends in the road in the car racing game.Preprint. Under review.

Control-Aware Prediction Objectives for Autonomous Driving

“…We also avoid KL distance losses in our work since this assumes non-trivial stochasticity in the controller or data, which is not always the case. Other works have also used KL policy distances to investigate how observations can be compressed while preserving human-like actions had they remained uncompressed [28]. Work by Piazzoni et al [27] also investigates how perceptual metrics affect downstream planning but are specific to perception.…”

Expressive Communication: A Common Framework for Evaluating Developments in Generative Models and Steering Interfaces

“…While most of this work is focused on downstream tasks such as teaching artificial agents to play video games [19], an emerging thread of research is exploring ways of evaluating and optimizing for Human-AI collaboration [2]. Studies have show human feedback and interaction can be used to optimize generative models for collaboration on tasks such as classification or assistive game playing [16,23]. This work examines a new approach towards evaluating and optimizing for Human-AI collaboration in the domain of creative expression.…”

“…A new frontier for ML research is explicitly training models to optimize downstream Human-AI collaborative tasks [2,16,23]. As human interaction is expensive and does not scale well to training, these works use limited human interactions and evaluations to learn approximations of human behavior and preferences that can be used for automated training [4,28].…”