Superbloom: Bloom filter meets Transformer

Abstract

We extend the idea of word pieces in natural language models to machine learning tasks on opaque ids. This is achieved by applying hash functions to map each id to multiple hash tokens in a much smaller space, similarly to a Bloom filter. We show that by applying a multilayer Transformer to these Bloom filter digests, we are able to obtain models with high accuracy. They outperform models of a similar size without hashing and, to a large degree, models of a much larger size trained using sampled softmax with the same computational budget. Our key observation is that it is important to use a multi-layer Transformer for Bloom filter digests to remove ambiguity in the hashed input. We believe this provides an alternative method to solving problems with large vocabulary size.

Avances y discusiones sobre el uso de inteligencia artificial (IA) en educación

“…Estos grandes modelos de lenguaje contienen billones de parámetros en su diseño y son entrenados con un inmenso volumen de datos procedentes de diversas fuentes, como por ejemplo textos, libros, artículos, códigos y conversaciones en línea, lo que les permite aprender patrones y reglas gramaticales (Pereira & Moura, 2023). De esta forma, se entrenan para predecir palabras siguiendo la sintaxis y el sentido de un texto dado (Anderson et al, 2020). Algunos presentan usos únicos, como, por ejemplo, generar respuestas en conversaciones, describir conceptos o temas complejos, generar códigos nuevos o corregir errores en códigos informáticos existentes.…”

On the Regularity of Attention

“…This robustness has been used recently in Anderson et al (2020) to operate self-attention transformer models on reduced-size vocabularies by hashing, where the model must be robust to hash collisions of the larger original vocabulary. The authors of that paper compare this robustness to error correcting output codes (Berger, 1999;Dietterich and Bakiri, 1994).…”