Randomized Positional Encodings Boost Length Generalization of Transformers

Ruoss, Anian; Grégoire, Delétang,; Genewein, Tim; Grau-Moya, Jordi; Csordás, Róbert; Bennani, Mehdi; Legg, Shane; Veness, Joel

doi:10.18653/v1/2023.acl-short.161

Cited by 3 publications

(1 citation statement)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By introducing labeled tokens (class markers), the Transformer network can be directly used for image classification tasks. Anian Ruoss et al 18 proposed a new family of positional encodings to address this challenge.…”

Section: Related Workmentioning

confidence: 99%

FrseGAN: Free‐style editable facial makeup transfer based on GAN combined with transformer

Xu,

Wang,

Yang

2024

Computer Animation & Virtual

View full text Add to dashboard Cite

Makeup in real life varies widely and is personalized, presenting a key challenge in makeup transfer. Most previous makeup transfer techniques divide the face into distinct regions for color transfer, frequently neglecting details like eyeshadow and facial contours. Given the successful advancements of Transformers in various visual tasks, we believe that this technology holds large potential in addressing pose, expression, and occlusion differences. To explore this, we propose novel pipeline which combines well‐designed Convolutional Neural Network with Transformer to leverage the advantages of both networks for high‐quality facial makeup transfer. This enables hierarchical extraction of both local and global facial features, facilitating the encoding of facial attributes into pyramid feature maps. Furthermore, a Low‐Frequency Information Fusion Module is proposed to address the problem of large pose and expression variations which exist between the source and reference faces by extracting makeup features from the reference and adapting them to the source. Experiments demonstrate that our method produces makeup faces that are visually more detailed and realistic, yielding superior results.

show abstract

Section: Related Workmentioning

confidence: 99%

FrseGAN: Free‐style editable facial makeup transfer based on GAN combined with transformer

Xu,

Wang,

Yang

2024

Computer Animation & Virtual

View full text Add to dashboard Cite

show abstract

Rethinking tokenization

Yang

2024

IJChL

View full text Add to dashboard Cite

Tokenization significantly influences language models (LMs)’ performance. This paper traces the evolution of tokenizers from word-level to subword-level, analyzing how they balance tokens and types to enhance model adaptability while controlling complexity. Despite subword tokenizers like Byte Pair Encoding (BPE) overcoming many word tokenizer limitations, they encounter difficulties in handling non-Latin languages and depend heavily on extensive training data and computational resources to grasp the nuances of multiword expressions (MWEs). This article argues that tokenizers, more than mere technical tools, should drawing inspiration from the cognitive science about human language processing. This study then introduces the “Principle of Least Effort” from cognitive science, that humans naturally seek to reduce cognitive effort, and discusses the benefits of this principle for tokenizer development. Based on this principle, the paper proposes that the Less-is-Better (LiB) model could be a new approach for LLM tokenizer. The LiB model can autonomously learn an integrated vocabulary consisting of subwords, words, and MWEs, which effectively reduces both the numbers of tokens and types. Comparative evaluations show that the LiB tokenizer outperforms existing word and BPE tokenizers, presenting an innovative method for tokenizer development, and hinting at the possibility of future cognitive science-based tokenizers being more efficient.

show abstract

Tracking With Saliency Region Transformer

Liu,

Li,

et al. 2024

IEEE Trans. on Image Process.

View full text Add to dashboard Cite

Randomized Positional Encodings Boost Length Generalization of Transformers

Cited by 3 publications

References 0 publications

FrseGAN: Free‐style editable facial makeup transfer based on GAN combined with transformer

FrseGAN: Free‐style editable facial makeup transfer based on GAN combined with transformer

Rethinking tokenization

Tracking With Saliency Region Transformer

Contact Info

Product

Resources

About