Can Vision Transformers Learn without Natural Images?

Abstract

Is it possible to complete Vision Transformer (ViT) pre-training without natural images and human-annotated labels? This question has become increasingly relevant in recent months because while current ViT pre-training tends to rely heavily on a large number of natural images and human-annotated labels, the recent use of natural images has resulted in problems related to privacy violation, inadequate fairness protection, and the need for labor-intensive annotations. In this paper, we experimentally verify that the results of formula-driven supervised learning (FDSL) framework are comparable with, and can even partially outperform, sophisticated self-supervised learning (SSL) methods like SimCLRv2 and MoCov2 without using any natural images in the pre-training phase. We also consider ways to reorganize FractalDB generation based on our tentative conclusion that there is room for configuration improvements in the iterated function system (IFS) parameter settings of such databases. Moreover, we show that while ViTs pre-trained without natural images produce visualizations that are somewhat different from ImageNet pre-trained ViTs, they can still interpret natural image datasets to a large extent. Finally, in experiments using the CIFAR-10 dataset, we show that our model achieved a performance rate of 97.8, which is comparable to the rate of 97.4 achieved with SimCLRv2 and 98.0 achieved with ImageNet.

FractalAD: A simple industrial anomaly segmentation method using fractal anomaly generation and backbone knowledge distillation

“…Instead, the model learns from images automatically generated using fractal geometry, computer graphics, and other methods. Existing studies [15,16,21,23,24] have shown that such models can effectively learn representations through fractal images, Bessel curves [25], and Perlin noise [26], improving the interpretability of the features and performing almost as well as pretrained models based on ImageNet.…”

Does Formula-Driven Supervised Learning Work on Small Datasets?

“…This finding shows that fractal geometry plays an important role in dataset construction using FDSL. Nakashima et al [48] confirmed that the FDSL framework is effective for pre-training Vision Transformers (ViTs). More interestingly, they suggest that FDSL are more likely to benefit ViTs than CNNs.…”

“…* indicates that 5,000 epochs were learned during fine-tuning. † refers to a quote from[48]. The Underlined bold and bold scores indicate the best and second-best values, respectively.…”