Basics of Distributed Machine Learning

Jiang, Jiawei; Cui, Bin; Zhang, Ce

doi:10.1007/978-981-16-3420-8_2

Cited by 3 publications

(11 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Importance sampling for efficient training. Samplingbased methods [42,43,14,30] aims to idendity a compact, yet representative subset of the training dataset that satisfies the original objectives for efficient model training. This is usually achieved through exploring the early training stage [43], constructing a proxy model [14], or utilizing consistency score (C-score) [30].…”

Section: Related Workmentioning

confidence: 99%

“…Samplingbased methods [42,43,14,30] aims to idendity a compact, yet representative subset of the training dataset that satisfies the original objectives for efficient model training. This is usually achieved through exploring the early training stage [43], constructing a proxy model [14], or utilizing consistency score (C-score) [30]. However, the empirical performance gap between sampling-based methods and the baseline approach of random selection is insignificant, particularly in large-scale datasets like ImageNet-1k (See Fig.…”

Section: Related Workmentioning

confidence: 99%

“…The blue and yellow lines in Fig. 1 depict the results of random sampling and sampling based on c-score [30], respectively. Random sampling of the dataset (yellow) had the most significant negative impact on performance as storage capacity decreased.…”

Section: Imagenet-1k Classificationmentioning

confidence: 99%

“…There are two major directions to storage-efficient vision model training. The first direction aims to reduce the total number of data points by discarding less important samples [42,43,30] or synthesizing more "condensed" images than natural images [74,73]. However, this approach shows a significant performance drop compared to the full dataset (the blue and yellow lines in Fig.…”

Section: Introductionmentioning

confidence: 99%

“…Our method has several advantages over previous storage-efficient methods. First, as we use a frozen pretrained tokenizer (only using ImageNet-1k) that only requires forward operations to extract tokens from images, we do not need an additional optimization for compressing a dataset, such as importance score-based sampling [30], image synthesis methods [74,73], or neural compression [7,8]. Hence, SeiT is easily applicable to unseen datasets or newly upcoming data streams directly.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

SeiT: Storage-Efficient Vision Training with Tokens Using 1% of Pixel Storage

Park¹,

Chun²,

Heo³

et al. 2023

Preprint

View full text Add to dashboard Cite

We need billion-scale images to achieve more generalizable and ground-breaking vision models, as well as massive dataset storage to ship the images (e.g., the LAION-4B dataset needs 240TB storage space). However, it has become challenging to deal with unlimited dataset storage with limited storage infrastructure. A number of storageefficient training methods have been proposed to tackle the problem, but they are rarely scalable or suffer from severe damage to performance. In this paper, we propose a storage-efficient training strategy for vision classifiers for large-scale datasets (e.g., ImageNet) that only uses 1024 tokens per instance without using the raw level pixels; our token storage only needs <1% of the original JPEG-compressed raw pixels. We also propose token augmentations and a Stem-adaptor module to make our approach able to use the same architecture as pixel-based approaches with only minimal modifications on the stem layer and the carefully tuned optimization settings. Our experimental results on ImageNet-1k show that our method significantly outperforms other storage-efficient training methods with a large gap. We further show the effectiveness of our method in other practical scenarios, storage-efficient pre-training, and continual learning. Code is available at https://github.com/naver-ai/seit

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Imagenet-1k Classificationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

SeiT: Storage-Efficient Vision Training with Tokens Using 1% of Pixel Storage

Park¹,

Chun²,

Heo³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

Tight-Sketch: A High-Performance Sketch for Heavy Item-Oriented Data Stream Mining with Limited Memory Size

Li,

Patras

2023

Proceedings of the 32nd ACM International Conference on Information and Knowledge Management

View full text Add to dashboard Cite

Accurate and fast data stream mining is critical and fundamental to many tasks, including time series database handling, big data management and machine learning. Different heavy-based detection tasks, such as heavy hitter, heavy changer, persistent item and significant item detection, have drawn much attention from both the industry and academia. Unfortunately, due to the growing data stream speeds and limited memory (L1 cache) available for real-time processing, existing schemes face challenges in simultaneously achieving high detection accuracy, high memory efficiency, and fast update throughput, as we reveal. To tackle this conundrum, we propose a versatile and elegant sketch framework named Tight-Sketch, which supports a spectrum of heavybased detection tasks. Considering that most items are cold (nonheavy/persistent/significant) in practice, we employ different eviction treatments for different types of items to discard these potentially cold ones as soon as possible, and offer more protection to those that are hot (heavy/persistent/significant). In addition, we propose an eviction method that follows a stochastic decay strategy, enabling Tight-Sketch to only bear small one-sided errors (no overestimation). We present a theoretical analysis of the error bounds and conduct extensive experiments on diverse detection tasks to demonstrate that Tight-Sketch significantly outperforms existing methods in terms of accuracy and update speed. Lastly, we accelerate Tight-Sketch's update throughput by up to 36% with Single Instruction Multiple Data (SIMD) instructions. CCS CONCEPTS• Information systems → Data stream mining.

show abstract

Application of machine learning in chemical engineering: outlook and perspectives

Al Sharah,

Abu Owida,

Alnaimat

et al. 2024

IJ-AI

View full text Add to dashboard Cite

Chemical engineers' formulation, development, and stance processes all heavily rely on models. The physical and economic consequences of these decisions can have disastrous effects. Attempts to employ a hybrid form of artificial intelligence for modeling in various disciplines. However, they fell short of expectations. Due to a rise in the amount of data and computational resources during the previous five years. A lot of recent work has gone into developing new data sources, indexes, chemical interface designs, and machine learning algorithms in an effort to facilitate the adoption of these techniques in the research community. However, there are some important downsides to machine learning gains. The most promising uses for machine learning are in time-critical tasks like real-time optimization and planning that require extreme precision and can build on models that can self-learn to recognize patterns, draw conclusions from data, and become more intelligent over time. Due to their limited exposure to computer science and data analysis, the majority of chemical engineers are potentially vulnerable to the development of artificial intelligence. But in the not-too-distant future, chemical engineers' modeling toolbox will include a reliable machine learning component.

show abstract

Basics of Distributed Machine Learning

Cited by 3 publications

References 39 publications

SeiT: Storage-Efficient Vision Training with Tokens Using 1% of Pixel Storage

SeiT: Storage-Efficient Vision Training with Tokens Using 1% of Pixel Storage

Tight-Sketch: A High-Performance Sketch for Heavy Item-Oriented Data Stream Mining with Limited Memory Size

Application of machine learning in chemical engineering: outlook and perspectives

Contact Info

Product

Resources

About