Large-Scale Analysis of the Docker Hub Dataset

Zhao, Nannan; Tarasov, Vasily; Albahar, Hadeel; Anwar, Ali; Rupprecht, Lukas; Skourtis, Dimitrios; Warke, Amit; Mohamed, Mohamed; Butt, Ali R.

doi:10.1109/cluster.2019.8891000

Cited by 28 publications

(24 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With the number of images increasing, duplicate data grows dramatically in the registry. It is reported that when the total size of unpacked images is 167 TB (unpacked from 47 TB compressed images), only 3.2% of files left after file-level deduplication, which in total only occupy 24 TB [11]. While file-level and chunk-level deduplication show similar space-saving effort, chunk-level deduplication can cause a dramatic increase in the number of unique objects, i.e., the number of unique objects of chunk- Data sharing when launching different containers.…”

Section: Motivationmentioning

confidence: 99%

“…There are two reasons why the image format cannot support efficient image storage and deployment. (1) Regarding container storage, there is substantial redundant data between different layers of images, which results in a large waste of storage space [10,11]. Although layered images allow layer-level deduplication to reduce storage footprint, there is still substantial data redundancy that cannot be detected and removed at this coarse deduplication granularity.…”

Section: Introductionmentioning

confidence: 99%

“…On the one hand, only required data are downloaded to deploy containers in the new format [9,[18][19][20]. On the other hand, the new image allows file-level sharing among images [11,21] to eliminate duplicate data in the registry as well as further reduce data to be downloaded in clients. To the best of our knowledge, this is the first work on improving both storage and deployment efficiencies of containers while being compatible with the current Docker framework.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Gear: Enable Efficient Container Storage and Deployment with a New Image Format

Fan

Bian

et al. 2021

2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS)

View full text Add to dashboard Cite

Containers have been widely used in various cloud platforms as they enable agile and elastic application deployment through their process-based virtualization and layered image system. However, different layers of a container image may contain substantial duplicate and unnecessary data, which slows down its deployment due to long image downloading time and increased burden on the image registry. To accelerate the deployment and reduce the size of the registry, we propose a new image format, named Gear image, that consists of two parts: a Gear index describing the structure of the image's file system and a set of files that are required when running an application. The Gear index is represented as a single-layer image compatible with the existing deployment framework. Containers can be launched by pulling a Gear index and on demand retrieving files pointed to by the index. Furthermore, the Gear image enables a file-level sharing mechanism, which helps remove duplicate data in the registry and avoid repeated downloading of identical files by a client. We implement a prototype of the container framework, named Gear, supporting the new image format. Evaluation shows that Gear saves 54% storage capacity in the registry, speeds up container startup by up to 5×, and reduces 84% bandwidth demands.

show abstract

Section: Motivationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Gear: Enable Efficient Container Storage and Deployment with a New Image Format

Fan

Bian

et al. 2021

2021 IEEE 41st International Conference on Distributed Computing Systems (ICDCS)

View full text Add to dashboard Cite

show abstract

“…Their study differs from ours as we focus on image upgrades. Zhao et al [18] studied duplication of files in images on Docker Hub from the point of view of reducing the overall storage requirements of Docker Hub.…”

Section: Related Workmentioning

confidence: 99%

Should you Upgrade Official Docker Hub Images in Production Environments?

Gholami

Khazaei

Bezemer

2021

2021 IEEE/ACM 43rd International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER)

View full text Add to dashboard Cite

Docker, one of the most popular software containerization technologies, allows a user to deploy Docker images to create and run containers. While Docker images facilitate the deployment and in-place upgrading of an application in a production environment by replacing its container with one based on a newer image, many dependencies could change at once during such an image upgrade, which can potentially be a source of risk. In this paper, we study the official Docker images on Docker Hub and explore how packages are changing in these images. We found that the number of package changes varies across different types of applications and that often the changing packages are utility packages. Our study takes a first important look at potential risks when doing an in-place upgrade of a Docker image.

show abstract

“…Further, though one is advised to use volumes, read-write data still commonly exists in container images. Over the analysis of 500,000 public container images [63], 44% are document files such as Microsoft office files, texts, source code, scripts, etc., many of which are supposed to be modified. It also reports a certain amount of database-related files in container images indicating that Docker developers run databases inside containers.…”

Section: Startup Timementioning

confidence: 99%

Baoverlay

Sun

Lei

Shin

et al. 2020

Proceedings of the 11th ACM Symposium on Cloud Computing

View full text Add to dashboard Cite

Container storage commonly relies on overlay file systems to interpose read-only container images upon backing file systems. While being transparent to and compatible with most existing backing file systems, the overlay file-system approach imposes nontrivial I/O overhead to containerized applications, especially for writes: To write a file originating from a read-only container image, the whole file will be copied to a separate, writable storage layer, resulting in long write latency and inefficient use of container storage. In this paper, we present BAOverlay, a lightweight, block-accessible overlay file system: Equipped with a new block-accessibility attribute, BAOverlay not only exploits the benefit of using an asynchronous copy-on-write mechanism for fast file updates but also enables a new file format for efficient use of container storage space. We have developed a prototype of BAOverlay upon Linux Ext4. Our evaluation with both micro-benchmarks and real-world applications demonstrates the effectiveness of BAOverlay with improved write performance and on-demand container storage usage. CCS CONCEPTS • Information systems → Storage virtualization; • Software and its engineering → File systems management.

show abstract

Large-Scale Analysis of the Docker Hub Dataset

Cited by 28 publications

References 13 publications

Gear: Enable Efficient Container Storage and Deployment with a New Image Format

Gear: Enable Efficient Container Storage and Deployment with a New Image Format

Should you Upgrade Official Docker Hub Images in Production Environments?

Baoverlay

Contact Info

Product

Resources

About