Measurements of a distributed file system

Abstract

We analyzed the user-level file access patterns and caching behavior of the Sprite distributed file system. The first part of our analysis repeated a study done in 1985 of the BSD UNIX file system. We found that file throughput has increased by a factor of 20 to an average of 8 Kbytes per second per active user over 10-minute intervals, and that the use of process migration for load sharing increased burst rates by another factor of six. Also, many more very large (multi-megabyte) files are in use today than in 1985. The second part of our analysis measured the behavior of Sprite's main-memory file caches. Client-level caches average about 7 Mbytes in size (about one-quarter to onethird of main memory) and filter out about 50% of the traffic between clients and servers. 35% of the remaining server traffic is caused by paging, even on workstations with large memories. We found that client cache consistency is needed to prevent stale data errors, but that it is not invoked often enough to degrade overall system performance.

On the existence of a spectrum of policies that subsumes the least recently used (LRU) and least frequently used (LFU) policies

“…We used two different types of real workload traces: file system traces from the Sprite network file system [18] and database traces that consist of the DB2 trace used by Johnson and Shasha [10] and the OLTP trace used by both O'Neil et al [11] and Johnson and Shasha [10].…”

Scalable Storage for Digital Libraries

“…These have a negative impact on cache performance, as they can effectively stream through and flush the cache for large enough files, if action is not taken to mitigate such behaviour. Baker et al report cache read miss rates of about 40% in their study, rising to 97% on machines processing large files [12].…”

“…Vogels [200] found that files are open for even shorter periods: 75% are open for less than 10 milliseconds, and 55% of new files are deleted within 5 seconds; 26% are overwritten within 4 milliseconds. Similarly, both Baker et al [12] and Vogels [200] confirm that most files accessed are short, and sequentially accessed (but shifting more towards random access), but that most bytes transferred belong to large files, and that large files are getting larger (20% are 4 MB or larger).…”

“…Baker et al [12] discovered the time between a file being created and subsequently either deleted or truncated to zero length to be less than 30 seconds for 65%-80% of all files, and those files tend to be small, accounting for only about 4%-27% of all new bytes deleted or overwritten within 30 seconds. Vogels [200] found that files are open for even shorter periods: 75% are open for less than 10 milliseconds, and 55% of new files are deleted within 5 seconds; 26% are overwritten within 4 milliseconds.…”

Distributed Caching in Networked File Systems

“…Simulated models are fed with the Sprite File System trace data from the University of California at Berkeley Baker, Hartman, et al 1991 . These traces are publically available.…”