Lilith: Scalable execution of user code for distributed computing

Evensky, David A.; Gentile, Ann C.; Camp, L. Jean; Armstrong, Robert C.

doi:10.1109/hpdc.1997.626436

Cited by 7 publications

(7 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For example, software multicast/reduction networks are crucial to scalable tool use [1,6,11,16]. The RM must be aware of and willing to launch this second kind of non-application entity.…”

Section: Figure 1: Remote Execution With Resource Manager and Run-timmentioning

confidence: 99%

The Tool Dæmon Protocol (TDP)

Miller

Cortés

Senar

et al. 2003

Proceedings of the 2003 ACM/IEEE Conference on Supercomputing

View full text Add to dashboard Cite

Run-time tools are crucial to program development. In our desktop computer environments, we take for granted the availability of tools for operations such as debugging, profiling, tracing, checkpointing, and visualization. When programs move into distributed or Grid environments, it is difficult to find such tools. This difficulty is caused by the complex interactions necessary between application program, operating system and layers of job scheduling and process management software. As a result, each run-time tool must be individually ported to run under a particular job management system; for m tools and n environments, the problem becomes an m × n effort, rather than the hoped-for m + n effort. Variations in underlying operating systems can make this problem even worse. The consequence of this situation is a paucity of tools in distributed and Grid computing environments.In response to the problem, we have analyzed a variety of job scheduling environments and run-time tools to better understand their interactions. From this analysis, we isolated what we believe are the essential interactions between the runtime tool, job scheduler and resource manager, and application program. We are proposing a standard interface, called the Tool Daemon Protocol (TDP) that codifies these interactions and provides the necessary communication functions. We have implemented a pilot TDP library and experimented with Parador, a prototype using the Paradyn Parallel Performance tools profiling jobs running under the Condor batch-scheduling environment.

show abstract

“…For example, software multicast/reduction networks are crucial to scalable tool use [1,6,11,16]. The RM must be aware of and willing to launch this second kind of non-application entity.…”

Section: Figure 1: Remote Execution With Resource Manager and Run-timmentioning

confidence: 99%

The Tool Dæmon Protocol (TDP)

Miller

Cortés

Senar

et al. 2003

Proceedings of the 2003 ACM/IEEE Conference on Supercomputing

View full text Add to dashboard Cite

show abstract

“…Furthermore, the moderate fan-outs at each MRNet process allows data reductions to be pipelined as they pass through the network, keeping reduction throughput high as application size increases. The trends in MRNet's micro-benchmark scalability studies are perhaps to be expected; previous tool infrastructures using a hierarchy of processes such as the Ladebug parallel debugger [4] and Lilith [11] show similar scalability trends.…”

Section: Micro-benchmark Resultsmentioning

confidence: 88%

“…This type of communication structure has been examined previously (e.g., [3,11,17,20,21,25,28]). However, several features make MRNet especially wellsuited as a general facility for building scalable parallel tools: • Flexible organization.…”

Section: Introductionmentioning

confidence: 99%

MRNet

Roth

Arnold

Miller

2003

Proceedings of the 2003 ACM/IEEE Conference on Supercomputing

140

View full text Add to dashboard Cite

We present MRNet, a software-based multicast/reduction network for building scalable performance and system administration tools. MRNet supports multiple simultaneous, asynchronous collective communication operations. MRNet is flexible, allowing tool builders to tailor its process network topology to suit their tool's requirements and the underlying system's capabilities. MRNet is extensible, allowing tool builders to incorporate custom data reductions to augment its collection of built-in reductions. We evaluated MRNet in a simple test tool and also integrated into an existing, real-world performance tool with up to 512 tool back-ends. In the realworld tool, we used MRNet not only for multicast and simple data reductions but also with custom histogram and clock skew detection reductions. In our experiments, the MRNet-based tools showed significantly better performance than the tools without MRNet for average message latency and throughput, overall tool start-up latency, and performance data processing throughput.

show abstract

“…Although there exist tools [7] to accomplish some of these tasks, they are either not scalable or rely on relatively weak security. We describe Lilith's functionality and API, present an example tool that capitalizes on Lilith's features, and discuss the security model under design.…”

Section: Introductionmentioning

confidence: 99%

The Lilith Framework for the Rapid Development of Secure Scalable Tools for Distributed Computing

Evensky

Gentile

Wyckoff

et al. 1999

Distributed Applications and Interoperable Systems II

View full text Add to dashboard Cite

Lilith is a general purpose framework written in Java, that provides a highly scalable distribution of user code across a heterogeneous computing platform. By creation of suitable user code, the Lilith framework can be used for tool development. Lilith promotes rapid development since it handles all the details of code distribution and communication, while the user code need only define the tool functionality on a single node, in accordance with a simple API. The scalable performance provided by Lilith is crucial to the production of time-effective tools for large distributed systems. This timeefficiency, however, allows both constructive and destructive tasks to be accomplished quickly; therefore, security is a concern. We present the Lilith API and a prototype example of the usage of Lilith in distributed computing and we discuss the security model, which is currently under design.

show abstract

Lilith: Scalable execution of user code for distributed computing

Cited by 7 publications

References 1 publication

The Tool Dæmon Protocol (TDP)

The Tool Dæmon Protocol (TDP)

MRNet

The Lilith Framework for the Rapid Development of Secure Scalable Tools for Distributed Computing

Contact Info

Product

Resources

About