Project Kittyhawk: building a global-scale computer

Abstract

This paper describes Project Kittyhawk, an undertaking at IBM Research to explore the construction of a nextgeneration platform capable of hosting many simultaneous web-scale workloads. We hypothesize that for a large class of web-scale workloads the Blue Gene/P platform is an order of magnitude more efficient to purchase and operate than the commodity clusters in use today. Driven by scientific computing demands the Blue Gene designers pursued an aggressive system-on-a-chip methodology that led to a scalable platform composed of air-cooled racks. Each rack contains more than a thousand independent computers with highspeed interconnects inside and between racks.We postulate that the same demands of efficiency and density apply to web-scale platforms. This project aims to develop the system software to enable Blue Gene/P as a generic platform capable of being used by heterogeneous workloads. We describe our firmware and operating system work to provide Blue Gene/P with generic system software, one of the results of which is the ability to run thousands of heterogeneous Linux instances connected by TCP/IP networks over the high-speed internal interconnects.

Many-task computing for grids and supercomputers

“…They normally come well balanced, with proprietary, highspeed, and low-latency network interconnects to give tightly-coupled applications good opportunities to scale well at full system scales. Even IBM has proposed in their internal project Kittyhawk [15] that Blue Gene/P can be used to run non-traditional workloads (e.g. HTC).…”

“…2) The cost to manage and run on petascale systems like the Blue Gene/P is less than that of conventional clusters or Grids [15]. For example, a single 13.9 TF Blue Gene/P rack draws 40 kilowatts, for 0.35 GF/watt.…”

Toward loosely coupled programming on petascale systems

“…They normally come well balanced, with proprietary, high-speed, and low-latency network interconnects to give tightly-coupled applications that use MPI good opportunities to scale well at full system scales. Even IBM has proposed in their internal project Kittyhawk [16] that the Blue Gene/P can be used to run non-traditional workloads, such as those found in the general Internet, which are by definition part of a loosely coupled system. Four factors motivate the support of MTC applications on petascale HPC systems.…”

“…[16] For example, a single 13.9 TF Blue Gene/P rack draws 40 kilowatts, for 0.35 GF/watt. Two other systems that get good compute power per watt consumed are the SiCortex with 0.32 GF/watt and the Blue Gene/L with 0.23 GF/watt.…”

“…However, it is far less amenable to the utilization and chaining of exiting application programs, and it often involves the development of custom filtering scripts. We have compared our work with Condor glide-ins [16] in the past [6], but our work focuses on performance and efficiency, while Condor emphasizes more on robustness and recoverability, which limits its efficiency for MTC applications in large-scale systems. An approach by Reid called "task farming" [26], also at the programming language level, has been evaluated on the Blue Gene/L as a proof of concept, but offered no performance evaluation for comparison, and required that applications be modified to run over the proposed middleware.…”