Continual monitoring and model driven quality improvement methods are becoming critical for product quality within software development organizations. But, setting up such procedures within a global delivery organization poses challenges like -variation in the engagement models across different client projects, variations in build strategies and code ownership model, process related hindrances like overlapping quality processes of the delivery and the product organizations. Moreover, these challenges emerge overtime as the processes become more understood and accepted. We present evolution and design of COSMOS toolkit which exemplifies and addresses such variations in the delivery projects and provides a consistent continual reporting for organizations. The toolkit has become a primary enabler for adoption of a continual quality control process and CQMM, a model based quality improvement process. It addresses among others, key concerns like tools & build variability, legacy code & code ownership, model based rules selection & reporting, variations in applicable thresholds.