Empirical systems research is facing a dilemma. Minor aspects of an experimental setup can have a significant impact on its associated performance measurements and potentially invalidate conclusions drawn from them. Examples of such influences, often called hidden factors, include binary link order, process environment size, compiler generated randomized symbol names, or group scheduler assignments. The growth in complexity and size of modern systems will further aggravate this dilemma, especially with the given time pressure of producing results. So how can one trust any reported empirical analysis of a new idea or concept in computer science?This paper introduces DataMill, a community-based easyto-use services-oriented open benchmarking infrastructure for performance evaluation. DataMill facilitates producing robust, reliable, and reproducible results. The infrastructure incorporates the latest results on hidden factors and automates the variation of these factors. Multiple research groups already participate in DataMill.DataMill is also of interest for research on performance evaluation. The infrastructure supports quantifying the effect of hidden factors, disseminating the research results beyond mere reporting. It provides a platform for investigating interactions and composition of hidden factors.
Reservation-based scheduling mechanisms have successfully been used for supporting real-time applications whose tasks exhibit high variability in their execution or release times. Indeed, such mechanisms are able to preallocate system bandwidth to the application tasks so that temporal isolation between them is ensured. However, bandwidth allocation is usually based on off-line policies, which may not be suitable for real-time applications that are structured as having several modes of operation, each one requiring a distinct level of system bandwidth. Variations in light conditions, the changing of energy levels, errordetection, or operator commands are examples of events that may trigger a different mode of operation in multi-mode adaptive real-time applications.In this paper we address the problem of dynamically reconfiguring scheduling parameters of reservation-based mechanisms, offering support for multi-mode adaptive realtime applications. Assuming that each reconfiguration option gives a benefit for the system, reconfiguration is seen as an optimization problem whose objective is to maximize the overall system benefit. Two different models for the problem are formulated, the Integer Programming (IP) and the Linear Programming (LP) formulations. The IP formulation gives rise to an NP-Hard problem for which we give efficient approximate solutions. Also, an optimal and polynomial solution to the LP formulation is derived. Results obtained from extensive simulation indicate the good performance of the proposed reconfiguration mechanisms. *
Research has shown that correctly conducting and analysing computer performance experiments is difficult. This paper investigates what is necessary to conduct successful computer performance evaluation by attempting to repeat a prior experiment: the comparison between two Linux schedulers.In our efforts, we found that exploring an experimental space through a series of incremental experiments can be inconclusive, and there may be no indication of how much experimentation will be enough. Analysis of variance (ANOVA), a traditional analysis method, is able to partly solve the problems with the previous approach, but we demonstrate that ANOVA can be insufficient for proper analysis due to the requirements it imposes on the data.Finally, we demonstrate the successful application of quantile regression, a recent development in statistics, to computer performance experiments. Quantile regression can provide more insight into the experiment than ANOVA, with the additional benefit of being applicable to data from any distribution. This property makes it especially useful in our field, since non-normally distributed data is common in computer experiments.
The complexity of real-time systems has substantially increased in the past few years regarding both hardware and software aspects. The use of modern sensors, able to capture image and audio data, demands predictable multimedia-like data processing. Moreover, applications like autonomous robots, surveillance, or modern multimedia players may well be characterized by several operation modes, each one associated with light conditions, vision angle, change in user requirements, etc. In this paper, we describe suitable scheduling mechanisms that address these aspects.Application modes are characterized by their required processing bandwidth and benefit values. By using bandwidth reservation schedulers, dynamic reconfiguring scheduling parameters is seen as an optimization problem whose goal is to maximize the overall system benefit subject to schedulability constraints. Two different models for the problem are defined, Discrete and Continuous. The former gives rise to an NP-Hard problem for which efficient approximate solutions are derived. An optimal and polynomial solution to the Continuous model is derived. Both models are then extended to incorporate task execution times described as probability distributions. Making use of this stochastic modeling one is able to dynamically reconfigure the scheduler subject to probabilistic schedulability guarantees. The derived solutions are evaluated by extensive simulation, which indicates the good performance of the proposed reconfiguration mechanisms.
scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.
hi@scite.ai
10624 S. Eastern Ave., Ste. A-614
Henderson, NV 89052, USA
Copyright © 2024 scite LLC. All rights reserved.
Made with đź’™ for researchers
Part of the Research Solutions Family.