In the past two decades, the use of digital libraries (DLs) has grown significantly. Accordingly, questions about the utility, usability and cost of DLs have started to arise, and greater attention is being paid to the quality evaluation of this type of information system. Since DLs are destined to serve user communities, one of the main aspects to be considered in DL evaluation is the user’s opinion. The literature on this topic has produced a set of varied criteria to judge DLs from the user’s perspective, measuring instruments to elicit users’ opinions, and approaches to analyse the elicited data to conclude an evaluation. This paper provides a literature review of the quality evaluation of DLs based on users’ perceptions. Its main contribution is to bring together previously disparate streams of work to help shed light on this thriving area. In addition, the various studies are discussed, and some challenges to be faced in the future are proposed.

No abstract

Several relevant analyses on configurable software systems remain intractable because they require examining vast and highly-constrained configuration spaces. Those analyses could be addressed through statistical inference, i.e., working with a much more tractable sample that later supports generalizing the results obtained to the entire configuration space. To make this possible, the laws of statistical inference impose an indispensable requirement: each member of the population must be equally likely to be included in the sample, i.e., the sampling process needs to be "uniform". Various SAT-samplers have been developed for generating uniform random samples at a reasonable computational cost. Unfortunately, there is a lack of experimental validation over large configuration models to show whether the samplers indeed produce genuine uniform samples or not. This paper (i) presents a new statistical test to verify to what extent samplers accomplish uniformity and (ii) reports the evaluation of four state-of-the-art samplers: Spur, QuickSampler, Unigen , and Smarch. According to our experimental results, only Spur satisfies both scalability and uniformity.

Variability models are broadly used to specify the configurable features of highly customizable software. In practice, they can be large, defining thousands of features with their dependencies and conflicts. In such cases, visualization techniques and automated analysis support are crucial for understanding the models. This paper contributes to this line of research by presenting a novel, probabilistic foundation for statistical reasoning about variability models. Our approach not only provides a new way to visualize, describe and interpret variability models, but it also supports the improvement of additional state-of-the-art methods for software product lines; for instance, providing exact computations where only approximations were available before, and increasing the sensitivity of existing analysis operations for variability models. We demonstrate the benefits of our approach using real case studies with up to 17,365 features, and written in two different languages (KConfig and feature models).