Peer Review #1 of "On the impact of service-oriented patterns on software evolvability: a controlled experiment and metric-based analysis (v0.2)"

Abstract

Background: Design patterns are supposed to improve various quality attributes of software systems. However, there is controversial quantitative evidence of this impact. Especially for younger paradigms such as service-and Microservice-based systems, there is a lack of empirical studies. Objective: In this study, we focused on the effect of four service-based patterns-namely Process Abstraction, Service Façade, Decomposed Capability, and Event-Driven Messaging-on the evolvability of a system from the viewpoint of inexperienced developers. Method: We conducted a controlled experiment with Bachelor students (N=69). Two functionally equivalent versions of a service-based web shop-one with patterns (treatment group), one without (control group)-had to be changed and extended in three tasks. We measured evolvability by the effectiveness and efficiency of the participants in these tasks. Additionally, we compared both system versions with nine structural maintainability metrics for size, granularity, complexity, cohesion, and coupling. Results: Both experiment groups were able to complete a similar number of tasks within the allowed 90 minutes. Median effectiveness was 1/3. Mean efficiency was 12% higher in the treatment group, but this difference was not statistically significant. Only for the third task, we found statistical support for accepting the alternative hypothesis that the pattern version led to higher efficiency. In the metric analysis, the pattern version had worse measurements for size and granularity while simultaneously having slightly better values for coupling metrics. Complexity and cohesion were not impacted. Interpretation: For the experiment, our analysis suggests that the difference in efficiency is stronger with more experienced participants and increased from task to task. With respect to the metrics, the patterns introduce additional volume in the system, but also seem to decrease coupling in some areas. Conclusions: Overall, there was no clear evidence for a decisive positive effect of using service-based patterns, neither for the student experiment nor for the metric analysis. This effect might only be visible in an experiment setting with higher initial effort to understand the system or with more experienced developers.