Mining Workflows for Anomalous Data Transfers

Tu, Huy; Papadimitriou, George; Kiran, Mariam; Mandal, Anirban; Deelman, Ewa

doi:10.1109/msr52588.2021.00013

Cited by 11 publications

(7 citation statements)

References 50 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…More specifically, Scott-Knott recursively partitions the list of candidates (c) into two sub-lists (c 1 and c 2 ) which the expected mean value before and after the division should be maximized [55], [56], [57]:…”

Section: Discussionmentioning

confidence: 99%

What Not to Test (For Cyber-Physical Systems)

Ling

2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

Testing complex Cyber Physical Systems (CPSs) can be expensive and time consuming. Current state-of-the-art methods that explore this problem are fully-supervised; i.e. they require that all examples are labeled. On the other hand, the GenClu system (introduced in this paper) takes a semi-supervised approach; i.e. (a) only a small subset of information is actually labeled (via simulation) and (b) those labels are then spread across the rest of the data. When applied to five open-source CPSs, GenClu's test generation can be multiple orders of magnitude faster than the prior state of the art. Further, when assessed via mutation testing, tests generated by GenClu were as good or better than anything else tested here. Hence, we recommend semi-supervised methods over prior methods (evolutionary search and fully-supervised learning). To enable open science, all the data and scripts used in this study are available at https://github.com/ai-se/CPS test generation.

show abstract

Section: Discussionmentioning

confidence: 99%

What Not to Test (For Cyber-Physical Systems)

Ling

2023

IIEEE Trans. Software Eng.

View full text Add to dashboard Cite

show abstract

“…After that, Scott-Knott will declare the one of the split as the best split. The best split should maximize the difference 𝐸 (Δ) in the expected mean value before and after the split [40,46]:…”

Section: Discussionmentioning

confidence: 99%

“…Therefore, our observation may differ when different parameters are used. We would consider hyper-parameter tuning [39,40] in future work to mitigate this threat.…”

Section: Threats To Validitymentioning

confidence: 99%

Faster Multi-Goal Simulation-Based Testing Using DoLesS (Domination with Least Square Approximation)

Liu¹

2021

Preprint

Self Cite

View full text Add to dashboard Cite

For cyber-physical systems, finding a set of test cases with the least cost by exploring multiple goals is a complex task. For example, Arrieta et al. reported that state-of-the-art optimizers struggle to find minimal test suites for this task. To better manage this task, we propose DoLesS (Domination with Least Squares Approximation) which uses a domination predicate to sort the space of possible goals to a small number of representative examples. Multi-objective domination then divides these examples into a "best" set and the remaining "rest" set. After that, DoLesS applies an inverted least squares approximation approach to learn a minimal set of tests that can distinguish best from rest in the reduced example space.DoLesS has been tested on four cyber-physical models: a tank flow model; a model of electric car windows; a safety feature of an AC engine; and a continuous PID controller combined with a discrete state machine. Comparing to the recent state-of-the-art paper attempted the same task, DoLesS performs as well or even better as state-of-the-art, while running 80-360 times faster on average (seconds instead of hours). Hence, we recommend DoLesS as a fast method to find minimal test suites for multi-goal cyberphysical systems. For replication purposes, all our code is on-line: https://github.com/hellonull123/Test_Selection_2021.

show abstract

“…To satisfy Sarro et al, we explored hyperparameter optimization. A common result in software analytics [1-3, 23, 24, 59, 60, 74, 76] (and other domains outside of SE [5,10,17,43,58,62]) is that automatic hyperparameter optimization algorithms can find good configuration settings. But for small data sets, these methods are highly error prone.…”

Section: Introductionmentioning

confidence: 99%

Optimizing Predictions for Very Small Data Sets: a case study on Open-Source Project Health Prediction

Lustosa¹

2023

Preprint

View full text Add to dashboard Cite

When learning from very small data sets, the resulting models can make many mistakes. For example, consider learning predictors for open source project health. The training data for this task may be very small (e.g. five years of data, collected every month means just 60 rows of training data). Using this data, prior work had unacceptably large errors in their learned predictors.We show that these high errors rates can be tamed by better configuration of the control parameters of the machine learners. For example, we present here a landscape analytics method (called SNEAK) that (a) clusters the data to find the general landscape of the hyperparameters; then (b) explores a few representatives from each part of that landscape.SNEAK is both faster and and more effective than prior state-of-the-art hyperparameter optimization algorithms (FLASH, HYPEROPT, OPTUNA, and differential evolution). For example, SNEAK found its configurations after assessing 10 2 options, while other tools needed to assess 350 to 5000 (for OPTUNA, HYPEROPT) options. More importantly, the configurations found by SNEAK had far less error that other methods. For example, for project health indicators such as 𝐶= number of commits; 𝐼 =number of closed issues, and 𝑅=number of closed pull requests, SNEAK's 12 month prediction errors are {I=0%, R=33% C=47%} while other methods have far larger errors of {I=61%,R=119% C=149%}. We conjecture that SNEAK works so well since it finds the most informative regions of the hyperparameters, then jumps to those regions. Other methods (that do not reflect over the landscape) can waste time exploring less informative options.From this, we make the following conclusions. Firstly, for predicting open source project health, we recommend landscape analytics (e.g. SNEAK). Secondly, and more generally, when learning from very small data sets, using hyperparameter optimization (e.g. SNEAK) to select learning control parameters. Due to its speed and implementation simplicity, we suggest SNEAK might also be useful in other "data-light" SE domains.To assist other researchers in repeating, improving, or even refuting our results, all our scripts and data are available on GitHub at https://github.com/zxcv123456qwe/niSneak.

show abstract

Mining Workflows for Anomalous Data Transfers

Cited by 11 publications

References 50 publications

What Not to Test (For Cyber-Physical Systems)

What Not to Test (For Cyber-Physical Systems)

Faster Multi-Goal Simulation-Based Testing Using DoLesS (Domination with Least Square Approximation)

Optimizing Predictions for Very Small Data Sets: a case study on Open-Source Project Health Prediction

Contact Info

Product

Resources

About