ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints

Jiang, Yue; Stuerzlinger, Wolfgang; Lutteroth, Christof

doi:10.1145/3411764.3445043

Cited by 11 publications

(3 citation statements)

References 81 publications

(100 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…They used the REINFORCE algorithm (Williams, 1992 ; Rennie et al, 2017 ) for their RL. To evaluate their system, they used the OSIE and the Ueyes (Jiang et al, 2023 , 2024b ) datasets. Unger et al ( 2024 ) proposed a Transformer-based deep learning architecture, called RETINA, to predict a user's product preference from raw eye-movement data.…”

Section: Discussionmentioning

confidence: 99%

A review of machine learning in scanpath analysis for passive gaze-based interaction

Mohamed Selim,

Barz,

Bhatti

et al. 2024

Front. Artif. Intell.

View full text Add to dashboard Cite

The scanpath is an important concept in eye tracking. It refers to a person's eye movements over a period of time, commonly represented as a series of alternating fixations and saccades. Machine learning has been increasingly used for the automatic interpretation of scanpaths over the past few years, particularly in research on passive gaze-based interaction, i.e., interfaces that implicitly observe and interpret human eye movements, with the goal of improving the interaction. This literature review investigates research on machine learning applications in scanpath analysis for passive gaze-based interaction between 2012 and 2022, starting from 2,425 publications and focussing on 77 publications. We provide insights on research domains and common learning tasks in passive gaze-based interaction and present common machine learning practices from data collection and preparation to model selection and evaluation. We discuss commonly followed practices and identify gaps and challenges, especially concerning emerging machine learning topics, to guide future research in the field.

show abstract

Section: Discussionmentioning

confidence: 99%

A review of machine learning in scanpath analysis for passive gaze-based interaction

Mohamed Selim,

Barz,

Bhatti

et al. 2024

Front. Artif. Intell.

View full text Add to dashboard Cite

show abstract

“…An alternative is to repurpose and augment existing applications. For example, existing UIs can be retargeted to support new modalities [46] or support responsive resizing [21,52]. Interaction proxies apply runtime modification to existing mobile apps by "repairing" inaccessible or difficult-to-use UI elements [57].…”

Section: Improving Existing Applicationsmentioning

confidence: 99%

“…First, the set of constraints that we automatically extract should be extended to support those available in conventional UI authoring tools, such as vertical and horizontal alignment guides, distributing vertical and horizontal spacing, and resizing elements across axes. Including these constraints and inference techniques [6,21,24,29] in Reflow would further minimize layout disruptions. Currently, Reflow internally represents UIs as a list of bounding boxes, but the limitations of this approach are that it is unable to capture semantic relationships between elements during inference.…”

Section: Opportunities For Reflowmentioning

confidence: 99%

Reflow: Automatically Improving Touch Interactions in Mobile Applications through Pixel-based Refinements

Wu¹,

Barik²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Touch is an ubiquitous way of interacting with smartphones. However, building mobile user interfaces (UI) where touch interactions work well for all users is a difficult problem, because users have different motor abilities, skills, and even preferences [51]. For example, consider a right-handed user who wants to access a menu item on the left side of the screen. For larger screens, this menu item is more difficult for the user to touch with their right hand than for a left-handed user.UI adaptivity is a promising approach towards improving touch interactions, because it allows systems to dynamically personalize the UI and tailor the UI to the users' needs. But for UI adaptivity to be practically useful for real-world apps, it must support two goals. First, the technique should be generally useful across a broad range of existing mobile applications. Second, the technique should apply adaptations in a way that respects the design intentions of the original applications. In other words, we expect that drastic UI adaptations are likely to make the user interface less familiar to the user and disruptive to the overall user experience [7].To operationalize these goals, we built a system, Reflow, which automatically applies small UI adaptations-called refinements-to mobile existing app screens to improve touch efficiency. Towards the first goal of supporting broad applicability, Reflow is entirely pixel-based: the system does not need knowledge of the applications' dependencies or view hierarchy to make its UI adaptations. Towards the second goal of respecting design intent and minimally disrupting the user experience, Reflow incorporates the theory of microstrategies in its model [12,13,17]. Microstrategies suggest that even small, principled adaptations to the user interface can significantly improve task efficiency-particularly over cumulative usage-and we postulate that the same principle applies when personalizing touch-based mobile applications.Reflow supports personalized optimization by constructing a spatial map from usage data, which identifies difficultto-access areas of the screen (e.g., elements on edges of the screen requires users to reach and reposition their hand to select). Reflow then (i) automatically detects the UI elements contained on the screen, (ii) uses a machine learning model to optimize the UI layout to better support the difficulty map, and then (iii) re-renders the existing UI pixels to match the new layout (Figure 1). Reflow improves on existing approaches because it works with a range of existing mobile applications and enables an end-to-end pipeline from layout optimization to re-rendering the application screens.To evaluate Reflow, we first conducted a study with 10 participants, where we found it improved interaction speed by 9% on average, and improved interaction speeds by up to 17% for some UIs. From lessons learned, we made further improvements to this model by detecting and applying an additional set of UI constraints (relative positioning, alignment).We then conducted a heuris...

show abstract

GTLayout: Learning General Trees for Structured Grid Layout Generation

Xu,

Shi,

et al. 2024

Lecture Notes in Computer Science

View full text Add to dashboard Cite

ReverseORC: Reverse Engineering of Resizable User Interface Layouts with OR-Constraints

Cited by 11 publications

References 81 publications

A review of machine learning in scanpath analysis for passive gaze-based interaction

A review of machine learning in scanpath analysis for passive gaze-based interaction

Reflow: Automatically Improving Touch Interactions in Mobile Applications through Pixel-based Refinements

GTLayout: Learning General Trees for Structured Grid Layout Generation

Contact Info

Product

Resources

About