Background & Summary

Human-machine interfaces (HMIs) serve as platforms for cognition and communication between humans and machines. They play a crucial role in transmitting information1. The study of User Experience (UX) in this field aims to enhance usability, performance, and overall user satisfaction by considering user motivation goals during interactions. According to Aranburu et al.2, users are motivated by a sense of autonomy, competence, and their emotional connection to the system. To enhance the UX, manufacturing companies should consider factors related to user motivation, such as intuitive and efficient task completion and the system’s anticipation of user needs. The latter is closely tied to the cognitive capacity of the user, which translates into the amount of memory required for the user to meet their goals3. Manufacturers can take advantage of the analysis of user interactions through artificial intelligence techniques that can respond to user motivators.

This paper presents a dataset that includes sequences of human-machine interaction with several machine elements to perform a particular process. Sequential behavior can indicate the level of expertise or familiarity with a given device or machine. In the literature, multiple authors have studied user interactions to understand user tasks and goals in a wide range of fields. For instance, in engineering design, McComb et al.4 demonstrated how to assist designers with previous design iterations using sequence learning. On the other hand, methods to analyze human-machine interactions exploit machine learning techniques. Recommendation systems can provide adaptation and personalization for users, which can help reduce the cognitive load and increase the user’s positive feelings3,5. For instance, the use of deep recurrent neural networks and collaborative filtering algorithms can perform sequential recommendation of content and control elements6. Furthermore, the analysis of interaction sequences benefits not only end-users directly but also assists UI evaluators in providing suggestions to improve the evaluated interface7. However, despite the clear contributions of these studies, the datasets are typically unavailable to practitioners and scholars.

The dataset presented in this article was collected with the primary aim to enable the development of Adaptive User Interfaces (AUIs) that can deliver benefits across a range of industrial applications. AUIs are artifacts designed to dynamically adjust to operator interaction patterns8. This means that any repetitive set of actions performed by an operator to complete a task can be considered an interaction pattern that can be automated9. Leading to more personalized interfaces that enhance productivity10. Hence, improving usability and UX overall. The dataset can also be applied to a wider range of use cases, including sequence-aware and context-aware applications that utilize data generated by devices and users to provide customized experiences and self-adaptable designs. By leveraging multiple sources of context, such as task, time and user context, these applications can provide personalized interactions that increase user productivity11. Overall, this dataset has the potential to support the development of adaptive systems that better meet the needs of users.

Methods

This section describes the data collection process. It starts by describing the design of the experiment and the setup, including a description of the acquisition and processing elements of the methodology.

Experimental set up

The experiment was conducted using a machine in which multiple operators interacted through the same HMI to perform a mixture creation task. In this scenario, an industrial mixing machine from the food sector was utilized, which offers the advantage of being regularly used throughout the day by several users across two working shifts. Each time a mixture was ordered, the operator carried out a series of individual interactions with the HMI. These interactions were related to adjusting various parameters, including additive quantity, mixture type, and the use of containers. These parameters directly influenced the properties of the final product.

Users interacted with the machine through a mobile app that was specifically designed for the experiment. Operators accessed the app by scanning a QR code, after which they proceeded to select the required mixture. The captured interactions included two key components: (i) the order and sequence of steps the user followed, and (ii) the time interval in which the user interacted with the machine.

Participants

Twenty-seven volunteer operators, aged between 23 and 45 years, participated in the experiment. Each operator granted formal consent to have their daily interactions recorded through the app. In total, 10,608 interactions were captured over a period of 151 days. All data was anonymized and does not contain sensitive user information.

Data acquisition methodology

Figure 1 illustrates the methodology for data acquisition, which begins with the preparation stage. This stage encompasses two steps: firstly, the user interface (UI) is formally described using a user interface description language (UIDL), which consists of a mark-up language that describes the entire HMI12. In this study, the JSON format was employed to represent each visual element in the HMI, with each element assigned a unique alphanumeric identifier.To provide an example of the UIDL utilized in this study, Fig. 2 displays a representation of the UI alongside its corresponding UIDL.

Fig. 1
figure 1

Data acquisition methodology.

Full size image
Fig. 2
figure 2

UIDL JSON description example of a UI.

Full size image

The HMI was implemented using Next.js, a React framework and Chakra UI. A dedicated function was created to programmatically generate the HMI using the user interface descriptor. The interface is designed to be responsive and can be used on tactile devices.

Next, the interaction process representation required to prepare a mixture in the machine is described as a Finite State Machine (FSM), which is a model consisting of states, transitions, and inputs used to represent processes or systems. In this process, the user adjusts the parameters of a mixture until the values are considered correct (Fig. 3).

Fig. 3
figure 3

Interaction process representation (FSM).

Full size image

During the active phase of the experiment, when users access the machine using the application, a non-intrusive layer captures the interactions and stores them in a database (capture interactions). The information captured includes the user identity, the timestamp of the interaction in EPOCH format, and the identification of the interacted element (store raw interactions) (see Table 1). Once this information is collected, the data processing step generates the sequences.

Table 1 Example of raw operators interactions collected by the system.
Full size table

Data processing

The goal of this step of the methodology is to generate valid sequences of interactions for each user. Perer & Wang13 define a sequence of events \(E=\langle {e}_{1},{e}_{2},...,{e}_{m}\rangle \) (ei ∈ D) as an ordered list of events ei, where D is a set of events known and the order is defined by i. This means that the event ei occurs before the event ei+1. Additionally, in this process is considered that E must contain at least two events e to be accepted as a sequence9.

Using this definition and taking as input the raw interactions, it is possible to define valid interaction sequences as \({s}_{i}=\left[{e}_{begin},{e}_{1}^{i},\ldots ,{e}_{k}^{i},{e}_{end}\right]\) where si is a set of events and:

  • The events ebegin and eend are known, determining the beginning and the ending of the interaction sequence

  • The variable l determines the length of the interaction sequence and its value should be > = 2

The sequences are extracted using the “Valid sequences extractor” algorithm presented by Reguera-Bakhache et al.9. As demonstrated in the FSM (Fig. 3), the interaction process initializes when an interaction occurs in any of the elements that allow the parametrization of the mixture and finalizes when the user clicks the button BTN1OK.

From the 10,608 interactions recorded, 1358 valid sequence interactions were generated. The composition of each interaction sequence is described in the following section.

Data Records

The files can be downloaded from the Figshare14 data platform in a CSV format. The individual files are described below.

  • Raw interactions (raw_interactions.csv): Raw interactions represent each event performed by an user when interacting with an element of the machine, with their corresponding timestamp (see Table 1).

    • Users(users.csv): This file presents information about the profile of users that participated in the study. Table 2 presents the distribution of the participants based on age, gender and role.

      Table 2 Participants distribution.
      Full size table
      Table 3 User interactions sequences descriptor.
      Full size table

  • User Interaction sequences (sequences_df_prep_EN.csv): Each row of this CSV file represents a valid interaction sequence to perform a task. This dataset was generated from the raw interactions following the “data processing” method previously described. Table 3 presents a description of the fields, classifying each field by the type of information they deliver, including some UX metrics relevant to the efficiency and effectiveness of the process. These represent more pragmatic aspects that focus on the task-oriented nature of an experience15.

  • User interface description (ui.json & hmi_elements.csv): The ui.json file serves as a JSON-formatted representation of every visual element within the study’s user interface. These elements are dynamically instantiated based on their component type. Table 4 provides a comprehensive overview of the various component types used, with the “interactive” column indicating which components are clickable for user interaction. To render the UI on the web application, we created a function in the app that reads and interprets the contents of the ui.json file. This allows to easily modify and update the UI as needed, without requiring significant changes to the underlying code (see sec:Code availability). The hmi_elements.csv is an informative file that lists the employed elements with their respective element IDs and component types.

    Table 4 Type of components referenced on ui.json.
    Full size table

Technical Validation

Sequential analysis can serve as an indicator of human expertise and behavior when performing both highly specialized tasks and common ones. To achieve this, the data must be consistent, complete, and preprocessed. For the latter, it was necessary to clean the data and ensure that its suitability for the intended use. This involved tasks such as removing duplicate interactions and correcting errors. Subsequently, starting from the raw interactions, the next step was to generate valid sequences, a process depicted in the subsection Data processing.

To validate the resulting dataset (sequences_df_prep_EN.csv) and facilitate the reuse, a data profiling analysis was conducted. First, it was important to guarantee the consistency, which could be defined as data presented in the same standard structure and its correctness in relationship with other data16. Therefore, it was validated that the elements within the interaction sequences correspond to the elements on the UI JSON file, and the users’ IDs exist in the users.csv file.

Other general aspects were evaluated using the IBM API of Data Quality for AI17, this toolkit offers a range of quality estimation and data profiling metrics to assess the quality of ingested data in an objective and systematic manner. These metrics produce a score between 0 and 1 that quantifies the presence of data issues, with a score of 1 indicating that no problems were detected. These metrics are designed for tabular datasets and accept input in the form of comma-separated values files17. The main used metrics are described in Table 5.

Table 5 Data Quality metrics score.
Full size table

The distribution of sequences was analyzed across different services, users, and time periods. Figure 4a, reflects that only a few services on the machine reach the maximum number of interactions, and the overall average of sequences per machine mixture is 84.85, whereas the median is 34.5. The data sparsity is an expected issue in applications that analyze user behavior, in which each user only interacts within a small set of items (i.e. recommendation systems)18. Similarly, in Fig. 4b there are users that engaged more with the application and record a higher number of interactions, having an average of 50.25 interaction sequences per user. These aspects must not be seen as a limitation in the utilization of the dataset but rather be a factor for data scientists to consider at the moment of developing and testing their models. Additional sources of contextual information regarding the interactions include the user roles and time periods. As the experiment was centered on machine usage, it was observed that the “operator” role accounted for the largest number of interactions (as illustrated in Fig. 4c), while weekdays exhibited a more evenly distributed pattern (Fig. 4d).

Fig. 4
figure 4

Sequences distribution (a) Distribution of Sequences Across Machine Services (b) Distribution of Sequences Across Users (c) Distribution of Sequences Across User Roles (d) Distribution of Sequences Across Weekdays.

Full size image

Density-based clustering algorithms can be used to determine the hourly distribution over time for operators. This is a type of unsupervised learning technique that identifies different clusters based on the density of the points. Each detected cluster in a 1-Dimensional space determines the time interval where the interaction is most frequent.

To detect the clusters, first, we represented on a time axis each sequence from sequences_df_prep_EN.csv over time with a point. Second, MeanShift Algorithm19 is used to detect clusters and discard outliers. Figure 5a illustrates in a 1-Dimensional space the distribution of the interaction sequences from a single operator over time.

Fig. 5
figure 5

Distribution over time of interaction sequences (a) Distribution over time of interaction sequences from a single operator. (b) Identified clusters after applying Meanshift Algorithm for time interval detection.

Full size image

Figure 5b shows different clusters detected by Meanshift Algorithm for previous sequences. For each cluster, the leftmost point determines the beginning of the time interval, and the rightmost point the end of the interval. Each of these clusters represents the time intervals where the interactions are more frequent.

Overall, the analysis of the dataset can provide valuable insights into user behavior and usage patterns that can aid in the development of recommendation systems, adaptive user interfaces, or other applications. The insights obtained from analyzing the distribution of sequences across different services, users, and time periods can assist data scientists in the usage of the dataset to consider these factors.

Usage Notes

Generating Adaptive HMI

When designing AUIs, several key dimensions should be taken into consideration. Oestrich et al.20 presented a morphological box that structures these dimensions. In Table 6, the aspects are related to the data available on the dataset to highlight reusability.

Table 6 Mapping adaptive user interface dimensions to dataset fields.
Full size table

However, defining the adaptation goal is key, whether it focuses on enhancing performance or providing instructional support, as it guides the entire process. The goal will define the techniques that can be used. From our previous works10, the analysis of the clickstream sequences (interactionwu_prep field) aimed to provide performance support by identifying recurring operator-machine interaction patterns and automatically detecting the time intervals during which these patterns manifest most frequently. This, in turn, enables the creation of “time-based” adaptations through the generation of Event Condition Action (ECA) rules. Additionally, these adaptations extend to altering the “presentation” of interface elements, specifically the layout order automatically. This is achieved through a Python-based engine that adjusts the user interface descriptor according to the rules. Notably, our research findings have demonstrated a reduction of over 40% in operator interaction time, showcasing the practical benefits of AUIs in enhancing efficiency and productivity. Facilitating the design process, we have employed a straightforward user interface descriptor in JSON format. This descriptor simplifies the modification process necessary for generating these adaptations, thus enhancing the usability and accessibility of AUI design. However, as indicated in Table 6, the adaptation target can vary, and the methods for presenting the adaptation can include overlaying cues or highlighting elements.

For instance, in order to provide learning support, one approach is to predict the next step based on past instructions and automatically execute these instructions. In such cases, the presentation of visual elements serves a communicative purpose. We have developed an example of next-step prediction using Markov Chains, which serves as a baseline approach for researchers interested in reusing this dataset. The associated code will be made available for use, facilitating further research and development in this area. However, the discussion of the results is out of the scope of this paper.

For further processing of the Human-Machine interactions dataset, we recommend well-known Python libraries such as Pandas and Scikit-learn.