Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
arXiv:2606.24781v1 [cs.AI] 23 Jun 2026
REBECCA ADAIMI, The University of Texas at Austin, USA EDISON THOMAZ, The University of Texas at Austin, USA While the field of Human Activity Recognition (HAR) continues to draw interest from researchers and advance in important ways, some key challenges remain. One of the most difficult aspects of building HAR models that show good performance in real-world settings is dealing with data diversity from device and sensor heterogeneity, and contextual changes that are intrinsic to real-world applications. While data diversity in HAR has been well-acknowledged in the literature, there remains a gap in understanding the effect of various types of distribution shifts on HAR models and the domain generalization problem that arises. Towards that end, this paper systematically evaluates 4 different types of distribution shifts, including variations in device type, sensor placement, sampling rate, and user behavior. Quantifying their effects, we illustrate that diversity shifts predominantly define all types of shifts, indicating the existence of unique features that are not shared across different domains. We then introduce a uniform HAR-based distribution shift benchmarks and conduct a comprehensive evaluation of up to 28 domain generalization methods. Our analysis exposes the limitations of current domain generalization algorithms in achieving model generalizability, marginally outperforming the empirical risk minimization baseline. This work represents the first systematic exploration of domain generalization and adaptation concerning specific distribution shifts in sensor-based HAR, offering an open-source benchmark platform and datasets to spur further research. Additional Key Words and Phrases: Distribution Shift, Domain Generalization, Human Activity Recognition
1
INTRODUCTION
Over the past decade, thanks to advances in smartphones, sensors and powerful machine learning methods, interest in the field of Human Activity Recognition (HAR) has gained significant strength. This growing area of research has been driven by a plethora of opportunities in mobile health [16, 48], disorder diagnosis [9], personal assistance, smart environments [1], and numerous other domains. HAR, once primarily limited to controlled laboratory environments, has now transcended these boundaries to operate in the complex and dynamic contexts of our everyday routines. However, despite much progress, a vast majority of HAR methods is still largely based on the assumption that training and test data used for modeling have similar distributions. This assumption is often not valid in real-world applications due to ubiquitous distribution shifts introduced by variations in device types, sensor placements, and contextual environments. As a result, state-of-the-art methods continue to face challenges in bridging the gap between controlled validation conditions and the intricate complexities of the real world [6]. From a machine learning perspective, the challenge of data diversity in HAR models can be formulated as a domain shift problem. To tackle the aforementioned challenges, it is crucial to assess the generalizability of a model on novel data. An effective approach frequently employed to address domain shift issues is transfer learning or domain adaptation [19, 36, 59, 69], which involves utilizing knowledge acquired from other available domains during the training phase, and then fine-tuning the model on a new target domain to minimize the distribution gap and improve overall performance. Nevertheless, a primary assumption for such techniques is the availability or accessibility of the target data, which is often unfeasible in practical applications. This is particularly true in the realm of sensor-based activity data, where it is not practical to amass sensor data for all potential distribution changes. Thus, domain generalization is a research topic that focuses on the Authors’ addresses: Rebecca Adaimi, [email protected], The University of Texas at Austin, , Austin, Texas, USA, ; Edison Thomaz, [email protected], The University of Texas at Austin, , Austin, Texas, USA, .
2
Adaimi, et al.
problem of building a model that generalizes well to unseen data without access to the new data during training. As noted earlier, the challenge of generalization in human activity recognition (HAR) can arise from various factors, such as changes in device type, sensor heterogeneity in terms of sampling rate or sensor placement, environments, and activity patterns, among others. While these challenges have been recognized in previous research, it is still unclear to the scientific community which of these factors specifically makes domain generalization in HAR particularly difficult. This paper takes the initial step in conducting a methodical evaluation of the generalizability of HAR models across these distribution changes. In our investigation of the problem space, we structured our analysis to address two primary research questions: 1. Quantifying Distribution Shifts. Focusing on sensor-based HAR data, what are the different causes of distribution shifts, such as sensor-related changes (e.g. device type, sensor placement and sampling rate) and user-related changes (e.g. skill improvement)? And can we represent and quantify each shift to better understand its influence on the data? Taking inspiration from the work of Ye et al. [65], we quantify each distribution shift using a two-dimensional spectrum, which includes measurements of diversity shift and correlation shift. Our analysis uncovered that all types of shifts and datasets are primarily characterized by diversity shifts, indicating the existence of unique features that are not shared across different domains. 2. Domain Generalization. From a domain generalization perspective, how do state-of-theart domain generalization methods perform in the context of these sensor-based distribution shifts? To answer this question, we developed a uniform benchmark platform1 and evaluated up to 28 domain generalization methods on 4 types of distribution shifts. Our results show that existing algorithms only marginally improve performance over the empirical risk minimization baseline, emphasizing the challenge of domain generalization across sensorbased sensing heterogeneity and distribution shifts. To the best of our knowledge, our research is the first to investigate domain generalization and adaptation in the context of specific distribution shifts relevant to sensor-based HAR, with a focus on understanding the challenges each shift poses to model generalizability. It should be noted that our primary objective is not to critique or highlight any particular existing methods, but rather to (1) underscore and demonstrate the significance and deficiency of model generalizability when it comes to sensor-based human activity and (2) present a benchmark platform and datasets for future research. 2
RELATED WORK
In this section, we start by providing a brief overview of the recent progress in the area of domain adaptation, with a specific emphasis on HAR. Subsequently, we redirect our focus towards the issue of domain generalization, which serves as the main focus of this paper. 2.1
Domain Adaptation
The study of domain adaptation in HAR, also known as cross-domain human activity recognition, is a complex issue that has been explored with different goals. Experts have acknowledged the difficulties involved in transferring knowledge from HAR models that are trained on data from one 1 www.to-be-added.com
Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
3
domain to another [43]. Each research endeavor focuses on various sub-problems related to crossdomain HAR, such as cross-sensor-modalities [36], cross-location [59], and cross-environment [15]. In all of these problems, the primary aim is to modify the model during training so that it can adapt to the target domain and decrease the divergence in distribution. An effective and commonly used approach is the pretrain-then-finetune paradigm, also known as transfer learning [36]. The idea is using some data from the target domain, the weights of a pretrained model can be finetuned to reduce the distribution shift. This adaptation process has been explored in both supervised [33] and unsupervised settings [11]. HDCNN applied feature matching at each layer of the neural network, reducing the Kullback-Leibler divergence and enabling adaptation of the model to a smartwatch using unlabeled data [23]. Chen et al. [13] proposed, MotionTransformer, which utilized a confusion maximization approach to translate data from any wearable sensor to match the data collected from a smartphone. Similarly, Akbari et al.[3] utilized variational autoencoders with feature matching to minimize the distribution gap between labeled source and unlabeled target datasets. As mentioned previously, these techniques operate under the presumption that the target data, either labeled or unlabeled, is present during the training phase. However, this is not always the case in real-world situations. Domain generalization is a learning task that seeks to create a generalizable model capable of generalizing to new domains without prior access, thereby addressing this limitation. 2.2
Domain Generalization
The task of building a model that can generalize across multiple domains is a longstanding challenge in the field of machine learning. Recently, several algorithms have been proposed to address this issue, which can be broadly categorized into three groups: (1) data manipulation that utilize data augmentation techniques to support model generalization [67], (2) representation learning that aim to develop a generalized feature space [5, 18], and (3) learning strategy that focuses on training procedures [28, 56, 64]. This problem has been extensively studied in other domains such as computer vision [22, 30, 32, 45, 60] and natural language processing [31, 38]. In order to facilitate research in this field, several benchmark platforms have been developed, such as DomainBed [20], DeepDG [58], and WILDS [25]. While domain generalization has been extensively studied in the context of image-based tasks, its application to sensor-based data has only recently begun to garner attention. These works include studies on cross-dataset generalization [61], cross-user generalization [41], and cross-position generalization [35]. For instance, Qin et al. proposed an adaptive feature fusion approach that learns both domain-invariant and domain-specific features to build a generalizable model across users [41]. Another work focused on cross-dataset generalization for depression detection using two multi-year longitudinal behavior datasets [61]. They additionally introduce a benchmark platform, GLOBEM, to support future research in this field. Nonetheless, there remains a gap in our understanding of how effectively these algorithms can handle the various types of heterogeneities present in sensor-based data. With its inherent complexity and wide range of possible distribution shifts, HAR can serve as another ideal platform to study the domain generalization abilities of learning algorithms. 3 CAUSES OF DISTRIBUTION SHIFTS IN HAR Differences in wearable sensing devices can cause changes in the distribution of data, potentially leading to reduced performance of a pre-existing model. These changes can occur due to multiple factors, including but not limited to the type of device used, the location of the sensor, the sampling rate of data collection, and variations in user behavior (Figure 1).
4
Adaimi, et al.
(a) Sensor Location
(b) Device Type
(c) User Behavior
(d) Sampling Rate
Fig. 1. Accelerometer signals captured with varying (a) sensor location (wrist, ankle, chest), (b) Device type (LG Nexus 4 and Samsung Galaxy S3 mini), (c) User Behavior or skill change (juggling 3, 5, and 7 balls), and (d) Sampling Rate (50Hz and 200Hz).
(1) Device Type. Variations in sensor biases across devices can result in changes to the data distribution. For instance, the data collected from accelerometers on different devices such as a Samsung Galaxy S3 mini and LG Nexus 4 can display significant biases and variations, as depicted in Figure 1b. We can see slight differences in the accelerometer range between the two devices owing to the differences in the inertial sensors integrated. (2) Sensor Location. Mobile and wearable devices can be worn or carried by users in various ways due to differences in form factors and personal preferences. For instance, a smartphone can be held in hand or carried in a pocket. As Figure 1a illustrates, the accelerometer data obtained from wearable devices across three different body locations exhibit significant divergence, indicating a domain shift due to variations in sensor location. We can clearly see differences in the accelerometer range as well as the walking pattern captured from each body location. (3) Sampling Rate. The maximum sampling rate that a mobile device can support can vary significantly across device models. For instance, while some devices can only attain a maximum sampling rate of 50 Hz, others can reach up to 200 Hz [51]. This disparity can result in a mismatch in sampling rate between the training and test data, which can adversely affect the performance of an activity recognition model. As shown in Figure 1d, for a fixed window size, the data captured at 50Hz looks widely different from data captured at 200Hz.
Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
5
Table 1. Benchmark Datasets Overview Distribution Shift
Sensor Location
Dataset
Sampling Rate
Activity
PAMAP2 [44]
3
(wrist, ankle, chest)
100 Hz
5
DSADS [4]
5 (torso, right arm, left arm, right leg, left leg)
25 Hz
5
30 Hz
5
Opportunity [12]
Sampling Rate
Domains
6 (right wrist, right knee, left upper arm, left shoe, hip, back)
PAMAP2
3
(10Hz, 50Hz, 100Hz)
100 Hz
5
RealWorld [54]
3
(10Hz, 25Hz, 50Hz)
50 Hz
8
HHAR Nexus4 [51]
4 (10Hz, 50Hz, 100Hz, 200Hz)
200 Hz
6
50-200 Hz
6
100 Hz
4
Device Type
HHAR [51]
User Behavior
Juggling
4 (Nexus4, Galaxy S3, Galaxy S3 mini, Galaxy S Plus) 4 (juggling 3, 4, 5, and 7 balls)
(4) User Behavior. Changes in a user’s behavior can also result in distribution shifts. Firstly, the same activity can be performed slightly differently by different users, leading to variations in the data. Moreover, the way a user performs an activity can change over time, particularly in the case of skill-learning tasks. For instance, if a user is learning to juggle, they may start with juggling two balls and then progress to juggling more balls as their skill improves. This progression results in changes in the way the juggling activity is performed over time, as shown in Figure 1c, which illustrates the change in the accelerometer data when a user juggles 3 balls compared to juggling 5 and 7 balls, in terms of accelerometer range as well as juggling pattern. As expected, the juggling hand movements are faster when juggling 7 balls compared to 3. 4
BENCHMARK EXPERIMENTS
In this section, we introduce a set of HAR-related domain generalization benchmarks representing each type of distribution shift as listed in Section 3. We leverage one or several widely-used publicly available datasets for each of the 3 distribution shifts: (1) sensor location, (2) sampling rate, (3) device type, and collect a new dataset simulating skill improvement for the user behavior shift. As such, we begin by giving a brief description of the datasets used and then explain the steps taken to prepare the datasets for each distribution shift scenario. When selecting the datasets to accompany our analysis, we focused on datasets (see Table 1) that include at least one of the distribution shifts described in Section 3 to create a benchmark for domain generalization. If a public dataset was not readily available, we collected our own dataset to complete the analysis. 4.1
Public HAR Datasets
4.1.1 PAMAP2. Physical Activity Monitoring dataset [44] contains 18 physical activities captured from 9 subjects sampled at 100 Hz. Each subject wears 3 inertial measurement units (IMU) placed at 3 distinct locations (wrist, ankle, chest) and a heart rate monitor. We use the 3-axis accelerometer
6
Adaimi, et al.
data from each IMU. We focus on common full-body activities, and as such cluster activities into: (1) idle (lying, sitting, standing), (2) walking (walking, ascending/descending stairs, nordic walking), (3) running, (4) cycling, and (5) other (vacuuming, ironing, rope jumping, watching tv, folding laundry, house clearning, playing soccer, car driving, computer work). 4.1.2 DSADS. Daily and Sports Activities Dataset [4] collects 19 activities from 8 subjects. Each subject wears 5 on-body sensors that capture triaxial accelerometer, gyroscope, and magnetometer sampled at 25 Hz from 5 locations: torso, right arm, left arm, right leg, and left leg. Similar to PAMAP2, we cluster the activities to form a set of 5 common activities: (1) idle (sitting, standing, lying on back/right, standing in elevator), (1) walking (ascending/descending stairs, moving in elevator, walking in parking lot/treadmill flat/treadmill at incline, (3) running, (4) cycling (cycling vertical/horizontal), (5) other (stepper, cross trainer, jumping, rowing, basketbatll). 4.1.3 Opportunity. The Opportunity dataset [12] consists of motion data collected from 16 bodyworn sensors sampling at 30 Hz worn by 4 individuals while performing 5 typical daily activities: (1) standing, (2) walking, (3) sitting, (4) lying. (5) other. However, the dataset contains several repetitive sensor locations. For the purpose of defining distinct domains in our sensor location analysis, we focus on 6 key locations: right wrist, right knee, left upper arm, left shoe, hip, and back. 4.1.4 RealWorld. The RealWorld dataset [54] contains accelerometer and gyroscope data from 15 participants sampled at 50 Hz. Data is captured from 7 on-body sensors placed at forearm, thigh, head, upper arm, waist, chest, and shin. Each participant performed 8 activities: (1) climbing stairs down, (2) climbing stairs up, (3) jumping, (4) lying, (5) standing , (6) sitting, (7) running, (8) walking. 4.1.5 HHAR. Heterogeneity Human Activity Recognition dataset [51] covers locomotion activities with accelerometer data collected at the wrist and waist using a wide range of 2 smartwatches (LG G and Samsung Galaxy Gear) and 4 smartphones (LG Nexus 4, Samsung Galaxy S3, Samsung Galaxy S3 mini, and Samsung Galaxy S Plus) respectively. Each device yielded different maximum sampling rates, ranging from 50 Hz to 200 Hz. 4.2
Data Collection: Juggling Dataset
So far, the public datasets discussed in the previous section cover at least one of the distribution shifts, mainly changes in sensor location, sampling rate, device type, and context environment. An important remaining change that is especially relevant to HAR is change in user behavior. Towards addressing this aspect, we design a data collection study with the goal of capturing a change in user behavior for a certain activity, such as in the case of learning a new skill. Simplifying the problem space, we choose an activity that (1) can be captured with an off-the-shelf smartwatch equipped with accelerometer sensor and (2) can be easily assessed in terms of skill. 4.2.1 Study Protocol. With that in mind, an IRB-approved juggling study was conducted with 9 participants (8 males and 1 female) wearing an Apple Watch on each wrist. The participants were recruited from a local juggling society that includes people from various backgrounds, age (mean 46.1 ± 19.9), and juggling skill level. The study lasted roughly 30 minutes. We asked participants to wear a watch on each wrist. At the beginning of the data collection study, each participant was asked to complete a survey to collect demographics information as well as a self-reported juggling skill assessment using a 1-5 Likert-Scale question. On average, participant’s self-reported skill was 3.4 with lowest score being 2 and highest being 5. The study was split into three 5-min juggling sessions and four 1-min non-juggling sessions in between. For all participants, the first juggling session entailed juggling 3 balls. Depending on their skill level and the maximum number of balls they are able to juggle, we increased the number of
Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
7
Fig. 2. Screenshots from video recorded sessions of a participant juggling 5 and 7 balls.
balls juggled in the following sessions. By including sessions with varying numbers of balls, we aimed to capture the progression of skill improvement in juggling, as users gradually increased their ability to juggle more balls over time. We specifically captured juggling activities with 3, 4, 5, and 7 balls. The number of balls defined in each session for each participant varied based on the participant’s skill level. For instance, only one participant was able to juggle 7 balls. For non-juggling activities, participants were asked to perform a set of 4 miscellaneous activities (one minute each) randomly picked from a set of 7 activities including (1) idle activities (standing or sitting), (2) walking, (3) jogging, (4) writing, (5) typing on keyboard, and (6) jumping jacks. Accelerometer data was captured continuously throughout a session at 100 Hz sampling rate using the SensorLog App [55]. The study session was video recorded to aid in data annotation at the end of the study. Figure 2 depicts example recorded frames of participants juggling 3 and 7 balls. The accelerometer data from both watches was synchronized along with the video recording using the ELAN annotation tool [8]. 4.3 Benchmarks To investigate the different types of distributions and evaluate the performance of domain generalization algorithms, benchmarks are created for each type of distribution shift based on the characteristics of each dataset. Each setting or change in the sensor data is considered as a domain, and each dataset is preprocessed and prepared to contain a set of 𝑘 domains with 𝑘 > 2 that follow the corresponding distribution shift. Table 1 summarizes the datasets and domains studied for each type of distribution shift. In order to simplify the analysis and have more control over the different types of shifts, we focus on one modality for all datasets, which is the most common and most power efficient sensor–the accelerometer. For all datasets, the 3-axis accelerometer data is first normalized along each axis using standard scaling and then segmented using a fixed sliding window of 100 samples with 50% overlap. These benchmarks are used for shift quantification and
8
Adaimi, et al.
comprehensive performance analysis of the domain generalization algorithms listed in Section 6.2. We run each experiment 5 times with a random seed and report the average results. 4.3.1 Sensor Location. In the sensor location analysis, one of the main criteria for dataset selection was the availability of multiple on-body sensor locations capturing synchronized inertial data for common locomotion activities. Based on this criterion, four large datasets were chosen: PAMAP2, DSADS, Opportunity, and RealWorld. Each dataset was divided into separate domains, with each domain representing a specific sensor location. This was done to construct a domain-generalization experiment where the sensor location serves as the domain. Specifically, PAMAP2 was divided into three domains, DSADS into five domains, Opportunity into six domains, and RealWorld into seven domains. 4.3.2 Sampling Rate. The aim of this domain generalization analysis is to assess the model’s ability to generalize to data captured at a different sampling rate from the training data. To ensure that other variables remain consistent, we adopt 3 datasets with high enough sampling rate and simulate the sampling rate changes by downsampling the data. More specifically, we leverage (1) PAMAP2, originally sampled at 100 Hz, and create 3 domains by downsampling to [10Hz, 50Hz]; (2) RealWorld, captured at 50 Hz, and downsample to [10Hz, 25Hz]; and (3) HHAR data captured by LG Nexus4 which was sampled at 200 Hz and downsample it to [10Hz, 50Hz, 100Hz]. Thus, this results in 3,3, and 4 domains for each of the datasets respectively. To reduce data leakage between training and testing data, a stratified k-fold split is applied to the original dataset, with k set to the number of domains in each dataset. Each fold is then preprocessed to correspond to a sampling rate. 4.3.3 Device Type. To explore the domain generalization problem related to sensor heterogeneity across devices, we utilized the HHAR dataset, which consists of locomotion accelerometer data captured from various off-the-shelf smartphones (LG Nexus 4, Samsung Galaxy S3, Samsung Galaxy S3 mini, and Samsung Galaxy S Plus). Each smartphone is equipped with different inertial sensors and samples data at different rates. In this analysis, our main focus was to investigate the impact of sensor biases on data distribution while maintaining other variables constant. To achieve this, we downsampled the data from each device to the lowest sampling rate of 50 Hz. Consequently, data from each device was treated as a distinct domain, resulting in a total of four domains in the HHAR dataset. The objective of this analysis was to examine the generalization capability of models across different devices, which is crucial for the widespread implementation of HAR, given the inherent heterogeneity of devices in real-world scenarios. 4.3.4 User Behavior. Changes in a user’s behavior have been shown to cause distribution shifts. To investigate this aspect, we focused on the scenario of skill improvement, where a user aims to enhance their proficiency in a particular activity, and as such the way they perform the same activity may change over time as their skill level improves. To simulate this setting, we collected a dataset specifically for juggling, capturing juggling activities performed by participants at various skill levels, as explained in Section 4.2. In this context, we defined domains based on the number of balls being juggled. The underlying assumption was that as users progress in their juggling skills, they would be able to handle an increasing number of balls. To clarify, we divided the juggling activity in the dataset into four domains, with each domain corresponding to a specific number of juggling balls (3, 4, 5, and 7). In order to develop a model that can accurately distinguish juggling from other activities, we applied a stratified four-fold split to the non-juggling activities and included them in each domain dataset. This ensures that each domain dataset contains both juggling and non-juggling activities. A generalizable model should be able to learn the distinguishing features of juggling irrespective of the number of balls.
Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
5
9
QUANTIFYING DISTRIBUTION SHIFTS
In general, researchers have defined changes in sensor-based data as a general distribution shift that affects data and, as a result, impacts the performance of a model. However, identifying the cause of the shift and understanding its influence on the data can provide a better understanding of the nature of the distribution shift and guide the selection of the optimal method to address it. Therefore, we aim to first represent and quantify each of the defined causes of distribution shifts in Section 3. Following the characterization by Ye et al. [65], we quantify each domain change as a two-dimensional characterization: diversity shift and correlation shift. 5.1
Diversity Shift and Correlation Shift
In plain words, the diversity dimension refers to out-of-distribution samples that are different from the training in-distribution data, while the correlation shift refers to a change in the correlation structure between the input and output variable. With those in mind, a distribution shift can be dominated by one type or a mixture of both, especially in a real-world settings. Following [65], the quantification formula for each of the diversity and correlation shifts between two data distributions 𝑝 and 𝑞 is given by, ∫ 1 |𝑝 (𝑧) − 𝑞(𝑧)|𝑑𝑧 𝐷𝑑𝑖𝑣 (𝑝, 𝑞) = 2 S ∫ √︁ ∑︁ 1 𝐷𝑐𝑜𝑟 (𝑝, 𝑞) = 𝑝 (𝑧) · 𝑞(𝑧) |𝑝 (𝑦|𝑧) − 𝑞(𝑦|𝑧)|𝑑𝑧 2 T 𝑦∈Y
where S and T represent two sets of features in Z where 𝑧 ∈ Z possesses the following property, 𝑝 (𝑧) · 𝑞(𝑧) = 0 ∨ ∃𝑦 ∈ Y : 𝑝 (𝑦|𝑧) ≠ 𝑞(𝑦|𝑧) and S and T , that are respectively responsible for each diversity and correlation shift, are defined as, S := {𝑧 ∈ Z|𝑝 (𝑧) · 𝑞(𝑧) = 0} T := {𝑧 ∈ Z|𝑝 (𝑧) · 𝑞(𝑧) ≠ 0} Practically, the metrics can be estimated by training a neural network to discriminate between different domains. Domains can refer to any type of change, whether sensor location, device type, environment etc. as discussed in Section 3. Thus, for example, a neural network is trained to recognize the sensor location the data is captured from. More specifically, to differentiate two datasets (𝑋𝐴 , 𝑌𝐴 ) and (𝑋𝐵 , 𝑌𝐵 ) with different data distributions 𝜉𝐴 and 𝜉 𝐵 respectively, a neural network is trained to distinguish between the two domains. The network is composed of a feature extractor 𝑔 : X → F and a domain classifier ℎ : F × Y → [0, 1], where F represents a lower dimensional representation of X. This results in a joint distribution over X × Y × F for each domain. Figure 3 illustrates the network framework used to extract features whose joint distribution with 𝑌 varies across different domains. 5.2
Implementation
Using the benchmark datasets as explained in Section 4.3, we apply 5 random stratified splits using domain target label, with 25% set as test data and 25% as validation. The network, with a feature dimension m = 16, is optimized using an Adam optimizer with a fixed learning rate of 0.00001 and a batch size of 128 for 100 epochs. The network is trained to optimize a cross-entropy loss. We choose the models maximizing the macro f1-score (in predicting the environments) on validation sets. For estimating the diversity and correlation shifts, an importance sampling size M equal to the smallest class size in the dataset, and we empirically set the thresholds 𝜖𝑑𝑖𝑣 = 1 × 10−12 and 𝜖𝑐𝑜𝑟 = 5 × 10−4 . We use Gaussian kernels for all the KDEs.
10
Adaimi, et al.
Fig. 3. Domain Classifier model overview. The neural network extracts features from the accelerometer data using a Vggish architecture. The domain classifier learns a joint representation of both input and output, which is then used for domain classification.
6
DOMAIN GENERALIZATION
From a machine learning perspective, the challenge of data diversity in HAR models can be formulated as a domain shift problem. To tackle the aforementioned challenges, it is crucial to assess the generalizability of a model on novel data. An effective approach frequently employed to address domain shift issues is transfer learning or domain adaptation [19, 36, 59, 69], which involves utilizing knowledge acquired from other available domains during the training phase, and then fine-tuning the model on a new target domain to minimize the distribution gap and improve overall performance. Nevertheless, a primary assumption for such techniques is the availability or accessibility of the target data, which is often unfeasible in practical applications. This is particularly true in the realm of sensor-based activity data, where it is not practical to amass sensor data for all potential distribution changes. Thus, domain generalization is a research topic that focuses on the problem of building a model that generalizes well to unseen data without access to the new data during training. As noted earlier, the challenge of generalization in HAR can arise from various factors, such as changes in device type, sensor heterogeneity in terms of sampling rate or sensor placement, and activity patterns, among others (see Section 3). While these challenges have been recognized in previous research, it is still unclear to the scientific community which of these factors specifically makes domain generalization in HAR particularly difficult. In this chapter, we take the initial step in conducting a methodical evaluation of the generalizability of HAR models across these distribution changes. There have been significant advancements in domain generalization methods within the machine learning community, particularly in the fields of computer vision and natural language processing [70]. However, the problem of domain generalization in HAR has received limited attention. To the best of our knowledge, our research is the first to investigate domain generalization in the context of specific distribution shifts relevant to sensor-based HAR, with a focus on understanding the challenges each shift poses to model generalizability. In particular, our main research question is: how effective are domain generalization algorithms in realistic HAR scenarios? To answer this question, we developed a uniform benchmark platform and evaluated up to 28 domain generalization methods on 4 types of distribution shifts.
Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
6.1
11
Problem Formulation
Let D = {𝐷 1, 𝐷 2, ..., 𝐷𝑘 } be a set of 𝑘 domains, where each domain 𝐷𝑖 is a joint distribution over feature space X and label space Y. In the HAR context, a domain corresponds to a particular setting or condition in which the sensor data was collected (e.g., different device types, sensor locations, or environments). The goal of domain generalization is to learn a model 𝑓𝜃 : X → Y that performs well on a new, unseen domain 𝐷𝑘+1 , without having access to any labeled data from that domain during training. Formally, given the set of 𝑘 domains D, a domain generalization algorithm seeks to find a model 𝜃 that minimizes the expected risk over the joint distribution of all the domains: 𝑘
min 𝜃
1 ∑︁ E (𝑥,𝑦)∼𝐷𝑖 [ℓ (𝑦, 𝑓𝜃 (𝑥))] 𝑘 𝑖=1
where ℓ (𝑦, 𝑓𝜃 (𝑥)) is a loss function that measures the prediction error of the model 𝑓𝜃 on a given input (𝑥, 𝑦), and E (𝑥,𝑦)∼𝐷𝑖 is the expected value taken over the joint distribution 𝐷𝑖 . The idea is to learn a model that generalizes well to the unseen domain 𝐷𝑘+1 by minimizing the differences among the learned models from each domain in D. In practice, this is achieved by exploiting the commonalities among the domains and learning features that are invariant to the distribution shifts, such that the model can generalize to new domains. 6.2
Algorithms
Leveraging DomainBed [20], we evaluated 28 baseline techniques to cover the major approaches of domain generalization, including (1) data manipulation, (2) representation learning, and (3) learning strategy. (1) ERM (Empirical Risk Minimization) [57] is a standard approach to training machine learning models on a single dataset. In domain generalization, ERM is modified to train models on multiple source domains without access to target domain data. (2) Fish (First-Order Gradient Matching) [50] is a first-order algorithm that solves an interdomain gradient matching objective between gradients of different domains. (3) IRM (Invariant Risk Minimization) [5] is a regularization-based approach that encourages models to be invariant to domain shifts. It seeks a representation that remains invariant across multiple domains. (4) GroupDRO (Group Distributionally Robust Optimization) [47] is a robust optimizationbased approach that addresses the domain shift problem by minimizing the worst-case empirical risk over a set of data groups. (5) Mixup (ERM-Mixup) [63] is a data manipulation and augmentation technique that applies a weighted linear interpolation between pairs of examples from different domains. (6) MLDG (Meta-Learning for Domain Generalization) [28] leverages a meta-learning strategy to meta-learn how to generalize to various domains. (7) CORAL (Correlation Alignment) [52] aligns the second-order statistics of feature distributions. (8) MMD (Maximum Mean Discrepancy) [29] is a kernel-based measure of distribution distance. It is used to minimize the difference between the feature distributions of the source and target domains. (9) DANN (Domain-Adversarial Neural Network) [18] trains a feature extractor to be domaininvariant by adding a domain classification head as a discriminator and optimizing the adversarial loss.
12
Adaimi, et al.
(10) CDANN (Conditional Domain-Adversarial Neural Network) [34] extends DANN by adding a conditional feature extractor that takes into account the source and target domain labels. (11) MTL (Marginal Transfer Learning) [7] is a transfer learning-based approach that learns shared representations across multiple domains and a domain-specific classifier for each domain. (12) SagNet (Style Agnostic Network) [37] is a domain generalization approach that learns to ignore style-specific information by maximizing the distance between style-specific and content-specific feature distributions. (13) ARM (Adaptive Risk Minimization) [68] is a risk minimization-based approach that reduces the risk of making a wrong prediction on the target domain by minimizing the empirical risk with an adaptive penalty term. (14) VREx (Variance Risk Extrapolation) [27] is a regularization-based approach that extrapolates the variance of the model’s predictions from the source domains to the target domain to reduce the risk of overfitting. (15) RSC (Representation Self-Challenging) [21] is a self-supervised learning approach that trains a model to predict the representation of a transformed input and then minimizes the distance between the predicted and actual representations to learn more robust features. (16) SD (Spectral Decoupling) [40] is a data transformation-based approach that decouples the spectral components of the data to improve generalization performance. (17) ANDMask (Learning Explanations that are Hard to Vary) [39] an explanation method that produces explanations that are robust to small perturbations in the input. It does this by masking out input features that are not important for the model’s output. (18) IGA (Inter Gradient Alignment) [26] proposes a method for improving out-of-distribution generalization by finding the maximal invariant predictor of the output. (19) SelfReg (Self-supervised Contrastive Regularization) [24] trains a network to distinguish between pairs of augmented versions of the same sample data, and pairs of data from different classes. (20) SANDMask (Smoothed-AND Mask) [49] proposes an explanation method that produces smoothed-AND masks, which are more robust to adversarial attacks than traditional masks. The smoothed-AND mask is computed by taking the minimum of the activation map and a Gaussian kernel. (21) TRM (Transfer Risk Minimization) [62] proposes a method for learning representations that support the transfer of predictors across different domains. (22) IB_ERM (Information Bottleneck-ERM) [2] combines the invariance principle with the information bottleneck principle, which encourages the model to learn representations that are both invariant to changes in the input distribution and informative for the task. (23) IB_IRM (Information Bottleneck-IRM) [2] is similar to IB-ERM but uses a different training objective based on IRM instead of ERM. (24) CAD (Contrastive Adversarial Domain) [46] proposes a method for learning representations that are optimal for handling covariate shift, by learning a contrastive adversarial domain adaptation model. (25) CondCAD(Conditional Contrastive Adversarial Domain) [46] proposes a conditional variation of CAD. (26) Transfer [66] quantifies and analyzes the transferability of features learned by using a similarity measure of features between domains. It then augments the training data with domain-specific transformations for domain generalization. (27) CausIRL (Invariant Representation Learning) [14] uses the invariant representation Learning to learn causal mechanisms that are invariant across different environments. It leverages
Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
13
Table 2. Estimation of diversity and correlation shift for each dataset and type of distribution shift. Distribution Change Sensor Location
Sampling Rate Device Type User Behavior
Dataset PAMAP2 DSADS Opportunity PAMAP2 RealWorld HHAR Nexus4 HHAR Juggling
Div. Shift 0.058 ± 0.04 0.997 ±0.14 0.768 ± 0.2 0.586 ± 0.30 0.291 ± 0.21 0.632 ± 0.21 0.857 ± 0.04 0.733 ± 0.17
Corr. Shift 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00
domain adaptation techniques such as CORAL or MMD to match the distributions of features across environments and trains a model that captures the underlying causal structure of the data. (28) EQRM (Empirical Quantile Risk Minimization) [17] aims to improve the robustness of machine learning models to outliers by minimizing the empirical quantile risk of the model. It proposes a new loss function that is a weighted sum of the quantile regression losses at different quantiles of the target distribution, where the weights are determined by the expected loss under the target distribution. The proposed method is shown to be more robust to outliers than traditional mean-squared error loss. 6.3
Evaluation
To ensure fair and reliable performance evaluation of the benchmarks and algorithms, we developed a benchmark platform inspired by DomainBed (Gulrajani et al., 2020) that incorporates all the algorithms and prepares the datasets as described in Section 4.3. For model selection, we employ a technique called "leave-one-domain-out validation." This approach involves iteratively splitting the k domains in a dataset, where at each iteration, one domain is held out as the test domain, while the remaining k-1 domains are used for training. Additionally, within the k-1 training domains, a portion is set aside as a validation set for hyperparameter tuning. The objective is to evaluate the model’s performance not only on the training domains but also on unseen test domains. Therefore, the model that achieves the highest combined performance on both the validation and test domains is selected and reported as the best domain generalization model. This methodology ensures that the chosen model demonstrates robust generalization across different domains. In our evaluation, we report the accuracy as the evaluation metric for each domain in the dataset. We calculate the in_accuracy for each domain in the validation set and the out_accuracy for the test domain. These accuracy values provide insights into how well the model performs on individual train and test domains. To compare the algorithms, we compute the average performance across both the training and test domains. Additionally, we calculate a ranking score for each algorithm relative to the ERM approach, indicating whether the performance is lower than (−1), within [-1,1] difference (0), or higher (+1) than the baseline ERM method. 7 7.1
RESULTS Quantifying Distribution Shifts
Based on the quantification formulas for diversity and correlation shifts, we computed the twodimensional shift for each shift type and dataset. The results are summarized in Table 2.
14
Adaimi, et al.
Table 3. Average performance (in_accuracy / out_accuracy) of ERM and domain generalization algorithms on different datasets dominated by different shift types. Sensor Location
ERM Fish IRM GroupDRO Mixup MLDG CORAL MMD DANN CDANN MTL SagNet ARM VREx RSC SD ANDMask SANDMask IGA SelfReg TRM IB (ERM) IR (IRM) CAD CondCAD Transfer CausIRL EQRM
Sampling Rate
PAMAP2
DSADS
Opportunity
PAMAP2
RealWorld
0.83 / 0.34 0.74 / 0.34 0.82 / 0.35 0.69 / 0.35 0.60 / 0.33 0.14 / 0.14 0.71 / 0.32 0.72 / 0.33 0.77 / 0.34 0.78 / 0.33 0.70 / 0.32 0.83 / 0.33 0.84 / 0.34 0.83 / 0.34 0.83 / 0.35 0.83 / 0.34 0.69 / 0.37 0.22 / 0.24 0.32 / 0.29 0.83 / 0.33 0.83 / 0.36 0.84 / 0.33 0.83 / 0.32 0.49 / 0.38 0.53 / 0.34 0.74 / 0.35 0.69 / 0.37 0.69 / 0.36
0.80 / 0.58 0.74 / 0.54 0.77 / 0.57 0.68 / 0.53 0.65 / 0.50 0.28 / 0.27 0.68 / 0.54 0.69 / 0.54 0.72 / 0.53 0.73 / 0.52 0.68 / 0.52 0.78 / 0.57 0.81 / 0.53 0.78 / 0.58 0.78 / 0.58 0.79 / 0.57 0.66 / 0.51 0.09 / 0.09 0.18 / 0.17 0.79 / 0.58 0.80 / 0.57 0.80 / 0.59 0.77 / 0.58 0.46 / 0.37 0.47 / 0.38 0.65 / 0.52 0.68 / 0.53 0.68 / 0.53
0.52 / 0.37 0.46 / 0.38 0.50 / 0.37 0.44 / 0.37 0.42 / 0.35 0.27 / 0.27 0.44 / 0.36 0.45 / 0.38 0.48 / 0.38 0.49 / 0.40 0.43 / 0.36 0.50 / 0.36 0.52 / 0.35 0.51 / 0.35 0.50 / 0.37 0.51 / 0.36 0.44 / 0.39 0.07 / 0.08 0.12 / 0.12 0.50 / 0.36 0.52 / 0.38 0.51 / 0.36 0.49 / 0.35 0.41 / 0.34 0.40 / 0.36 0.42 / 0.33 0.44 / 0.38 0.44 / 0.37
0.98 / 0.88 0.97 / 0.88 0.98 / 0.88 0.98 / 0.88 0.97 / 0.86 0.15 / 0.15 0.97 / 0.88 0.97 / 0.87 0.96 / 0.88 0.96 / 0.88 0.98 / 0.87 0.98 / 0.89 0.98 / 0.89 0.98 / 0.87 0.98 / 0.89 0.98 / 0.88 0.97 / 0.87 0.16 / 0.17 0.43 / 0.42 0.98 / 0.89 0.98 / 0.87 0.98 / 0.88 0.98 / 0.88 0.71 / 0.68 0.73 / 0.68 0.97 / 0.90 0.97 / 0.88 0.97 / 0.88
0.96 / 0.75 0.95 / 0.74 0.95 / 0.77 0.94 / 0.73 0.92 / 0.74 0.06 / 0.06 0.95 / 0.74 0.95 / 0.73 0.94 / 0.75 0.93 / 0.77 0.94 / 0.72 0.96 / 0.76 0.97 / 0.77 0.96 / 0.74 0.96 / 0.76 0.97 / 0.77 0.94 / 0.73 0.13 / 0.13 0.27 / 0.26 0.97 / 0.76 0.96 / 0.77 0.96 / 0.75 0.96 / 0.74 0.49 / 0.45 0.61 / 0.51 0.95 / 0.76 0.94 / 0.75 0.95 / 0.73
HHARNexus4 0.80 / 0.58 0.74 / 0.57 0.79 / 0.57 0.72 / 0.58 0.72 / 0.56 0.30 / 0.30 0.72 / 0.57 0.72 / 0.56 0.75 / 0.57 0.74 / 0.57 0.71 / 0.58 0.81 / 0.60 0.80 / 0.55 0.79 / 0.58 0.78 / 0.60 0.82 / 0.58 0.71 / 0.57 0.09 / 0.10 0.25 / 0.26 0.81 / 0.59 0.80 / 0.58 0.81 / 0.59 0.79 / 0.56 0.50 / 0.44 0.47 / 0.42 0.70 / 0.58 0.71 / 0.57 0.72 / 0.58
Device Type
User Behavior
HHAR
Juggling
0.80 / 0.72 0.84 / 0.76 0.78 / 0.71 0.81 / 0.74 0.77 / 0.70 0.30 / 0.30 0.80 / 0.73 0.81 / 0.72 0.75 / 0.69 0.75 / 0.69 0.79 / 0.73 0.81 / 0.74 0.81 / 0.72 0.80 / 0.73 0.79 / 0.72 0.80 / 0.73 0.79 / 0.73 0.09 / 0.09 0.18 / 0.18 0.81 / 0.73 0.80 / 0.72 0.80 / 0.73 0.78 / 0.71 0.48 / 0.47 0.54 / 0.52 0.73 / 0.70 0.80 / 0.73 0.80 / 0.73
0.97 / 0.89 0.97 / 0.90 0.97 / 0.88 0.97 / 0.90 0.97 / 0.89 0.57 / 0.56 0.97 / 0.90 0.97 / 0.89 0.96 / 0.89 0.96 / 0.89 0.97 / 0.91 0.97 / 0.90 0.97 / 0.89 0.97 / 0.89 0.98 / 0.90 0.97 / 0.89 0.97 / 0.90 0.14 / 0.09 0.53 / 0.57 0.97 / 0.89 0.97 / 0.89 0.97 / 0.90 0.97 / 0.89 0.73 / 0.72 0.73 / 0.73 0.95 / 0.88 0.97 / 0.90 0.97 / 0.89
From the analysis, we observe that all types of shifts and datasets are predominantly characterized by diversity shift, while exhibiting zero correlation shift. This implies that, for all shift types (sensor location, sampling rate, device type, and user behavior) and datasets in question, the distribution shift in the data is embodied by novel features that are not shared across domains and have no correlation with the activity classes. Thus, as demonstrated in the domain generalization analysis in the following section, creating a model that can effectively generalize to various distribution shifts is a challenging task. 7.2 Domain Generalization Analysis One of our key research goals is to explore the generalizability of human activity recognition models across different types of distribution shifts. Using the benchmark experiments discussed in Section 4.3, we evaluate the performance of several domain generalization algorithms (Section 6.2). Table 3 reports the average in_accuracy and out_accuracy over 5 runs for each dataset/shiftalgorithm pair. We, further, report a ranking score for each algorithm in Table 4. We underline that the ranking score does not indicate whether an algorithm is definitely better or worse than the other algorithms, but instead reflects a relative degree of robustness against a distribution shift. From Tables 3 and 4, we observe that none of the domain generalization algorithms achieves consistently better performance than ERM on all types of distribution shifts and datasets. For sensor location shift, almost all algorithms achieve lower performance compared to ERM on all 3 datasets. Similarly, for sampling rate shift, we observe an improvement in performance for SelfReg across all datasets, however the results are not statistically significant. For device type, Fish achieves a statistically significant improvement in performance compared to ERM, while MTL improves performance for user behavior shift.
Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
15
Table 4. Ranking score of generalization algorithms w.r.t. ERM on different datasets dominated by different shift types. Values marked with * indicate that the difference in performance across 5 runs compared to ERM baseline are statistically significant with p-value < 0.05. Sensor Location
Fish IRM GroupDRO Mixup MLDG CORAL MMD DANN CDANN MTL SagNet ARM VREx RSC SD ANDMask SANDMask IGA SelfReg TRM IB (ERM) IB (IRM) CAD CondCAD Transfer CausIRL EQRM
8
Sampling Rate
PAMAP2
DSADS
Opportunity
PAMAP2
RealWorld
−1 *
−1 * −1 −1 * −1 * −1 * −1 * −1 * −1 *
0 0 0
−1
+1
−1 * −1 * −1 * −1 * −1 * −1 * −1 * −1 * −1 * −1 * −1 −1 * −1 −1 −1 −1 * −1 * −1 * −1 −1
0
0
−1 −1 * −1 * −1 * −1 * −1 *
−1 * −1 * −1 * −1 * −1 * −1 *
−1 −1 * −1 * −1 * −1 * −1 * −1 *
0
−1 * −1 * −1 * −1 * −1 * −1 * −1 * −1 * −1 +1 0 0 0
−1 * −1 * −1 * 0
0
−1 * −1 * −1 −1 * −1 −1 −1 * −1 * −1 * −1 * 0
−1 −1 * 0
−1 −1 −1 −1 +1 +1 0
+1* 0
−1 −1 * −1 * +1 0 0 0
−1 * −1 * +1 * 0 0
0
−1 * −1 * −1 * −1 −1 −1 −1 −1 * +1 +1 0 0
HHARNexus4 −1 *
−1 −1 * −1 * −1 * −1 * −1 * −1 * −1 * −1 * +1 * −1 −1
Device Type
User Behavior
HHAR
Juggling
+1 * −1 * +1 −1 * −1 * 0
+1 −1 * −1 * 0
+1 *
0
−1 0 0
−1 * 0 0
−1 * −1 * +1 *
0 0
0 0 0
+1
0
−1
+1 −1 * −1 * −1 * +1 +1
+1 −1 * −1 * −1 * +1
0 0
0 0
−1 * −1 * +1
−1 * −1 *
0
+1 −1 −1 * −1 * −1 * −1 * −1 *
−1 −1 −1 * 0
−1 −1
0
0 0
0 0 0
−1 * −1 * −1 * −1 *
−1 −1 * −1 * −1 *
0 0
0 0
DISCUSSION
This work provides a unified benchmark framework to evaluate several domain generalization methods on multiple distribution shift scenarios. Our experiments unveil various key insights on the challenges of domain generalization. 8.1
Distribution Gap and Challenges of Domain Generalization
Investigating various types of distribution shifts, we observe a large performance gap as a result of the distribution shift caused by changes in either the wearable sensing device, sensor characteristics, or user’s activity pattern. These changes can induce feature distribution bias into a model, making it challenging to generalize well to diverse datasets and data collection setups. The performance results of ERM presented in Table 3 reveal that distinct shift types and datasets exhibit varying performance gaps between the in_accuracy and out_accuracy. Specifically, changes in sensor location induce a feature distribution shift, resulting in an approximately 0.2 performance gap for DSADS and Opportunity, and even a substantial drop in performance (∼0.5) for PAMAP2. None of the domain generalization algorithms demonstrate an an improved performance on the out-of-distribution target data, underscoring the considerable challenge of domain generalization. On the other hand, a distribution shift caused by changes in sampling rate yield a performance gap of ∼0.2 in ERM for RealWorld and HHAR-Nexus4, and ∼ 0.1 for PAMAP2. The smaller performance gap for PAMAP2 may be attributed to the smaller number of activities being learned, leading to a more simplified problem. In this scenario, applying domain generalization leads to a statistically significant slight improvement in performance for PAMAP2 and HHAR-Nexus4 datasets. A small performance gap (∼0.08) is also observed for shifts in device type and user behavior, with certain domain generalization methods effectively mitigating this performance gap. This suggests that
16
Adaimi, et al.
Table 5. Performance (average target_accuracy ± standard deviation) of domain adaptation algorithms on different datasets and cross-domain scenarios dominated by different shift types. Sensor Location
Target Only Deep Coral SASA CoDATS DSAN AdvSKM DIRT CDAN MMDA DANN DDC CoTMix No Adapt
Sampling Rate
Device Type
User Behavior
PAMAP2
DSADS
Opportunity
PAMAP2
RealWorld
HHARNexus4
HHAR
Juggling
0.91 ± 0.033 0.44 ± 0.022 0.40 ± 0.028 0.48 ± 0.028 0.36 ± 0.055 0.40 ± 0.032 0.37 ± 0.021 0.44 ± 0.044 0.44 ± 0.051 0.49 ± 0.035 0.43 ± 0.041 0.36 ± 0.047 0.35 ± 0.016
0.97 ± 0.001 0.48 ± 0.006 0.45 ± 0.018 0.55 ± 0.026 0.37 ± 0.035 0.47 ± 0..019 0.40 ± 0.020 0.49 ± 0.036 0.47 ± 0.024 0.53 ± 0.025 0.46 ± 0.016 0.37 ± 0.034 0.44 ± 0.025
0.71 ± 0.008 0.30 ± 0.010 0.38 ± 0.015 0.36 ± 0.016 0.39 ± 0.036 0.29 ± 0.018 0.37 ± 0.028 0.43 ± 0.033 0.30 ± 0.10 0.38 ± 0.016 0.31 ± 0.017 0.17 ± 0.002 0.24 ± 0.008
0.98 ± 0.009 0.80 ± 0.040 0.84 ± 0.007 0.83 ± 0.003 0.79 ± 0.025 0.81 ± 0.025 0.87 ± 0.025 0.87 ± 0.044 0.76 ± 0.035 0.86 ± 0.018 0.80 ± 0.027 0.64 ± 0.040 0.76 ± 0.029
0.99 ± 0.001 0.58 ± 0.023 0.73 ± 0.009 0.68 ± 0.018 0.75 ± 0.016 0.59 ± 0.019 0.65 ± 0.039 0.72 ± 0.032 0.59 ± 0.013 0.69 ± 0.031 0.58 ± 0.023 0.57 ± 0.025 0.53 ± 0.025
0.94 ± 0.004 0.39 ± 0.005 0.47 ± 0.014 0.48 ± 0.016 0.55 ± 0.015 0.39 ± 0.008 0.42 ± 0.034 0.49 ± 0.032 0.41 ± 0.014 0.46 ± 0.031 0.39 ± 0.013 0.18 ± 0.024 0.36 ± 0.013
0.96 ± 0.005 0.71 ± 0.006 0.76 ± 0.006 0.81 ± 0.005 0.80 ± 0.007 0.71 ± 0.008 0.78 ± 0.007 0.81 ± 0.006 0.71 ± 0.011 0.80 ± 0.010 0.71 ± 0.005 0.27 ± 0.095 0.67 ± 0.013
0.99 ± 0.001 0.72 ± 0.010 0.69 ± 0.006 0.69 ± 0.016 0.65 ± 0.027 0.71 ± 0.016 0.65 ± 0.056 0.71 ± 0.009 0.78 ± 0.022 0.69 ± 0.019 0.72 ± 0.014 0.63 ± 0.038 0.74 ± 0.059
challenges in domain generalization are more pronounced with shifts in sensor location and sampling rate compared to temporal user behavior changes and variations in device type. 8.2
Domain Adaptation and Model Personalization
Considering the challenges inherent in domain generalization, there is a compelling argument to transition towards domain adaptation, wherein the model adapts to the target domain given a limited subset of target data. In real-world scenarios, data is susceptible to endless distribution shifts that prove challenging for generalization. Additionally, it is practically impossible to account for all distribution shifts and generalize to all of them. Consequently, a more viable approach involves enabling a model to adapt and personalize to new domains, be it a new device, sensor, or individual. With this perspective, we explore the potential for adapting models trained on one source domain to another. In a practical real-world scenario, assuming a model trained on one source domain (e.g., wrist data for sensor location), we adapt the model to unlabeled data from a new target domain (e.g., ankle data). Leveraging the same benchmark datasets outlined in Section 4.3 and drawing inspiration from the domain adaptation framework, AdaTime [42], we assess 11 unsupervised domain adaptation methods [42] across five types of distribution shifts and eight datasets (Table 5). For each distribution shift and dataset, we conduct five experiments, randomly sampling one source and one target domain. Within each source-target experiment, five runs are performed with different random splitting seeds. Both the source and target data are divided into 75% for training and 25% for testing. To gauge the impact of domain adaptation, we establish an upper bound (Target Only - model trained solely on target data) and a lower bound (No Adapt - model trained on source data without adaptation). We observe a performance gap on the target data between a model trained only on source data (No Adapt) and a model trained solely on target data (Target Only). This gap underscores the existence of a domain gap and dataset bias in the model, mirroring our observations in the domain generalization analysis. Applying unsupervised domain adaptation from one source to one target, the performance gap diminishes across all datasets and distribution shifts. Some shifts exhibit more substantial reduction than others, indicating varying degrees of shifts across different datasets. Specifically, shifts caused by sensor location, sampling rate, and device type result in a more pronounced distribution shift, with domain adaptation improving the target performance by approximately 0.11-0.2 w.r.t the lower bound No Adapt. Conversely, user behavior shifts impact only one class in the dataset, leading to a smaller yet relatively significant drop in performance.
Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
17
In comparison to domain generalization, there are noteworthy distinctions that should be noted. A key difference between domain generalization and domain adaptation, in this paper, lies not only in accessing the target domain during training but also in the utilization of multiple source domains during training. More specifically, while domain generalization draws on data from multiple related source domains to train a model that can generalize to unseen target domains, domain adaptation seeks to adapt a model trained on only one source domain to another target domain. Prior research has proposed multi-source domain adaptation methods [10, 53]. Given the observed advantage in multi-source training with simple ERM, this analysis addresses the question of whether adapting the model from only one source is sufficient to reduce the performance gap. By comparing the out_accuracy performance of ERM in Table 3 with the No Adapt target_accuracy in Table 5, we observe the benefit of incorporating multiple sources in the training data for certain datasets and distribution shifts without any domain generalization techniques. This benefit is evident in cases of sensor location shift in DSADS and Opportunity datasets, as well as sampling rate shifts, device type changes, and user behavior. In contrast to the domain generalization results, where no significant improvement was observed compared to ERM for sensor location, domain adaptation enhances the target_accuracy performance, surpassing even the corresponding top performance achieved in domain generalization analysis for PAMAP2 and Opportunity. For sampling rate, the top domain generalization method for each dataset, and even simple ERM, performs slightly better than the top domain adaptation method. This suggests that incorporating data from varying sampling rates during training reduces the domain gap without the need for adaptation. Similarly, a shift in user behavior can be effectively mitigated by including multiple domains during training (ERM), with a slight improvement when applying MTL for domain generalization. However, domain adaptation is less effective in handling this domain gap, possibly due to the change being focused on one activity class. The unsupervised domain adaptation problem typically assumes that source and target domains are sampled from different marginal distributions (𝑃𝑠 (𝑥) ≠ 𝑃𝑡 (𝑥)), a condition not met in our juggling dataset. Conversely, shifts caused by device type exhibit additional improvement in target performance when applying domain adaptation compared to domain generalization. Similar to sampling rate and sensor location shifts, this type of shift affects the feature distribution in all activity classes. 8.3
Limitation and Future Work
While this paper comprehensively explores 27 domain generalization methods and 11 domain adaptation methods across four types of distribution shifts and various datasets, it is important to acknowledge key limitations and identify potential avenues for improvement. First, although this study is the first to delve into domain generalization and adaptation concerning specific isolated distribution shifts relevant to Human Activity Recognition (HAR), it is crucial to acknowledge that in real-world settings, data is often subject to multiple simultaneous and intricate distribution shifts. These shifts are challenging to capture and quantify in a controlled, non-real-world setting. In reality, a model needs to be capable of adapting and personalizing to a broad spectrum of changes. However, in this work, we leveraged the acknowledged challenges in model generalizability existing in the literature to simplify the problem. The objective was to understand and underscore the influence of different distribution shifts on the model’s performance. Furthermore, despite our efforts to select a dataset that highlights a change in user behavior, it’s important to note that a more longitudinal dataset would be essential to capture the full extent of distribution shifts observed in real-world scenarios. The present dataset’s limitations constrain our understanding of how effectively a model can adapt or generalize to changes in behavior. However, the noticeable performance gap observed in the juggling dataset underscores the significance of studying and addressing this specific behavioral shift, particularly in the context of HAR. Future
18
Adaimi, et al.
research could benefit from incorporating datasets with longer temporal perspectives to better capture real-world scenarios. A third limitation of our study pertains to the constraints imposed on the analysis of domain generalization and domain adaptation algorithms. Our current exploration was extensive but restricted to a specific feature extractor and training implementation. For instance, the crossentropy loss function employed in our study is not explicitly designed to handle imbalanced data. It is crucial to emphasize that our primary objective in this research was not necessarily to achieve the highest performance, but rather to highlight the challenges posed by distribution shifts and their impact on model generalizability and adaptation. Thus, model performance can be further be improved through hyperparameter tuning, experimentation with different loss functions, and exploration of other feature extractors. Our study serves as a foundational exploration of the overarching challenges, but there is room for further refinement and optimization of model performance under varying distribution shifts. Future research may consider these aspects to gain a more nuanced understanding of the capabilities and limitations of domain generalization and adaptation algorithms. 9
CONCLUSION
In this work, we shed light on the challenges faced by HAR models in generalizing to diverse sensor-relative and activity-related distribution shifts. Taking the initial steps toward a systematic evaluation, we focused on four types of sensor-based distribution shifts. Our analysis revealed that all types of shifts and datasets are primarily characterized by diversity shifts, indicating the presence of novel features not shared across domains. This highlights the complexity of the challenge faced by models in maintaining generalizability across these distribution shifts. To assess the performance of HAR models under these conditions, we conducted a comprehensive evaluation of 27 domain generalization methods. The results indicated that existing algorithms only marginally outperform the ERM baseline on all types of distribution shifts and datasets. While some improvements were statistically significant, the overall performance advantage remains marginal, leaving considerable room for enhancement. Domain adaptation further emphasized the performance gap across domains but showcased the potential benefit of adaptation and personalization in reducing this gap. Moreover, domain adaptation emphasizes the performance gap across domains but also demonstrates the tangible benefits of adaptation and personalization in mitigating this gap. Our benchmark platform, shared with the research community, serves as a valuable resource for testing existing methods and fostering the development of novel algorithms aimed at addressing the diverse challenges posed by distribution shifts in sensor data. This collaborative effort aims to advance the field and pave the way for more effective solutions in the realm of HAR model generalization and adaptability. REFERENCES [1] Rebecca Adaimi, Howard Yong, and Edison Thomaz. 2021. Ok Google, What Am I Doing? Acoustic Activity Recognition Bounded by Conversational Assistant Interactions. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (2021), 1–24. [2] Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean-Christophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. 2021. Invariance principle meets information bottleneck for out-of-distribution generalization. Advances in Neural Information Processing Systems 34 (2021), 3438–3450. [3] Ali Akbari and Roozbeh Jafari. 2019. Transferring activity recognition models for new wearable sensors with deep generative domain adaptation. In Proceedings of the 18th International Conference on Information Processing in Sensor Networks. 85–96.
Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
19
[4] Kerem Altun and Billur Barshan. 2010. Human activity recognition using inertial/magnetic sensor units. In Human Behavior Understanding: First International Workshop, HBU 2010, Istanbul, Turkey, August 22, 2010. Proceedings 1. Springer, 38–51. [5] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2019. Invariant risk minimization. arXiv preprint arXiv:1907.02893 (2019). [6] Sarnab Bhattacharya, Rebecca Adaimi, and Edison Thomaz. 2022. Leveraging Sound and Wrist Motion to Detect Activities of Daily Living with Commodity Smartwatches. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 2, Article 42 (jul 2022), 28 pages. https://doi.org/10.1145/3534582 [7] Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. 2021. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research 22, 1 (2021), 46–100. [8] Hennie Brugman, Albert Russel, and Xd Nijmegen. 2004. Annotating Multi-media/Multi-modal Resources with ELAN.. In LREC. 2065–2068. [9] Ayse S Cakmak, Nina Thigpen, Garrett Honke, Erick Perez Alday, Ali Bahrami Rad, Rebecca Adaimi, Chia Jung Chang, Qiao Li, Pramod Gupta, Thomas Neylan, et al. 2020. Using Convolutional Variational Autoencoders to Predict Post-Trauma Health Outcomes from Actigraphy Data. In Machine Learning for Mobile Health Workshop at NeurIPS. [10] Avijoy Chakma, Abu Zaher Md Faridee, Md Abdullah Al Hafiz Khan, and Nirmalya Roy. 2021. Activity recognition in wearables using adversarial multi-source domain adaptation. Smart Health 19 (2021), 100174. [11] Youngjae Chang, Akhil Mathur, Anton Isopoussu, Junehwa Song, and Fahim Kawsar. 2020. A systematic study of unsupervised domain adaptation for robust human-activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (2020), 1–30. [12] Ricardo Chavarriaga, Hesam Sagha, Alberto Calatroni, Sundara Tejaswi Digumarti, Gerhard Tröster, José del R Millán, and Daniel Roggen. 2013. The Opportunity challenge: A benchmark database for on-body sensor-based activity recognition. Pattern Recognition Letters 34, 15 (2013), 2033–2042. [13] Changhao Chen, Yishu Miao, Chris Xiaoxuan Lu, Linhai Xie, Phil Blunsom, Andrew Markham, and Niki Trigoni. 2019. Motiontransformer: Transferring neural inertial tracking between domains. In Proceedings of the AAAI conference on artificial intelligence, Vol. 33. 8009–8016. [14] Mathieu Chevalley, Charlotte Bunne, Andreas Krause, and Stefan Bauer. 2022. Invariant causal mechanisms through distribution matching. arXiv preprint arXiv:2206.11646 (2022). [15] Yi-Ting Chiang, Ching-Hu Lu, and Jane Yung-Jen Hsu. 2017. A Feature-Based Knowledge Transfer Framework for Cross-Environment Activity Recognition Toward Smart Home Applications. IEEE Transactions on Human-Machine Systems 47, 3 (2017), 310–322. https://doi.org/10.1109/THMS.2016.2641679 [16] Keum San Chun, Ashley B. Sanders, Rebecca Adaimi, Necole Streeper, David E. Conroy, and Edison Thomaz. 2019. Towards a Generalizable Method for Detecting Fluid Intake with Wrist-Mounted Sensors and Adaptive Segmentation. In Proceedings of the 24th International Conference on Intelligent User Interfaces (Marina del Ray, California) (IUI ’19). Association for Computing Machinery, New York, NY, USA, 80–85. https://doi.org/10.1145/3301275.3302315 [17] Cian Eastwood, Alexander Robey, Shashank Singh, Julius Von Kügelgen, Hamed Hassani, George J Pappas, and Bernhard Schölkopf. 2022. Probable domain generalization via quantile risk minimization. arXiv preprint arXiv:2207.09944 (2022). [18] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. 2016. Domain-adversarial training of neural networks. The journal of machine learning research 17, 1 (2016), 2096–2030. [19] Martin Gjoreski, Stefan Kalabakov, Mitja Luštrek, Matjaž Gams, and Hristijan Gjoreski. 2019. Cross-dataset deep transfer learning for activity recognition. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. 714–718. [20] Ishaan Gulrajani and David Lopez-Paz. 2020. In search of lost domain generalization. arXiv preprint arXiv:2007.01434 (2020). [21] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. 2020. Self-challenging improves cross-domain generalization. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer, 124–140. [22] Naoto Inoue, Ryosuke Furuta, Toshihiko Yamasaki, and Kiyoharu Aizawa. 2018. Cross-domain weakly-supervised object detection through progressive domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5001–5009. [23] Md Abdullah Al Hafiz Khan, Nirmalya Roy, and Archan Misra. 2018. Scaling human activity recognition via deep learning-based domain adaptation. In 2018 IEEE international conference on pervasive computing and communications (PerCom). IEEE, 1–9.
20
Adaimi, et al.
[24] Daehee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. 2021. Selfreg: Self-supervised contrastive regularization for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 9619–9628. [25] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. 2021. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning. PMLR, 5637–5664. [26] Masanori Koyama and Shoichiro Yamaguchi. 2020. When is invariance useful in an Out-of-Distribution Generalization problem? arXiv preprint arXiv:2008.01883 (2020). [27] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. 2021. Out-of-distribution generalization via risk extrapolation (rex). In International Conference on Machine Learning. PMLR, 5815–5826. [28] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. 2018. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, Vol. 32. [29] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. 2018. Domain generalization with adversarial feature learning. (2018), 5400–5409. [30] Haoliang Li, YuFei Wang, Renjie Wan, Shiqi Wang, Tie-Qiang Li, and Alex Kot. 2020. Domain generalization for medical imaging classification with linear-dependency regularization. Advances in Neural Information Processing Systems 33 (2020), 3118–3129. [31] Jing Li, Shuo Shang, and Lisi Chen. 2020. Domain generalization for named entity boundary detection via metalearning. IEEE Transactions on Neural Networks and Learning Systems 32, 9 (2020), 3819–3830. [32] Pan Li, Da Li, Wei Li, Shaogang Gong, Yanwei Fu, and Timothy M Hospedales. 2021. A simple feature augmentation for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 8886–8895. [33] Xinyu Li, Yuan He, J Andrew Zhang, and Xiaojun Jing. 2021. Supervised domain adaptation for few-shot radar-based human activity recognition. IEEE Sensors Journal 21, 22 (2021), 25880–25890. [34] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. 2018. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European conference on computer vision (ECCV). 624–639. [35] Wang Lu, Jindong Wang, Yiqiang Chen, Sinno Jialin Pan, Chunyu Hu, and Xin Qin. 2022. Semantic-discriminative mixup for generalizable sensor-based cross-domain activity recognition. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 2 (2022), 1–19. [36] Francisco Javier Ordóñez Morales and Daniel Roggen. 2016. Deep Convolutional Feature Transfer across Mobile Activity Recognition Domains, Sensor Modalities and Locations. In Proceedings of the 2016 ACM International Symposium on Wearable Computers (Heidelberg, Germany) (ISWC ’16). Association for Computing Machinery, New York, NY, USA, 92–99. https://doi.org/10.1145/2971763.2971764 [37] Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. 2021. Reducing domain gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8690–8699. [38] Shirin Nilizadeh, Hojjat Aghakhani, Eric Gustafson, Christopher Kruegel, and Giovanni Vigna. 2019. Think outside the dataset: Finding fraudulent reviews using cross-dataset analysis. In The World Wide Web Conference. 3108–3115. [39] Giambattista Parascandolo, Alexander Neitz, Antonio Orvieto, Luigi Gresele, and Bernhard Schölkopf. 2020. Learning explanations that are hard to vary. arXiv preprint arXiv:2009.00329 (2020). [40] Mohammad Pezeshki, Oumar Kaba, Yoshua Bengio, Aaron C Courville, Doina Precup, and Guillaume Lajoie. 2021. Gradient starvation: A learning proclivity in neural networks. Advances in Neural Information Processing Systems 34 (2021), 1256–1272. [41] Xin Qin, Jindong Wang, Yiqiang Chen, Wang Lu, and Xinlong Jiang. 2022. Domain Generalization for Activity Recognition via Adaptive Feature Fusion. ACM Transactions on Intelligent Systems and Technology 14, 1 (2022), 1–21. [42] Mohamed Ragab, Emadeldeen Eldele, Wee Ling Tan, Chuan-Sheng Foo, Zhenghua Chen, Min Wu, Chee-Keong Kwoh, and Xiaoli Li. 2023. Adatime: A benchmarking suite for domain adaptation on time series data. ACM Transactions on Knowledge Discovery from Data 17, 8 (2023), 1–18. [43] Sreenivasan Ramasamy Ramamurthy and Nirmalya Roy. 2018. Recent trends in machine learning for human activity recognition—A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 8, 4 (2018), e1254. [44] Attila Reiss and Didier Stricker. 2012. Creating and benchmarking a new dataset for physical activity monitoring. In Proceedings of the 5th International Conference on PErvasive Technologies Related to Assistive Environments. 1–8. [45] Alexander Robey, George J. Pappas, and Hamed Hassani. 2021. Model-Based Domain Generalization. In Advances in Neural Information Processing Systems, M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (Eds.), Vol. 34. Curran Associates, Inc., 20210–20229. https://proceedings.neurips.cc/paper_files/paper/2021/file/ a8f12d9486cbcc2fe0cfc5352011ad35-Paper.pdf
Assessing Distribution Shift in Human Activity Recognition for Domain Generalization
21
[46] Yangjun Ruan, Yann Dubois, and Chris J. Maddison. 2022. Optimal Representations for Covariate Shift. In International Conference on Learning Representations. https://openreview.net/forum?id=Rf58LPCwJj0 [47] Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. 2019. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. arXiv preprint arXiv:1911.08731 (2019). [48] Keum San Chun, Hyoyoung Jeong, Rebecca Adaimi, and Edison Thomaz. 2020. Eating episode detection with jawbonemounted inertial sensing. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC). IEEE, 4361–4364. [49] Soroosh Shahtalebi, Jean-Christophe Gagnon-Audet, Touraj Laleh, Mojtaba Faramarzi, Kartik Ahuja, and Irina Rish. 2021. Sand-mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization. arXiv preprint arXiv:2106.02266 (2021). [50] Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. 2021. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937 (2021). [51] Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kjærgaard, Anind Dey, Tobias Sonne, and Mads Møller Jensen. 2015. Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In Proceedings of the 13th ACM conference on embedded networked sensor systems. 127–140. [52] Baochen Sun and Kate Saenko. 2016. Deep coral: Correlation alignment for deep domain adaptation. (2016), 443–450. [53] Shiliang Sun, Honglei Shi, and Yuanbin Wu. 2015. A survey of multi-source domain adaptation. Information Fusion 24 (2015), 84–92. [54] Timo Sztyler and Heiner Stuckenschmidt. 2016. On-body localization of wearable devices: An investigation of position-aware activity recognition. In 2016 IEEE International Conference on Pervasive Computing and Communications (PerCom). IEEE, 1–9. [55] Bernd Thomas. 2022. SensorLog. App Store. https://apps. apple. com/us/app/sensorlog/id388014573 (accessed Nov. 13, 2021) (2022). [56] Eric Tzeng, Judy Hoffman, Trevor Darrell, and Kate Saenko. 2015. Simultaneous deep transfer across domains and tasks. In Proceedings of the IEEE international conference on computer vision. 4068–4076. [57] Vladimir N Vapnik. 1999. An overview of statistical learning theory. IEEE transactions on neural networks 10, 5 (1999), 988–999. [58] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. 2022. Generalizing to unseen domains: A survey on domain generalization. IEEE Transactions on Knowledge and Data Engineering (2022). [59] Jindong Wang, Vincent W Zheng, Yiqiang Chen, and Meiyu Huang. 2018. Deep transfer learning for cross-domain activity recognition. In proceedings of the 3rd International Conference on Crowd Science and Engineering. 1–8. [60] Shujun Wang, Lequan Yu, Caizi Li, Chi-Wing Fu, and Pheng-Ann Heng. 2020. Learning from extrinsic and intrinsic supervisions for domain generalization. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IX. Springer, 159–176. [61] Xuhai Xu, Xin Liu, Han Zhang, Weichen Wang, Subigya Nepal, Yasaman Sefidgar, Woosuk Seo, Kevin S Kuehn, Jeremy F Huckins, Margaret E Morris, et al. 2023. GLOBEM: Cross-Dataset Generalization of Longitudinal Human Behavior Modeling. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 4 (2023), 1–34. [62] Yilun Xu and Tommi Jaakkola. 2021. Learning representations that support robust transfer of predictors. arXiv preprint arXiv:2110.09940 (2021). [63] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. 2020. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677 (2020). [64] Yi Yao and Gianfranco Doretto. 2010. Boosting for transfer learning with multiple sources. In 2010 IEEE computer society conference on computer vision and pattern recognition. IEEE, 1855–1862. [65] Nanyang Ye, Kaican Li, Haoyue Bai, Runpeng Yu, Lanqing Hong, Fengwei Zhou, Zhenguo Li, and Jun Zhu. 2022. Ood-bench: Quantifying and understanding two dimensions of out-of-distribution generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 7947–7958. [66] Guojun Zhang, Han Zhao, Yaoliang Yu, and Pascal Poupart. 2021. Quantifying and improving transferability in domain generalization. Advances in Neural Information Processing Systems 34 (2021), 10957–10970. [67] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 (2017). [68] Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. 2021. Adaptive risk minimization: Learning to adapt to domain shift. Advances in Neural Information Processing Systems 34 (2021), 23664–23678.
22
Adaimi, et al.
[69] Jiachen Zhao, Fang Deng, Haibo He, and Jie Chen. 2020. Local domain adaptation for cross-domain activity recognition. IEEE Transactions on Human-Machine Systems 51, 1 (2020), 12–21. [70] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. 2022. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).
Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009