One Person, One Model—Learning Compound Router for Sequential Recommendation

Liu, Zhiding; Cheng, Mingyue; Li, Zhi; Liu, Qi; Chen, Enhong

doi:10.1109/icdm54844.2022.00039

Cited by 9 publications

(3 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, n = − log(− log(a)) and a is uniform samples from U(0, 1). In our implementation, we adopt Gumbel prior distribution to implement U by following [54], [55]. With such a designed relaxed softmax, the maximum selection is replaced by approximately sampling from the probability distribution so that the collapse issue can be largely alleviated.…”

Section: Masked Codeword Classificationmentioning

confidence: 99%

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Cheng¹,

Liu²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Enhancing the expressive capacity of deep learning-based time series models with self-supervised pre-training has become ever-increasingly prevalent in time series classification. Even though numerous efforts have been devoted to developing self-supervised models for time series data, we argue that the current methods are not sufficient to learn optimal time series representations due to solely unidirectional encoding over sparse point-wise input units. In this work, we propose TimeMAE, a novel self-supervised paradigm for learning transferrable time series representations based on transformer networks. The distinct characteristics of the TimeMAE lie in processing each time series into a sequence of non-overlapping sub-series via window-slicing partitioning, followed by random masking strategies over the semantic units of localized sub-series. Such a simple yet effective setting can help us achieve the goal of killing three birds with one stone, i.e., (1) learning enriched contextual representations of time series with a bidirectional encoding scheme;(2) increasing the information density of basic semantic units; (3) efficiently encoding representations of time series using transformer networks. Nevertheless, it is a non-trivial to perform reconstructing task over such a novel formulated modeling paradigm. To solve the discrepancy issue incurred by newly injected masked embeddings, we design a decoupled autoencoder architecture, which learns the representations of visible (unmasked) positions and masked ones with two different encoder modules, respectively. Furthermore, we construct two types of informative targets to accomplish the corresponding pretext tasks. One is to create a tokenizer module that assigns a codeword to each masked region, allowing the masked codeword classification (MCC) task to be completed effectively. Another one is to adopt a siamese network structure to generate target representations for each masked input unit, aiming at performing the masked representation regression (MRR) optimization. Comprehensively pre-trained, our model can efficiently learn transferrable time series representations, thus benefiting the classification of time series. We extensively perform experiments on five benchmark datasets to verify the effectiveness of the TimeMAE. Experimental results show that the TimeMAE can significantly surpass previous competitive baselines. Furthermore, we also demonstrate the universality of the learned representations by performing transfer learning experiments. For the reproducibility of our results, we make our experiment codes publicly available to facilitate the self-supervised representations of time series in https://github.com/Mingyue-Cheng/TimeMAE.

show abstract

Section: Masked Codeword Classificationmentioning

confidence: 99%

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

Cheng¹,

Liu²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, the electrocardiogram (ECG) signals [33] in electronic health records (EHRs) can be formulated as multivariate time series data since they can be obtained over time and multiple sensors. Comprehensive analysis of such data can facilitate decision-making in real applications [27,30], such as human activity recognition, healthcare monitoring, and industry detection. Particularly, multivariate time series classification (MTSC) tasks, as one fundamental problem of time series analysis, received significant attention in both academia and industry.…”

Section: Introductionmentioning

confidence: 99%

FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification

Cheng¹,

Liu²,

Liu³

et al. 2023

Preprint

View full text Add to dashboard Cite

Deep learning-based algorithms, e.g., convolutional networks, have significantly facilitated multivariate time series classification (MTSC) task. Nevertheless, they suffer from the limitation in modeling longrange dependence due to the nature of convolution operations.Recent advancements have shown the potential of transformers to capture long-range dependence. However, it would incur severe issues, such as fixed scale representations, temporal-invariant and quadratic time complexity, with transformers directly applicable to the MTSC task because of the distinct properties of time series data. To tackle these issues, we propose FormerTime, an hierarchical representation model for improving the classification capacity for the MTSC task. In the proposed FormerTime, we employ a hierarchical network architecture to perform multi-scale feature maps. Besides, a novel transformer encoder is further designed, in which an efficient temporal reduction attention layer and a well-informed contextual positional encoding generating strategy are developed. To sum up, FormerTime exhibits three aspects of merits: (1) learning hierarchical multi-scale representations from time series data, (2) inheriting the strength of both transformers and convolutional networks, and (3) tacking the efficiency challenges incurred by the self-attention mechanism. Extensive experiments performed on 10 publicly available datasets from UEA archive verify the superiorities of the FormerTime compared to previous competitive baselines. CCS CONCEPTS• Computing methodologies → Neural networks; • Mathematics of computing → Time series analysis.

show abstract

“…Sci. 2023, 13, 10933 2 of 15 scenes [13]. It analyzes the learning resources that are most suitable for this specific group in the current learning environment.…”

Section: Introductionmentioning

confidence: 99%

Research on Learning Resource Recommendation Based on Knowledge Graph and Collaborative Filtering

Niu,

Lin,

Xue

2023

Applied Sciences

View full text Add to dashboard Cite

This study aims to solve the problem of limited learning efficiency caused by information overload and resource diversity in online course learning. We adopt a recommendation algorithm that combines knowledge graph and collaborative filtering, aiming to provide an application that can meet users’ personalized learning needs and consider the semantic information of learning resources. In addition, this article collects and models implicit data in online courses and compares the impact of video and text learning resources on user learning needs under different weights in order to deeply understand the different contributions of video and text learning resources to meeting learning needs. The experimental results show that the video high-weight experimental group performs better than the text high-weight experimental group; students tend to prefer video resources. This experiment can help students cope with the challenges brought by numerous types of learning resources and provide personalized and high-quality learning experiences for learners. At the same time, adjusting and innovating teaching models for teachers has great reference value.

show abstract

One Person, One Model—Learning Compound Router for Sequential Recommendation

Cited by 9 publications

References 29 publications

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

TimeMAE: Self-Supervised Representations of Time Series with Decoupled Masked Autoencoders

FormerTime: Hierarchical Multi-Scale Representations for Multivariate Time Series Classification

Research on Learning Resource Recommendation Based on Knowledge Graph and Collaborative Filtering

Contact Info

Product

Resources

About