This work proposes a transfer learning-based encoder-decoder framework to predict the relationship between loading conditions and residual stiffness in composites and adhesives. The encoder, built from a convolutional neural network (CNN) and bidirectional long short-term memory (Bi-LSTM), extracts time-series loading signals into latent variables and captures their dependencies. The decoder employs a multilayer perceptron (MLP) to map these latent features to residual stiffness. Transfer learning strategy is used to account for individual variability and further improve accuracy. The model’s effectiveness and robustness are validated through random and constant loading fatigue experiments from two different material systems. Under random fatigue data, the model demonstrates strong learning capabilities, achieving a residual stiffness prediction error of less than 0.5 GPa. In constant amplitude fatigue datasets, the model accurately identifies different materials and exhibits satisfactory robustness when reasonable training dataset size is used.