Objective: Monitoring fetal movement is essential for ensuring fetal and maternal health. A well-known approach to measuring fetal health is to keep track of fetal kicks regularly. As a result, various devices and algorithms that can count the number of fetal kicks are being developed. Our goal is to utilize accelerometric and gyroscopic sensor recordings of abdominal movements to detect the occurrence of fetal kicks. In this article, we introduce a Long Short-Term Memory (LSTM)-based channel attention mechanism that can learn pertinent information by observing the evolution of each channel over time alongside a sensor fusion classification model, which is the proposed model of this study to detect fetal kicks. Method: We collected a dataset from forty-four pregnant women using our proprietary multi-sensory device equipped with 4 Inertial Measurement Units (IMUs). The proposed model, incorporating the channel attention mechanism, was employed to detect fetal kicks based on the recorded accelerometric and gyroscopic data of abdominal movements. Results: The standalone classification model achieved an 84% accuracy rate, which increased to 88% with the addition of the channel attention mechanism. We were able to outperform previous state-of-the-art channel attention models in terms of accuracy. Significance: Notably, the proposed channel attention mechanism offers broad applicability in scenarios where channel prioritization based on temporal information significance is required. Hence, this work represents a substantial improvement in the field of fetal monitoring and has the potential & flexibility to be utilized for applications beyond fetal kick counting.