The tradeoff between internet of things (IoT) network growth and user requirements significantly maximized which affected by the dynamically devolving malware attacks. The conventional security systems have utilized for scalable and intelligent data security, especially intrusion detection systems have used for IoT network by using the different network parameters. Several recent solutions also utilized for IoT network to address the problems, this work present the cryptography with transfer learning model for malware detection in IoT network. Here, the dynamic chaos encryption model is used for encryption purpose which generates the encrypted data with proper key generation process to ensure the data protection. The transfer learning model evolved from the fine-tuned rule-set includes convolutional neural network (CNN) and recurrent neural network (RNN) is used for network traffic patter analysis and detect the malicious types in the IoT network. The proposed work utilizes the Bot-IoT and TON_IoT datasets to validate the performance. The Chaos+RNN model achieved 99.624% accuracy on the Bot-IoT dataset, outperforming the previous best by 1.272%, with precision, recall, and specificity improvements of 1.301%, 1.308%, and 1.055%, respectively. On the TON_IoT dataset, it reached 99.472% accuracy, exceeding the top model by 1.365%, with respective gains of 1.447%, 1.449%, and 1.168%. The findings show that the system achieves superior malware detection performance, with increased precision and recall across various attack scenarios.