As the number of Internet of Things (IoT) devices increases, securing IoT communication protocols becomes critical. Due to the resource constraints of IoT networks, these protocols are particularly vulnerable to cyberattacks. Traditional security measures often fail to address the unique challenges posed by IoT communication, highlighting the need for specialized solutions. This research evaluates security vulnerabilities in key IoT communication protocols: MQTT, CoAP, and XMPP by identifying their strengths and weaknesses in handling various attack scenarios. A practical comparison is made for MQTT, examining the impact of using Transport Layer Security (TLS) on its security, while for XMPP, a theoretical comparison for using JSON Web Token (JWT) authentication is conducted. Additionally, the study explores the use of JWT in combination with a complementary nonce-based solution to enhance security and protect against inadequately addressed attacks in CoAP. The findings offer valuable insights that contribute to the development of more secure implementations for the three IoT communication protocols.