Modular arithmetic is a fundamental operation in cryptography, particularly in public-key cryptography algorithms like RSA (Rivest-Shamir-Adleman) and Elliptic Curve Cryptography. It involves multiplying two numbers and taking the remainder when divided by a modulus value. Hardware implementation of modular multiplication involves designing a digital circuit that performs the multiplication operation followed by modular reduction. The hardware implementation looks more complex due to the design complexity and resource utilization. To reduce the complexity several optimization approaches were published in which the focus is on either area or time. In large modular exponentiation operations, which are common in asymmetric key cryptography algorithms like RSA, the Montgomery modular arithmetic (MMA) technique is used to perform modular multiplication efficiently. This method minimizes the number of modular multiplications needed by breaking down the modular reduction step into several quicker and easier operations. Several implementation methods of the Montgomery algorithm were presented in the past focusing on area and time optimization. In this work, a novel parity-based Finite State Machine (FSM) approach for implementing the MMA algorithm is presented. The implementation of the proposed work is done in FPGA Zynq-7000. The performance analysis was carried out for the standard bit size of 8, 256, 512 as stated by the National Institute of Standards and Technology (NIST) and the proposed approach shows a greater reduction in computation delay time and area utilization.