The lower bound decay rate of global solution to the compressible viscous quantum magnetohydrodynamic model in three-dimensional whole space under the $H^5\times H^4\times H^4$ framework is investigated in this paper. We firstly show that the lower bound of decay rate for the density, velocity and magnetic field converging to the equilibrium state (1,0,0) in $L^2$-norm is $(1+t)^{-\frac{3}{4}}$ when the initial data satisfies some low frequency assumption. Moreover, we prove that the lower bound of decay rate of $k(k\in [1,3])$ order spatial derivative for the density, velocity and magnetic field converging to the equilibrium state (1,0,0) in $L^2$-norm is $(1+t)^{-\frac{3+2k}{4}}$. Then we show that the lower bound of decay rate for the time derivatives of density and velocity converging to zero in $L^2$-norm is $(1+t)^{-\frac{5}{4}}$, but the lower bound of decay rate for the time derivative of magnetic field converging to zero in $L^2$-norm is $(1+t)^{-\frac{7}{4}}$.