Large language models (LLMs) are deep learning (DL) algorithms that can perform a range of natural language processing (NLP) tasks. LLMs use transformer models trained using large datasets and have revolutionized NLP tasks. However, their large-scale implementation poses significant computational resources, efficiency, and scalability challenges. This systematic literature review (SLR) examines optimization techniques applied to transformer and LLM models from 2020 to 2024. We analyzed 32 papers to provide an overview of optimization methods and highlight their impact on model performance and computational efficiency. Additionally, we examine the evolution of these techniques, highlight implementation challenges, and propose future research directions for optimization. In addition, we discuss the evaluation factors, environments, and datasets commonly used to evaluate optimization techniques and explore the applications of LLMs in real-world scenarios. Finally, we emphasize the critical role of optimization in improving the performance, reducing costs, and enhancing the scalability of transformer models and LLMs and provide insights into emerging trends and opportunities for future research in this area.