This paper proposes a transformer-based neural network model for predicting credit card default, a crucial task for financial institutions to minimize losses and improve lending practices. Existing state-of-the-art (SOTA) methods often rely on either tree-based models which struggle to capture the temporal latent features in the financial data or complex ensemble approaches which are computationally intensive. The proposed transformer-based model addresses these limitations using self-attention mechanisms to uncover hidden patterns in the temporal dimension. The model is evaluated on a credit card default dataset from American Express and a dataset of credit card clients from a Taiwan bank. The experimental results demonstrate the efficacy and generalizability of our proposed model. The proposed transformer based model outperforms the SOTA lightGBM model and achieves comparable performance to SOTA ensemble methods. An ensemble approach, incorporating the transformer based model and lightGBM model has also been proposed. It outperforms the SOTA ensemble approaches. This paper highlights the potential of transformer based models in financial risk assessment, suggesting that they can lead to more accurate and reliable predictions. It offers a foundation for future research in enhancing credit risk models and their applications in related financial domains.