Complex transformer models have become popular in practice, however, they function as “black boxes”. Therefore, there is a growing need for the quantitative evaluation of existing explainability techniques. This evaluation becomes challenging when there are no ground-truth explanations available in the text data. We address this by exploring evaluation approaches for local explainability techniques in Turkish text classification tasks. We use BERT-based models, specifically BERTurk and TurkishBERTweet, and apply SHAP, LIME, and Integrated Gradients (IG). We evaluate the explainability techniques based on their ability to preserve the original text’s prediction probability when their most important tokens are used during inference. We employ evaluation approaches like Mean of Probabilities and Incremental Deletion to compare the explainability techniques with a baseline approach, aiming to measure the faithfulness of the explanations. Our results demonstrate that the gradient-based technique, IG, effectively identifies salient tokens that correlate with the class of the original text. We conclude that the IG method is effective in computing the saliency scores of the explanations when using an encoder-based model for Turkish text classification tasks. In understanding Turkish morphological complexity, it captures and highlights the nuanced contributions of context-dependent words and phrases.