k-means is a clustering algorithm used to group observations into clusters. Due to the multidimensionality of datasets, interpreting clustering results has become increasingly challenging. In response, sparse clustering variants have emerged, allowing each feature to be weighted. In the sparse k-means algorithm, feature weights are computed based on the values of their associated observations. However, the sparse k-means algorithm is known to be sensitive to outliers. Hence, robust sparse k-means variants have emerged, performing sparse kmeans while detecting outliers. In numerous real-world cases, data entry or measurement errors can lead to poorly collected values for a feature, making them significantly different from other values in that feature. Due to dataset multidimensionality, these observations are often not detected as outliers by existing robust approaches. This negatively impacts the evaluation of feature weights, biases the interpretability of results, and leads to poor clustering quality. To fill this gap, this paper introduces a new robust clustering framework. This framework integrates robust initialization and detection techniques for such observations. The results demonstrate that robustness, interpretability, clustering quality are enhanced across several real and synthetic datasets. Impact Statement-Clustering algorithms are fundamental to understand datasets and also serve as a preprocessing step for many algorithms. In particular, sparse k-means builds on the interpretability of k-means and further improves it based on feature selection. Our method will help to robustify sparse kmeans with respect to abnormal feature values that may impede the estimation of feature weights. The applicability of sparse kmeans will be enhanced in more challenging, noisy settings.