Selecting the most appropriate time series forecasting method for real-world tasks is challenging, as performance can vary significantly depending on dataset characteristics. Since proving the universal superiority of any forecasting method is impractical, a structured evaluation framework is essential. This paper proposes a DoE framework for the comparative evaluation of time series forecasting methods using synthetic data. The framework offers a versatile evaluation workflow adaptable to diverse forecasting techniques and begins with user input to incorporate prior knowledge and select candidate methods. Using Design of Experiments (DoE) techniques—such as factorial designs or response surface methodology —the framework defines key experimental factors and generates synthetic time series datasets tailored to different forecasting scenarios. Model training and evaluation are performed using appropriate performance metrics. Results are then summarized, enabling rigorous, reproducible, and context-aware model comparison. A case study comparing two methods on a problem of forecasting the total oil & grease (TOG) illustrates the framework’s flexibility and practical utility.