Figure 5 TPE-CBiGRU Structure
Input Layer: The raw data is combined with the extracted temporal and power cycle features for the first fusion (data-level feature fusion) to construct the model’s input feature matrix.
Feature Extraction Layer: The feature matrix is input into the CNN and Bi-GRU networks. The hyperparameters of both CNN and Bi-GRU are optimized using the TPE-based Bayesian hyperparameter search. The CNN outputs high-dimensional spatial features, while the Bi-GRU outputs high-dimensional temporal features.
Feature Fusion Layer: The extracted multi-dimensional abstract spatio-temporal features are fused at the Add layer, completing the second fusion (model-level feature fusion).
Output Layer: The feature matrix obtained after the second fusion is input into the Dense layer to generate the prediction results.
TPE Parameter Optimization
Bayesian Optimization (BO) is an active optimization method and one of the black-box optimization algorithms. Proposed by Snoek et al. in 2012 [38], BO aims to solve extremum problems for functions with unknown expressions. Research has shown that Bayesian optimization outperforms traditional grid search and random search methods, particularly by avoiding local optima. However, traditional BO algorithms can suffer from the curse of dimensionality when handling large parameter spaces, which can degrade their performance [39].
To address the hyperparameter selection problem for the CBiGRU model, this paper proposes a CBiGRU parameter optimization method based on the Tree-Structured Parzen Estimator (TPE). The performance of the CBiGRU model is influenced by parameters such as the number of network units, the number of network layers, and the learning rate. TPE is a non-standard Bayesian optimization algorithm that employs a tree-structured Parzen density estimator [40]. Compared to traditional Bayesian optimization algorithms, TPE can more effectively handle large parameter spaces, thus avoiding the adverse effects of the curse of dimensionality on the algorithm’s performance. Its core idea is to model and optimize the parameter space by constructing a tree-structured model, thereby achieving an efficient parameter search and optimization process. TPE has been widely applied in hyperparameter optimization and automated machine learning, yielding impressive results. In traditional Bayesian optimization theory, the process involves:
The density function represents the observations where the loss of is less than the threshold , while denotes the density formed by the remaining observation values. In the Tree-structured Parzen Estimator (TPE) algorithm, the Bayesian formula is utilized to optimize the expected improvement function :
Let , representing a certain quantile of the TPE algorithm, which is used to partition and , and then construct:
Subsequently, based on equations (6) and (7), we obtain:
Finally, can be simplified to:
From equation (9), it is evident that to maximize , it is necessary to ensure that the probability of is high and the probability of is low at point . TPE utilizes the ratio to evaluate the selected candidate objects, and after a certain number of iterations, the algorithm returns the candidate with the maximum .
Prediction Process
The prediction process in this paper is divided into six steps, as illustrated in Figure 6: