Abstract Fractional O-U process is a very classical stochastic process which is used to describe the time series of financial volatility. Three parameters need to be estimated in this process, and the estimation method based on discrete observations can be realized by machine learning optimization algorithm. In this study, the parameter estimation method of fractional O-U process is briefly described, and three optimization algorithms, Newton method, quasi Newton method and genetic algorithm, are used to estimate the parameters. The comparison shows that genetic algorithm is relatively accurate and efficient. Finally, the minute data of stock index futures are estimated based on fractional O-U process. The results show that the estimation of theta and Hurst index is relatively accurate, and the estimation error of volatility is large.