Predicting volatility is of great importance in pricing financial derivatives, selecting portfolios, measuring and managing investment risk more accurately. Since 1980 volatility forecasting is based on day by day datasets. However, the last decade, the use of ultra-high frequency datasets provided more accurate volatility forecasts. In the case where we are interested in evaluating a method’s forecasting ability, a loss function, which takes into consideration the utility of the forecasts is mainly constructed. Although utility functions are measures of accuracy, which are constructed based upon the goals of their particular application, in the majority of the cases, their statistical properties are unknown. The superiority of a utility function against others must be judged by a statistical-theoretical ground and mot just from its empirical motivation. Even though we cannot investigate the statistical properties of a loss function, we are capable to use it for measuring whether two forecasts have statistically equal forecasting accuracy. The majority of the hypotheses tests, which exist in the forecasting literature, compare the ability of two models in producing accurate predictions. However, the simultaneous comparison of the available forecasts provides a more robust comparison of the competing methods of forecasting. The main research objective is the development of a volatility forecasting evaluation framework which would combine the state-of-the-art findings in financial and statistical literature. We seek to combine a) the recent findings in ultra-high frequency modelling, with b) the techniques of simultaneous multiple model comparison and c) the construction of a utility function (or loss function) whose statistical properties would be known.