HSE experts reveal low accuracy of technology forecasts in transportation

HSE researchers evaluated the accuracy of technology forecasts in the transportation sector over the past 50 years and found that the average accuracy rate does not exceed 25%, with the lowest accuracy observed in aviation and rail transport. According to the scientists, this is due to limitations of the forecasting method and the inherent complexities of the sector. The study findings have been published in Technological Forecasting and Social Change.

Today, foresight methods are widely used to inform science and technology policies, national strategies, and investment programmes. Delphi surveys are a key forecasting tool; they involve a structured process for reaching consensus among a group of experts through anonymous, multi-round surveys with controlled feedback. However, the question of how accurate such forecasts are—and whether they can reliably inform decision-making—remains open.

A team of scientists from the Foresight Centre at the Institute for Statistical Studies and Economics of Knowledge and the Institute of Transport Economics and Transport Policy Studies of HSE University analysed several rounds of Japan's Delphi surveys on technology foresight in the transportation sector. The analysis included 647 observations across 167 topics, ranging from high-speed rail to autonomous vehicles.

Japan presents a unique case: since the 1970s, the country has regularly conducted large-scale Delphi surveys of experts on the prospects of science and technology, making it possible not only to analyse current expectations but also to compare past forecasts with actual outcomes.

The researchers’ conclusions were unexpected: the accuracy of predictions about technological developments in transport made using the classical Delphi method is only 25%, meaning that three out of four forecasts have not been realised. The scientists suggest that the low accuracy rate may result from both the structural complexities inherent in the sector and the limitations of the Delphi method.

'Transport can be described as a system of systems: it consists of numerous heterogeneous infrastructure facilities and different modes of transport that are managed and operated independently, yet together form a unified system that is continuously evolving through innovation,' says Tatiana Kulakova, Director of the Institute for Transport Economics and Transport Policy Studies at the HSE Faculty of Urban and Regional Development.

However, environmental constraints, institutional inertia, and the long lifecycle of transport infrastructure significantly limit the sector’s development and delay the adoption of innovations beyond a reasonable planning horizon.

The second reason for the low accuracy of forecasts is the systemic limitation of the Delphi method. It was found that the second round of a Delphi survey—intended to refine and confirm forecasts—may not bring experts closer to consensus but instead increase the dispersion of their estimates.

'Traditionally, the Delphi method is regarded as the most reliable expert-based forecasting tool, which creates inflated expectations about the accuracy of its predictions. The observed decrease in accuracy in the second round (23.4% compared to 26.4% in the first round) calls into question one of the method’s key assumptions—that iteration automatically improves the accuracy of estimates,' explains Olesia Maibakh, Head of the Foresight Research Methodology and Organisation Unit at the HSE Foresight Centre, and the study author.

According to Maibakh, in practice, feedback can increase conformist pressure by suppressing unconventional but potentially accurate assessments. 'As an alternative to the classic second round, it may be more effective to hold foresight sessions that discuss disagreements, treating discrepancies in estimates as a resource rather than as noise,' Maibakh suggests.

Another unexpected finding was the 'forecastability pit' effect, where the accuracy of forecasts does not correlate with the stage of a technology’s maturity. Intuitively, one might expect that the closer a technology is to practical deployment, the easier it would be to assess its prospects. However, the scientists observed that technologies in the middle stages—when they are first introduced into practical use or undergoing initial improvements—exhibit the lowest forecasting accuracy. In contrast, forecasts for technologies at earlier stages of research and development, as well as those in later stages of adoption and scale-up, were much more accurate.

'The low accuracy of medium-term forecasts may reflect the so-called "valley of death" effect, where technologies fail to reach the market for economic rather than technological reasons. The frequent errors made by Delphi survey experts in medium-term predictions may stem from the difficulty of determining which of several similar technological solutions will succeed commercially. For example, predicting the dominance of battery electric vehicles over hydrogen fuel cell vehicles proved challenging,' says Alexey Sukharev, an expert at the Foresight Centre.

According to the authors, improving the accuracy of long-term forecasts requires a reassessment of traditional approaches to expert analysis. Forecasting should primarily focus on industry models that account for sector-specific regulatory and infrastructural constraints. Additionally, it is important to combine multiple foresight methods to improve predictive success. The classical Delphi method should be complemented by big data analysis, system dynamics modelling, and scenario planning to help mitigate experts’ cognitive biases.