There is increasing research and practice in the use of big data to provide advanced real time Labour Market Information about supply and demand in the Labor market and skills requirements and needs. In part this has been a result of technological advances in collecting, cleaning, storing and analysing large and dynamic databases. Driving it is changes in the Labour market, including the need for closer links between the Labor market and education and training provision, the rapid emergence of new and changed occupations and the speed of changes in skills demands. It also results from the growth of private sector employment agencies and consultancies and the trend towards advertising job vacancies online.

The European vocational education and training agency, Cedefop, have launched a large scale project on Real time LMI. The first stage was a feasibility study on the utility and effectiveness of real time data collection and analysis of the vacancies published via various web portals, and subsequently to develop a working prototype system. The prototype system was developed to gather and analyse vacancies from pre-defined websites in Germany, Czech Republic, Ireland Italy and UK. The rational was based on the premise that the World Wide Web contains large amount of data that is largely unexploited and can provide useful information for designing and realizing new models and tools for innovating Labor market services.

Cedefop report that there are several domains where the availability of web based data can constitute a major improvement over existing data sources. The speed of data retrieval and processing makes web based tools ideal for production of timely Labour Market Information. They believe that the data driven approach allows the early detection of emerging skill needs in certain occupations and sectors. More generally, the skill mismatch problem is usually explained by a combination of information asymmetry between employers and employees, incomplete information in the Labour market, differences between people, and transaction costs.

The project is now being extended to include all the countries of the European Union.