Big data and real time Labour Market Information

There is increasing research and practice in the use of big data to provide advanced real time Labour Market Information about supply and demand in the Labor market and skills requirements and needs. In part this has been a result of technological advances in collecting, cleaning, storing and analysing large and dynamic databases. Driving it is changes in the Labour market, including the need for closer links between the Labor market and education and training provision, the rapid emergence of new and changed occupations and the speed of changes in skills demands. It also results from the growth of private sector employment agencies and consultancies and the trend towards advertising job vacancies online.

The European vocational education and training agency, Cedefop, have launched a large scale project on Real time LMI. The first stage was a feasibility study on the utility and effectiveness of real time data collection and analysis of the vacancies published via various web portals, and subsequently to develop a working prototype system. The prototype system was developed to gather and analyse vacancies from pre-defined websites in Germany, Czech Republic, Ireland Italy and UK. The rational was based on the premise that the World Wide Web contains large amount of data that is largely unexploited and can provide useful information for designing and realizing new models and tools for innovating Labor market services.

Cedefop report that there are several domains where the availability of web based data can constitute a major improvement over existing data sources. The speed of data retrieval and processing makes web based tools ideal for production of timely Labour Market Information. They believe that the data driven approach allows the early detection of emerging skill needs in certain occupations and sectors. More generally, the skill mismatch problem is usually explained by a combination of information asymmetry between employers and employees, incomplete information in the Labour market, differences between people, and transaction costs.

The project is now being extended to include all the countries of the European Union.

Occupational, skills and course classification systems (2)

In the last article, I described the Standard Occupational Classification (SOC) system used in the UK to classify and describe occupations. People often tend to assume that the same system is used in other countries, especially in Europe. It is not so simple. The USA also calls its classification system SOC, but uses a considerably different system. Similarly there are different classification systems in different European countries.

The International Labour Organisation (ILO) produces the International Standard Classification of Occupations (ISCO). The present version – ISCO08 – is an update on the previous version – ISCO98. ISCO is often used as a tool for comparing occupations and labour markets in different countries and teh Office of National Statistics (ONS) tries to provide some alignment between the UK SOC system and ISCO08.

The LMI for All database provides access to the skills required in different occupations drawn from the USA O*Net survey. As a USA based system O*NET skills and associated data is tied to the US SOC system. We use a ‘fuzzy search’ process to match up the different  classification systems. We use O*Net simply because it is the most detailed data available of the skills required for different jobs although the European Union is developing its own ESCO skills directory.

The EU also produces some comparative data, based on the European Labour Force Survey, which each European country contributes data to and on surveys undertaken by the CEDEFOP agency on future skills demands. Because of how this data is collected, it tends to be less detailed than other data available at UK and national level in Europe.

One of the most frequent questions we are asked is what course people should do for a particular occupation. This is a perfectly sensible question, but the answer is not so easy. As wee have seen occupations are classified under SOC, according to the nature and the general degree of difficulty and hence level of qualification required for the job. But courses at university level are classified under a system called JACS (Joint Academic Coding System) which is a way of classifying academic subjects. JACS is a hierarchical system with codes made up of a letter followed by three numbers. The hierarchical logic was built into the codes themselves – so for example, if the F group is Physical Sciences, F100 is Chemistry, F140 is Environmental Chemistry and F141 is Marine Chemistry. Quite simply the two systems do not match up. And even if one attempted to look at different subjects and try to work out what jobs people might do you would end up with a one to many table.  JACS is will be replaced in 2019 and 2020 by the HE Classification of Subjects (HECoS). It contains a similar number of codes to JACS and defines subjects at a similar level of granularity. However, unlike JACS, HECoS is a simple list of subjects with no inherent hierarchy.

We have a number of ideas about how it might be possible to link courses to occupations and new AI technologies seem to have a lot of promise. We will post more on this laetr in the year.


Standards and Classification systems (1)

One thing I frequently get asked questions about is classification systems. How are jobs classified and where does the classification system come from? Are classification systems the same in the UK and in other countries? How are job classifications related to courses? And how does LMI for All use classification systems?

In this and a following post I will try to answer some of those questions. In the UK jobs are classified according to the Standard Occupation System 2010 (SOC 2010) which was an updated version of the previous SOC2000 classification. The Standard Occupational Classification, first introduced in 1990, is maintained by the Office for National Statistics (ONS).

Occupations are divided into groups and ONS explains that “the major group structure is a set of broad occupational categories that are designed to be useful in bringing together unit groups which are similar in terms of the qualifications, training, skills and experience commonly associated with the competent performance of work tasks.

SOC 2010 has nine major groups and 25 sub-major groups with 90 minor groups and 369 unit groups. The following table provides a list of the major groups and the general nature of qualifications, training and experience for occupations in the major group.

Major group General nature of qualifications, training and experience for occupations in the major group
Managers, directors and senior officials A significant amount of knowledge and experience of the production processes and service requirements associated with the efficient functioning of organisations and businesses.
Professional occupations A degree or equivalent qualification, with some occupations requiring postgraduate qualifications and/or a formal period of experience-related training.
Associate professional and technical occupations An associated high-level vocational qualification, often involving a substantial period of full-time training or further study.  Some additional task-related training is usually provided through a formal period of induction.
Administrative and secretarial occupations A good standard of general education.  Certain occupations will require further additional vocational training to a well-defined standard (e.g. office skills).
Skilled trades occupations A substantial period of training, often provided by means of a work based training programme.
Caring, leisure and other service occupations A good standard of general education. Certain occupations will require further additional vocational training, often provided by means of a work-based training programme.
Sales and customer service occupations A general education and a programme of work-based training related to Sales procedures. Some occupations require additional specific technical knowledge but are included in this major group because the primary task involves selling.
Process, plant and machine operatives The knowledge and experience necessary to operate vehicles and other mobile and stationary machinery, to operate and monitor industrial plant and equipment, to assemble products from component parts according to strict rules and procedures and subject assembled parts to routine tests. Most occupations in this major group will specify a minimum standard of competence for associated tasks and will have a related period of formal training.
Elementary occupations Occupations classified at this level will usually require a minimum general level of education (that is, that which is acquired by the end of the period of compulsory education). Some occupations at this level will also have short periods of work-related training in areas such as health and safety, food hygiene, and customer service requirements.

The next edition of SOC is expected in 2020 and work is already underway. The revisions are undertaken both to help employers and other users in using SOC and also to reflect the changing use of job descriptions and changes in employment. For instance, many more jobs include manager in the job title than would have been 20 years ago. At the same time there has been a rapid growth of jobs in the service sector and in computer based occupations, with the need for new descriptions to reflect this.

Standard Occupational Classification is sometimes mixed with the Standard Industrial Classification system (SIC). As the names imply while SOC describes and classifies occupations SIC provides a classification system for different industries.

One question that we are frequently asked about with LMI for All is why we cannot provide a breakdown of employment in different occupations at a city or local level. Certainly, it would be useful for job seekers and for people planning their career. But employment figures in occupations are derived from a quarterly survey – the Labour Force Survey – and the sample size is simply too small to provide reliable data at local level.

Also in some occupations we do not have sufficient data to provide numbers at a unit group level. In this case we revert to minor group data.

In the next article I will look at the relation between SOC 2010 and other international classification systems.







Jisc Innovation Lab

The UK JISC organisation which supports universities in the development and use of new technologies has also initiated an innovation Laboratory working with data. They are investigating how the huge and diverse sources of data that are now available be better used to address and inform key policy decisions in education and training, in ways that meet both the requirements of national and regional agencies and also the local nuances and concerns of colleges serving immediate communities (Footring, 2017).

The Jisc College Analytics Lab[1] digital modelling environment provides a means to address complex practice and policy questions using highly diverse sources of data. By engaging both with colleges, with their command of the details of learner data, and regional and national planning agencies, that need to aggregate intelligence across wider areas in order to generate policy recommendations.

In particular, Jisc are focusing on local area data to inform strategic planning and decision making. They say “presenting the information in a visual and interactive way helps leaders to communicate their vision and ideas to their funders, staff teams, students and the wider community.” However, they have found is that there is a wide variation in the effectiveness of the way in which colleges make use of the data available to them and a significant duplication of similar core processes across colleges.

The Analytics Lab environment provides a secure technical, legal and project management framework to enable the creation of new, experimental data dashboards. Participants use a mix of open and secure data from both new and established sources, to create visualisations and dashboards which address key business questions.


Data update schedule

As new data is published the LMI for All Database is regularly updated. The table below provides the schedule for updates in the next nine months.

Dataset to be updated


Date of API update

O*NET Update using O*Net version 21 data relating to skills,
interests, knowledge and abilities by occupation
January 2017
Annual Survey of Hours and Earnings (ASHE) ASHE November 2016 data for pay and hours estimates May 2017