Piloting a new online job-scraping tool
As part of extending the data available in LMI for All, the team at IER is currently piloting a new web scraping technique to collect UK vacancy data. LMI for All brings together different sources of labour market data, but a complete and reliable vacancy dataset has been difficult to find. Over the last few months, the LMI for All team has been busy developing a web scraping method to create a constant stream of updated vacancies and to map these to the UK occupational classification system. Web scraping consists of a computerised method to automatically collect information from across the Internet (e.g. job portals).
For the LMI for All pilot, the vacancy information is being scraped daily from three significant job portals in the UK. These job portals are significant because they receive a high number of visits per day and include lots of information about the jobs being advertised.
The pilot data will be released in Autumn this year and further refined over the next few months (we will probably have some gaps to fill and we refine our programming!). The dataset will include current vacancies and links to the advert, as well as some useful variables including occupational classification, location of job at local and regional level, education level required and salary offered. We are also working on identifying the skills in the advert. The aim is for users of LMI for All to be able to ask for vacancies by occupation, location and skills.
Analysing the trend data
Whilst these data will enable vacancy data to be added to the LMI for All database, extending the service, it also enables the creation of trend data to analyse real-time labour demand in the UK. So, we have been playing around with the data. Professor Chris Warhurst and Dr Jeisson Cárdenas-Rubio from IER did some analysis earlier in the year, see A tale of two job vacancies: waitering and nursing.
Occupational variation in vacancies
New labour market data released by the UK’s Office for National Statistics (ONS) shows a continued rise in unemployment levels. The collapse in job vacancies earlier this year was a strong indicator of the massive unemployment to come and this is now seen in the statistics. However, the aggregate vacancy data masks some significant variations by occupation and analysis of the vacancy data reveals falling and increasing demand for different jobs in the UK labour market.
New labour market data released by the UK’s Office for National Statistics (ONS) shows a continued rise in unemployment levels. The collapse in job vacancies earlier this year was a strong indicator of the massive unemployment to come and this is now seen in the statistics. However, the aggregate vacancy data masks some significant variations by occupation and analysis of the vacancy data reveals falling and increasing demand for different jobs in the UK labour market.
The data showed some very bad news for some occupations. The obvious example was waitering jobs, as vacancies for these jobs have collapsed during the Covid-19 crisis. There was some good news and other jobs showed a step rise in job vacancies. Not surprisingly given the health crisis, nurses have been in big demand, with a steep rise in the number of job vacancies. More broadly, this vacancy data shows that the shape of the UK workforce is likely to change and the impact of the Covid-19 crisis on jobs will not be even. The stock, or volume, of some jobs will decline, others will increase.
Over the coming months, the LMI for All team at IER will be analysing these data to capture changes and providing key insights into the UK labour market.