Part 2 - Finding All Companies In The UK With 250+ Staff

Now that we’re done building, we can start the scrape, with Apify providing logging and access to the results in real time. In a few minutes, we get a list of all 12K+ businesses and most importantly, a link to their individual page we can use to find the pay gap report. As mentioned in Part 1, we can add /2018 to each link in order to find the individual report for the 2018 reporting year. These links will be the input we give to the individual report scraper.

Building the Individual Report Scraper

Now that we have a list of links to the individual reports for each business, we can give this to our “Individual Report Scraper”, which will visit each link and pull out the information we need. For these pages, we don’t need to click any links or navigate around the page, we just need to extract the information displayed. This means scraping these pages will be much simpler.

results.push({
        company_name: $('#OrganisationName').text().trim(),
        report_date: $('.employer-report-metadata').find('.metadata-text-value').eq(0).text().trim(),
        employer_size: $('.employer-report-metadata').find('.metadata-text-value').eq(1).text().trim(),
        contact_name: $('.employer-report-metadata').find('.metadata-text-value').eq(2).find('span').eq(0).text().trim(), //.find('span').eq(0).text().trim()
        contact_title: $('.employer-report-metadata').find('.metadata-text-value').eq(2).find('span').eq(1).text().trim(),
        website: $('.employer-report-metadata').find('a').eq(2).attr('href')
    });

Individual Report Scraper Code Our  Page Function for these pages will simply use JQuery to select the elements we need and return these. As you can see we continue to use the find() function to grab the elements we want as well as text() and trim(). In this version, we also use the eq() function, which allows you to select an individual result from a list of results. For instance, find(‘span’) in our case returns two results, the tag that gave the contact’s name and the tag that gave the title. Using eq(0) we can select only the first item in that list - being the name, and eq(1) for the second item - the title.

With our Page Function written, we’re ready to start the scraper. For the report scraper however, we want it to visit multiple links, not just one. Luckily Apify lets us provide a file with a batch of links to visit and extract rather than just a single URL to start at. As there are over 12 thousand pages, the scraper will take some time to step through each page, but once complete you will be left with a CSV file of results ready to download.

Finishing Up

Now that we’ve completed our scrape, we ended up with details for approximately 12 thousand businesses in two files, one that gives the overview information and another that gives the individual report information. Stitching these together with in Google Sheets gives a complete list like the below;

Using the techniques in this project, you will be able to scrape just about any website without a login screen or employing anti-web scraping, so get thinking about what websites you could build a scraper for with this template. With that, I’ll leave this in your hands!

Happy Scraping

BD