Scrape jobs yourself, or use an API?
Thinking about scraping job postings yourself? Here is what it really takes, and when a jobs API is the better choice.
Updated
The short answer
You can scrape job postings yourself, but you must build and fix a scraper for every site, handle blocks and changes, and remove closed jobs. A jobs API does that work for you and gives you clean jobs in one format. Scraping makes sense for a few sites. An API makes sense for many.
What does scraping jobs involve?
- Finding each company’s career page
- Writing code that reads each hiring system
- Fixing that code when a site changes
- Going slowly so you do not overload sites
- Spotting when a job closes
- Cleaning places, dates and pay into one format
When does scraping make sense?
If you only need jobs from two or three companies, a small scraper can work well.
When is an API better?
If you need jobs from hundreds or thousands of employers, the work grows fast. An API lets you skip it and focus on your product.
How do we collect jobs?
We read public career pages at a polite pace and follow each site’s rate limits. We never get past logins or CAPTCHAs, and we do not scrape LinkedIn or Indeed.
Questions people ask
Is job scraping legal?
Rules depend on the site and where you are. Read each site’s terms. This is not legal advice.
Can I scrape LinkedIn or Indeed jobs?
Those sites limit automated access in their terms. We collect the same jobs from the employer’s own site instead.