The Web Scraping Trap Nobody Talks About

You want to build a tool that aggregates job postings. Your first instinct is natural: write a scraper that pulls data from Indeed, LinkedIn, and company career pages. Fire up BeautifulSoup or Selenium. Parse the DOM. Extract what you need.

This approach will work for about two weeks. Then the sites change their HTML structure. Your XPath selectors break. You spend Friday night debugging why your scraper is crashing on 20% of requests. Welcome to the maintenance nightmare that kills side projects.

The problem is that web scraping treats symptoms, not causes. You are fighting against constantly evolving interfaces designed for humans, not machines. Additionally, many companies explicitly forbid scraping in their terms of service, which puts you in a legal gray area even if your technical approach works.

Every Major Hiring System Has a Secret Public API

Here is what most developers miss: nearly every applicant tracking system (ATS) that powers a company careers page publishes a public JSON or XML feed. This is not hidden. It is not unofficial. It exists specifically so companies can embed job widgets on their own websites.

Think about it logically. When you visit Stripe's careers page, you see live job listings. Those listings are not hardcoded into HTML. The page is a client application that fetches data from an API endpoint. That same endpoint is available to you.

The Web Scraping Nightmare

2 weeks

Time Until Breakage

20%

Percentage of Requests That Fail

High

Legal Risks of Scraping

Job Boards with Public APIs

The major systems include Greenhouse, Lever, Ashby, SmartRecruiters, Workable, Recruitee, Personio, BambooHR, and Workday. Each one exposes its data differently, but all are unauthenticated and legal to read. This is fundamentally different from scraping. You are using the same data pipeline that the company's own website uses.

Why This Matters for Your Architecture

Unlike HTML scrapers, API endpoints are stable. They are documented (sometimes). They do not break when a designer decides to reorganize the page layout. You are consuming structured data rather than reverse-engineering HTML structure. This means your code is simpler, faster, and actually maintainable.

The Real Work Starts After You Find the Endpoints

Knowing where the APIs live is only the beginning. The actual complexity emerges in four specific areas that separate a working prototype from a production system.

Tenant discovery is your first challenge. The endpoint needs a board token or company slug, but there is no centralized directory. You find tenants by watching where career links point, crawling sitemaps, and monitoring when companies switch ATS providers. This is not a one-time task. Companies migrate constantly, old feeds keep serving stale jobs, and new ones appear daily.

Data normalization comes next. Nine different ATS platforms mean nine different schemas. One system calls it job_title, another calls it title. Dates are stored as epoch milliseconds in one feed and human-readable strings in another. A "Senior Backend Engineer (m/f/d)" from one vendor needs to become identical to "Sr. Backend Engineer" from another. This requires building a normalization layer, not just passing through raw JSON.

Deduplication is where most projects fail silently. The same role appears on the company ATS, Indeed, LinkedIn, and three regional job boards. Without cross-source deduplication, your analytics become fiction. You are counting the same position five times. Building reliable dedup logic requires comparing job details, location data, and posting dates across sources.

Freshness tracking is continuous work. Feeds do not push updates. You are polling thousands of tenants on a schedule, diffing results, and deciding what "closed" means when a job simply vanishes. How often should you crawl? Daily? Hourly? Different companies post at different frequencies. Too aggressive and you waste resources. Too slow and your data becomes stale.

A Practical Example: Parsing Dates Across Systems

Let me show you why this matters. Here is what you actually encounter in production:

// Lever: createdAt in epoch milliseconds
const leverDate = 1704067200000; // Jan 1, 2024

// Greenhouse: updated_at changes whenever post is edited
const greenhouseDate = "2024-01-15"; // Not creation date

// Workday: Human readable string
const workdayDate = "Posted 3 Days Ago"; // Relative time

// Your normalization function needs to handle all three
function normalizePostDate(date, atsType) {
  if (atsType === 'lever') {
    return new Date(date);
  } else if (atsType === 'greenhouse') {
    // You cannot use updated_at for freshness
    // Need to track when you first saw it instead
    return storedFirstSeen[jobId];
  } else if (atsType === 'workday') {
    // Parse English relative time
    return parseRelativeDate(date);
  }
}

This is not a one-line fix. Each ATS has quirks. Workday silently caps pagination depth on some tenants. SmartRecruiters uses limit and offset, but the behavior differs from Workable. Personio returns one massive XML document instead of paginated responses. Your code branches constantly.

Location Data: The Hidden Complexity

Location information appears straightforward until you encounter real data. Then you see "Remote, US", "New York / London / Remote", "EMEA", and empty strings on positions clearly based in San Francisco. Location is unstructured almost everywhere.

If you need country-level filtering, you are not just reading a field. You are building a location resolver. This means parsing ambiguous strings, handling regional abbreviations, and making assumptions about multi-location roles. Is a "London / Remote" job available in Germany? Probably, but you cannot know for certain from the data.

Furthermore, extracting salary ranges is an NLP problem, not a regex. Outside of Ashby's optional structured compensation field, salary lives buried in free text: "up to $185k", "£70,000, £90,000 DOE", "competitive". You cannot reliably extract this with pattern matching. You need semantic understanding or pre-trained models.

When to Build Versus When to Buy

Here is the honest question: should you build this yourself or use an existing service? The answer depends on your constraints.

Build if:

Why APIs Are Better Than Scraping

Stability High

Maintenance Effort Low

Legal Compliance Yes

Data Structure Changes Rare

Documentation Availability Sometimes

You need data from only one or two ATS providers
You are willing to maintain crawler logic as a core part of your product
Your use case is narrow enough that normalization is simple
You have time to handle the 25th ATS provider when it breaks

Buy if:

You need data from 10+ sources reliably
Your product is not specifically about job data, it is about solving something else
You cannot afford weekly maintenance surprises
You need normalized, deduplicated data across sources

The first six ATS integrations are actually fun to build. The complexity grows exponentially after that. You start dealing with edge cases, regional variations, and vendor-specific bugs. A service like JobsPipe handles this layer for you, serving everything through a consistent schema. However, if you only need Greenhouse data from 10 companies, building it yourself takes a weekend.

Testing Your Implementation

Once you have built your integration, testing becomes critical. You need to validate that your normalization is working correctly. Use a JSON formatter tool to inspect API responses and understand the structure before building your parsers. Additionally, validate your normalized output matches expected schemas.

Beyond Job Boards: API Data Aggregation Patterns

The lessons here apply far beyond job boards. Any time you are building an aggregator or data product, you face the same core problems:

First, determine if public APIs exist. Do not assume you need to scrape. Most SaaS products expose some form of public data access. Second, build normalization early. Do not wait until you have data from multiple sources to define your canonical schema. Third, handle pagination and freshness systematically. Do not bolt this on later when you have half a million stale records.

Additionally, consider using tools like Postman API Platform to explore and test endpoints before you write integration code. Postman lets you quickly understand API behavior, test different parameter combinations, and document your findings. This saves enormous amounts of debugging time.

When you are handling JSON responses from multiple sources, keeping your data organized matters. Meanwhile, validating and formatting JSON helps you catch structural issues early in development.

What to Watch Next

If you are building a job board aggregator or any data product that pulls from multiple sources, start here: pick one ATS provider and build a minimal integration. Do not try to handle all nine at once. Get comfortable with the pagination, date handling, and response structure for a single source first.

Then ask yourself honestly: does maintaining this crawler scale with my product goals? If your product is about job matching, salary analysis, or candidate sourcing, then the crawler is core infrastructure. If your product is something else and jobs are just one feature, strongly consider a service that handles this layer.

The developers who win in 2024 are not writing better scrapers. They are building better products on top of data that works reliably. Choose your architecture accordingly. Furthermore, remember that every hour spent maintaining brittle scrapers is an hour not spent building features customers actually care about.

Start small, measure maintenance burden, and scale strategically. That is how you build data products that survive beyond the first three months.