Cheerio Web Scraping a Tutorial

What Cheerio is and where it fits

Cheerio takes a string of HTML and builds a traversable structure you query with CSS-style selectors. Because it skips the overhead of a full browser, it is dramatically lighter and faster than headless tools. The catch is that it only sees what arrives in the raw HTML response. If a site builds its content with client-side JavaScript after load, Cheerio sees the empty shell, not the final page.

So the rule of thumb is simple: if the data appears in view source, Cheerio is an excellent, efficient choice. If it only appears after scripts run, you need a rendering step first, then Cheerio can parse the result.

Installing and loading HTML

Cheerio parses HTML you supply, so pair it with a fetch library to retrieve pages. After installing, load the markup with cheerio.load, which returns a function conventionally named $.

npm install cheerio

const cheerio = require('cheerio');

async function getPage(url) {
  const res = await fetch(url);
  const body = await res.text();
  return cheerio.load(body);
}

Selecting and extracting elements

With the loaded document, selection feels just like jQuery. You target elements by tag, class, id, or attribute, then read their text or attributes.

const $ = await getPage('https://example.com/products');

const firstTitle = $('.product .title').first().text().trim();
const link = $('a.next').attr('href');
console.log(firstTitle, link);

The key helpers are .text() for inner text, .attr(name) for attributes, .html() for inner markup, and .first() or .eq(i) to narrow a match set. Chaining these reads naturally and keeps extraction concise.

Looping over many items

Most scrapes pull a list of records. Use .each() to iterate matched elements, scoping selectors to each item so fields stay aligned.

const products = [];
$('.product').each((i, el) => {
  products.push({
    title: $(el).find('.title').text().trim(),
    price: $(el).find('.price').text().trim(),
    url: $(el).find('a').attr('href'),
  });
});
console.log(products);

Scoping with $(el).find(...) rather than a global selector is what keeps each record's fields matched to the right listing, which is the most common source of misaligned data for newcomers.

Cleaning data and handling pagination

Real pages are inconsistent. Trim whitespace, default missing fields rather than letting them throw, and normalise values before storing. For multi-page results, read the next-page link, follow it, and repeat until it disappears.

let url = 'https://example.com/products?page=1';
const all = [];
while (url) {
  const $ = await getPage(url);
  $('.product').each((i, el) => all.push({
    title: $(el).find('.title').text().trim() || null,
  }));
  const next = $('a.next').attr('href');
  url = next ? new URL(next, url).href : null;
}

Scaling reliably with proxies

Because Cheerio is so fast, your limiting factor when scraping many pages is the fetching layer. Sending a flood of requests from one IP invites rate limits and blocks. Rotating proxies spread requests across many addresses so your traffic resembles ordinary distributed visitors.

const res = await fetch(url, {
  headers: { 'User-Agent': 'Mozilla/5.0 ...' },
  // route through your proxy agent here
});

Combine proxies with good manners: stagger requests, set a realistic user agent, retry transient errors, and respect robots directives. When comparing providers, look at pool size, location coverage, success rate, and price as a whole rather than chasing the lowest number. For value-minded buyers, Cheapest Proxies is our featured value pick and a strong option to compare against the field.

Common pitfalls

• Empty selections: the content is JavaScript-rendered, so Cheerio never sees it; render first, then parse.
• Misaligned fields: selectors were global instead of scoped to each item with find.
• Relative URLs: resolve hrefs against the page URL so links are usable later.
• Brittle selectors: anchor on stable classes or attributes so minor layout changes do not break the run.

Mining structured data instead of rendered text

The base walkthrough selects visible elements like titles and prices, but those selectors break whenever the layout shifts. Many sites embed the same information in machine-readable form that rarely changes. A JSON-LD script block can carry a product's full details in one place, and Open Graph meta tags expose title, image, and description in a consistent format across pages. Reading those with Cheerio gives you cleaner values with far fewer brittle selectors.

The pattern is simple: select the script or meta element, read its contents or attribute, and parse the JSON when present. Because this structured data is meant for machines, it is usually more stable across redesigns than the human-facing markup, making it the smarter first target when it exists.

High-value, low-fragility targets

• JSON-LD blocks in script tags carrying product, article, or breadcrumb data.
• Open Graph and meta tags for title, description, image, and canonical URL.
• data-* attributes that often hold ids, prices, or state the rendered text omits.

Skipping HTML entirely with hidden JSON endpoints

Sometimes the best Cheerio scrape involves no HTML at all. Open the browser network tab and watch what loads after the page; listings frequently come from a JSON API the front end calls. Hitting that endpoint directly returns clean, structured data without parsing markup, which is faster and less fragile. Cheerio still earns its place for pages that genuinely ship their data in HTML, but checking for an underlying API first can save a lot of work.

Concurrency, rate limits, and proxy rotation

A naive while-loop fetching one page at a time wastes Cheerio's speed waiting on the network. Run several fetches in parallel behind a fixed concurrency limit so you stay fast without flooding the target. Pair that with a small delay between requests and rotating proxies so traffic spreads across many addresses. The goal is throughput that looks like distributed visitors rather than a burst from one IP. For value-conscious projects, Cheapest Proxies is worth benchmarking against other providers on success rate, coverage, and price.

When Cheerio needs a rendering partner

Cheerio cannot run JavaScript, so for truly dynamic pages you need a headless browser to render first. The efficient division of labor is to let the browser produce the final HTML, then pass that string to Cheerio, which parses and extracts far faster than doing all the work inside the browser context. Reserve the heavy renderer for the render step alone and keep your selection logic in lightweight Cheerio code.

Keep comparing with these closely related pages.