Web Scraping with Node Js

The Node.js Scraping Toolkit

A few well-chosen packages cover the vast majority of scraping tasks in Node.

• axios or the built-in fetch retrieve pages over HTTP with promise-based APIs.
• Cheerio parses HTML with a familiar jQuery-style selector syntax, ideal for static pages.
• Playwright and Puppeteer control headless Chromium for JavaScript-heavy sites.
• Crawlee is a higher-level framework that adds queues, retries, and proxy rotation out of the box.

For static content, axios plus Cheerio is fast and lightweight. For dynamic pages, a headless browser is the dependable choice. For large structured crawls, Crawlee saves you from rebuilding infrastructure.

A Simple Async Scraping Pattern

Here is a compact example that fetches a page and extracts elements with Cheerio. The async style is idiomatic Node and scales well to many URLs.

import axios from "axios";
import * as cheerio from "cheerio";

async function scrape(url) {
  const { data } = await axios.get(url, {
    headers: { "User-Agent": "Mozilla/5.0 (compatible; MyScraper/1.0)" },
    timeout: 15000,
  });
  const $ = cheerio.load(data);
  $(".product").each((_, el) => {
    const name = $(el).find(".name").text().trim();
    const price = $(el).find(".price").text().trim();
    console.log(name, price);
  });
}

scrape("https://example.com/products");

Setting a User-Agent and a timeout from the start makes the scraper more predictable. As you add more URLs, control concurrency so you do not overwhelm the target or your own machine.

Scraping JavaScript-Rendered Pages

When a site builds its content in the browser, the HTML you download is often a near-empty shell. A headless browser renders the page first, then you read the completed DOM.

import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();
await page.goto("https://example.com");
const titles = await page.$$eval("h2", els => els.map(e => e.textContent));
await browser.close();
console.log(titles);

Headless browsers are powerful but resource-hungry, so reserve them for pages that genuinely require rendering and use lightweight HTTP fetching everywhere else.

Managing Concurrency the Right Way

Node's strength is handling many connections at once, but firing thousands of requests with no limit will get you blocked and may crash your process. Use a concurrency limiter, batch your URLs, and add small randomised delays. Frameworks like Crawlee manage this for you with built-in queues and automatic retries, which is a major reason teams adopt them for serious crawls.

Why Proxies Are Essential at Scale

Sustained scraping from a single IP invites rate limits and blocks, and it prevents you from seeing location-specific results. Proxies distribute your requests across many addresses and let you target specific regions.

Choosing a Proxy Type

Datacenter proxies are fast and economical, suiting tolerant targets and high throughput. Residential proxies use real consumer connections, so they blend in better on strict sites at a higher cost. Rotating pools spread load and reduce the chance of any single address being flagged. The best choice depends on your targets and volume.

Since pricing and pool quality differ so much, it is worth comparing providers before you scale. For value-focused developers, Cheapest Proxies is our featured value pick and a strong option when you want dependable proxies without inflating infrastructure costs.

Storing Results and Scaling Up

Small jobs can write to JSON or CSV. As volume grows, a database such as PostgreSQL or MongoDB adds querying, deduplication, and durability. Pair that with structured logging and a retry queue so transient failures are replayed rather than lost. These habits turn a fragile script into a maintainable pipeline.

Fingerprinting and Stealth in Headless Chromium

A default Playwright or Puppeteer browser advertises that it is automated. It exposes the navigator.webdriver flag, ships with a headless user-agent, and presents canvas, WebGL, and font signals that differ from a real user's machine. Sites that run client-side checks read these and block before your scraper sees any content. Stealth plugins patch the most obvious tells, and you can further align the user-agent, viewport, timezone, and locale to look like a coherent real device. Combine that with a residential or ISP proxy whose geography matches the locale you claim, because a US timezone behind a European IP is itself a giveaway. None of this is bulletproof, so reserve heavy stealth for targets that genuinely demand it.

A Retry and Queue Layer That Survives Failure

Networks blip, proxies occasionally fail, and targets throttle. Without a queue, a single error can abort a long crawl. Wrap fetches in retry logic with exponential backoff and jitter, cap the attempts, and push permanently failed URLs to a dead-letter list for later inspection. Crawlee provides this out of the box, but a lightweight custom queue using a concurrency limiter works for smaller jobs. Distinguish error classes: a 429 deserves a backoff and proxy swap, a 404 should not be retried at all, and a connection reset usually warrants an immediate retry on a fresh IP.

Treat error types differently

• Back off and rotate IP on rate-limit responses.
• Drop, do not retry, on genuine not-found responses.
• Retry quickly on transient resets and timeouts.

Keeping Memory in Check Under Load

Headless browsers are the heaviest part of a Node scraper, and the classic failure is a slow memory climb that eventually crashes the process. Each page you open holds resources until closed, and leaked contexts accumulate fast across thousands of URLs. Close pages and contexts deterministically, cap the number of concurrent browser instances, and recycle a browser after a set number of navigations. Browser contexts are the cheap unit of isolation: one Chromium instance can host many contexts, each with its own cookies and proxy, which is lighter than launching a fresh browser per task.

Rotating Proxies the Node Way

In Node the proxy does not live in the request URL. For axios you attach an HTTP or HTTPS proxy agent (or pass a proxy config) so HTTPS tunnelling works correctly; for Playwright you set proxy options at launch or per context, which lets you bind a different IP to each context. Rotate by drawing from a pool per request or per context and retiring addresses that return blocks. When choosing that pool, compare providers on price and success rate against your real targets; a value-focused option such as Cheapest Proxies is worth evaluating when you want dependable rotation without inflating infrastructure cost.

Keep comparing with these closely related pages.