Playwright Web Scraping a Step by Step Tutorial

Step 1: Install Playwright

Playwright supports several languages, but the Node.js version is the most common starting point. Install the library and its bundled browsers.

npm init -y
npm install playwright
npx playwright install

The final command downloads the browser binaries (Chromium, Firefox, and WebKit) that Playwright controls. You only need to run it once per environment.

Step 2: Launch a browser and open a page

A minimal script launches a browser, opens a page, navigates, and reads the title.

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();
  await page.goto('https://example.com', { waitUntil: 'domcontentloaded' });

  console.log(await page.title());
  await browser.close();
})();

Run it with node scraper.js. Set headless: false while developing if you want to watch the browser work; switch it back to true for production.

Step 3: Select and extract data

Playwright's locators are robust and readable. To pull a list of items, use a selector that matches the elements you want.

const items = await page.$$eval('.product', cards =>
  cards.map(card => ({
    name: card.querySelector('.title')?.textContent.trim(),
    price: card.querySelector('.price')?.textContent.trim()
  }))
);

console.log(items);

The $$eval method runs your function in the browser context against every matching element, returning a clean array back to Node.

Step 4: Wait for dynamic content

The most common scraping bug is reading the page before its data has loaded. Playwright's auto-waiting helps, but for content that appears after an API call, wait explicitly.

• await page.waitForSelector('.product') waits until at least one item exists.
• await page.waitForLoadState('networkidle') waits until network activity settles.
• await page.waitForResponse(...) waits for a specific request to complete.

Preferring a precise waitForSelector over a fixed sleep makes your scraper both faster and more reliable.

Step 5: Handle pagination

Most real datasets span many pages. A simple loop clicks the "next" control until it disappears.

let results = [];
while (true) {
  const pageItems = await page.$$eval('.product', els =>
    els.map(el => el.textContent.trim())
  );
  results = results.concat(pageItems);

  const next = await page.$('a.next:not([disabled])');
  if (!next) break;
  await next.click();
  await page.waitForLoadState('networkidle');
}

For sites with URL-based paging, it is often cleaner to build each page URL directly and loop over those rather than clicking.

Step 6: Add a proxy

To avoid rate limits and blocks across many requests, route Playwright through a proxy. You can set it at launch.

const browser = await chromium.launch({
  proxy: {
    server: 'http://your-proxy-host:port',
    username: 'user',
    password: 'pass'
  }
});

For tougher targets, residential IPs tend to look more like genuine visitors, while datacenter IPs are cheaper and fine for friendlier sites. Rotating IPs across requests reduces the chance any single address gets flagged.

Good-practice tips

• Throttle your request rate and add small random delays to behave politely.
• Respect each site's terms and robots guidance, and collect only what you need.
• Block images and fonts if you only need text, to cut bandwidth.
• Handle errors and retries so one failed page does not stop the whole run.

Putting it together

With these steps you can build a scraper that launches a browser, navigates, waits intelligently, extracts structured data, walks through pages, and routes traffic through proxies. From here, scaling is mostly about reliability and cost control, and proxies are usually the biggest variable.

Comparing proxy providers on value keeps that line item in check. Cheapest Proxies is our featured value pick and a sensible option to evaluate when you need dependable IPs without a premium price tag.

Browser contexts: the cheap concurrency you are probably missing

The basic tutorial launches one browser and one page, but the real scaling unit in Playwright is the browser context. A context is an isolated session, its own cookies, cache, and storage, inside a single browser process. You can open many contexts far more cheaply than launching many browsers, and each can carry its own proxy, locale, and user-agent. This is the natural way to run parallel scrapes that must not contaminate each other, for example checking how a site looks to visitors in several regions at once.

const browser = await chromium.launch();
const ctx = await browser.newContext({
  proxy: { server: 'http://host:port', username: 'u', password: 'p' },
  locale: 'en-GB'
});
const page = await ctx.newPage();

Because the proxy is set per context, you can give each parallel worker a different IP without restarting the browser, which is both faster and lighter than the launch-per-IP pattern beginners often write.

Network interception to cut bandwidth and noise

Playwright lets you intercept requests with a route handler, which is one of the highest-leverage features for cost control. By aborting requests for images, fonts, media, and analytics, you download only the HTML and data you actually parse, often slashing bandwidth per page. Since proxy traffic is usually billed by the gigabyte, this directly lowers your bill.

await ctx.route('**/*', route => {
  const type = route.request().resourceType();
  if (['image', 'font', 'media'].includes(type)) return route.abort();
  return route.continue();
});

Other practical uses of interception

• Block known tracker and ad domains to reduce noise and detection surface.
• Capture an XHR/fetch response directly to grab structured JSON without scraping the DOM.
• Fail fast on unexpected redirects to a block or captcha page.

Reusing sessions with storage state

Re-authenticating on every run is slow and increases your chance of tripping login defences. Playwright can serialise a context's cookies and local storage to a file, then restore it later, so a scraper can log in once and reuse that session across runs. Combined with a stable proxy for that session, this looks far more like a returning human than a fresh login every time.

await ctx.storageState({ path: 'state.json' });
// later
const ctx2 = await browser.newContext({ storageState: 'state.json' });

Failure handling that makes blocks diagnosable

Production scrapers fail; the difference is whether you can see why. Wrap navigation in retries with backoff, and on failure capture a screenshot and a Playwright trace so you can replay exactly what the browser saw. A page that suddenly returns no items is usually a soft block or a markup change, and a saved screenshot tells you which in seconds. When blocks correlate with specific IPs, that is your signal to rotate proxies or upgrade IP quality; benchmarking a value-focused pool such as Cheapest Proxies against your current provider on success rate is a sensible way to keep that cost honest.

Keep comparing with these closely related pages.