tooolbox/blog/Debugging Puppeteer Web Scraping Errors and Anti-Bot Detection
troubleshooting2025-01-10Β·13Β·231/348

Debugging Puppeteer Web Scraping Errors and Anti-Bot Detection

Solutions for common Puppeteer scraping problems including CAPTCHA detection, proxy rotation, headless mode issues, and handling dynamic content loading.

Introduction

Web scraping with Puppeteer is powerful but fraught with anti-bot detection mechanisms that have become increasingly sophisticated. After building scraping systems that needed to collect data from sites with varying levels of protection, I developed a systematic approach to handling the most common failure modes.

This post covers the techniques that work against basic anti-bot detection, the debugging strategies for understanding why scraping fails, and the ethical considerations that guide responsible scraping practices.

Environment

node --version
# v20.11.0

npm list puppeteer puppeteer-extra puppeteer-extra-plugin-stealth
# puppeteer@22.0.0
# puppeteer-extra@3.3.6
# puppeteer-extra-plugin-stealth@2.11.2

The project structure:

scraper/
β”œβ”€β”€ src/
β”‚   β”œβ”€β”€ browser/
β”‚   β”‚   β”œβ”€β”€ launch.ts
β”‚   β”‚   └── proxy.ts
β”‚   β”œβ”€β”€ scrapers/
β”‚   β”‚   β”œβ”€β”€ base.ts
β”‚   β”‚   └── site-specific.ts
β”‚   β”œβ”€β”€ anti-detection/
β”‚   β”‚   β”œβ”€β”€ fingerprint.ts
β”‚   β”‚   └── human-behavior.ts
β”‚   └── storage/
β”‚       └── results.ts
β”œβ”€β”€ config/
β”‚   └── sites.json
└── package.json

Problem

The first issue was that Puppeteer's default headless mode was immediately detected by anti-bot systems. The browser fingerprint included telltale signs of automation:

// Error from target site's anti-bot system
{
  "error": "access_denied",
  "message": "Automated access detected. Please use a regular browser.",
  "detection": {
    "webdriver": true,
    "headless": true,
    "navigator_plugins": 0
  }
}

The second issue was that dynamic content loaded via JavaScript was not present when Puppeteer tried to extract it. The page appeared to load successfully, but the data tables were empty:

// This returns empty results because the table hasn't loaded
const data = await page.evaluate(() => {
  const rows = document.querySelectorAll("table tbody tr");
  return Array.from(rows).map((row) => {
    const cells = row.querySelectorAll("td");
    return {
      name: cells[0]?.textContent,
      value: cells[1]?.textContent,
    };
  });
});
// data is []

The third issue was rate limiting. After making approximately 50 requests to the same domain within 10 minutes, the scraper started receiving HTTP 429 responses:

HTTP/1.1 429 Too Many Requests
Retry-After: 60
X-RateLimit-Limit: 50
X-RateLimit-Remaining: 0

Analysis

Modern anti-bot systems detect automation through multiple vectors. The navigator.webdriver property is set to true in automated browsers, which is the most basic detection method. The Puppeteer stealth plugin patches this and several other detection vectors, but it is not sufficient against advanced detection systems.

The dynamic content issue occurs because Puppeteer's page.goto() waits for the load event, which fires when all initial resources are loaded but before JavaScript frameworks finish rendering. SPAs built with React, Vue, or Angular require explicit waiting for specific elements to appear.

Rate limiting is enforced at the server level based on IP address and request frequency. Without proper delay management and proxy rotation, scrapers quickly exhaust their request quotas.

Solution

Configure Puppeteer with anti-detection measures:

// src/browser/launch.ts
import puppeteer from "puppeteer-extra";
import StealthPlugin from "puppeteer-extra-plugin-stealth";
import { Browser, LaunchOptions } from "puppeteer";

puppeteer.use(StealthPlugin());

export async function launchBrowser(
  options: LaunchOptions = {}
): Promise {
  return puppeteer.launch({
    headless: "new", // Use the new headless mode
    args: [
      "--no-sandbox",
      "--disable-setuid-sandbox",
      "--disable-blink-features=AutomationControlled",
      "--disable-features=IsolateOrigins,site-per-process",
      "--disable-dev-shm-usage",
      "--window-size=1920,1080",
      "--disable-web-security",
      "--disable-features=site-per-process",
    ],
    ignoreDefaultArgs: ["--enable-automation"],
    ...options,
  });
}

Wait for dynamic content with multiple strategies:

// src/scrapers/base.ts
import { Page, ElementHandle } from "puppeteer";

export class BaseScraper {
  protected page: Page;

  async waitForContent(
    selector: string,
    timeout: number = 10000
  ): Promise {
    try {
      // Strategy 1: Wait for specific selector
      await this.page.waitForSelector(selector, { timeout });
      return this.page.$(selector) as Promise;
    } catch (error) {
      // Strategy 2: Wait for network idle
      await this.page.waitForNetworkIdle({ idleTime: 1000, timeout });
      return this.page.$(selector) as Promise;
    }
  }

  async waitForTableData(
    tableSelector: string,
    minRows: number = 1,
    timeout: number = 15000
  ): Promise {
    await this.page.waitForFunction(
      (selector, min) => {
        const table = document.querySelector(selector);
        if (!table) return false;
        const rows = table.querySelectorAll("tbody tr");
        return rows.length >= min;
      },
      { timeout },
      tableSelector,
      minRows
    );
  }

  async extractTableData(tableSelector: string): Promise {
    return this.page.evaluate((selector) => {
      const table = document.querySelector(selector);
      if (!table) return [];

      const headers = Array.from(
        table.querySelectorAll("thead th")
      ).map((th) => th.textContent?.trim().toLowerCase().replace(/\s+/g, "_"));

      const rows = Array.from(table.querySelectorAll("tbody tr"));

      return rows.map((row) => {
        const cells = Array.from(row.querySelectorAll("td"));
        const rowData: Record = {};
        cells.forEach((cell, index) => {
          if (headers[index]) {
            rowData[headers[index]] = cell.textContent?.trim() || "";
          }
        });
        return rowData;
      });
    }, tableSelector);
  }
}

Implement human-like behavior delays:

// src/anti-detection/human-behavior.ts
export class HumanBehavior {
  static async randomDelay(min: number = 500, max: number = 2000) {
    const delay = Math.random() * (max - min) + min;
    return new Promise((resolve) => setTimeout(resolve, delay));
  }

  static async scroll(page: Page, distance: number = 300) {
    const steps = Math.ceil(distance / 100);
    for (let i = 0; i < steps; i++) {
      await page.evaluate(() => {
        window.scrollBy(0, 100);
      });
      await this.randomDelay(100, 300);
    }
  }

  static async typeWithDelay(
    page: Page,
    selector: string,
    text: string
  ) {
    await page.click(selector);
    for (const char of text) {
      await page.keyboard.type(char, {
        delay: Math.random() * 100 + 50,
      });
    }
  }

  static async moveMouseRandomly(page: Page) {
    const viewport = page.viewport();
    if (!viewport) return;

    for (let i = 0; i < 5; i++) {
      const x = Math.random() * viewport.width;
      const y = Math.random() * viewport.height;
      await page.mouse.move(x, y, { steps: 10 });
      await this.randomDelay(200, 500);
    }
  }
}

Implement proxy rotation:

// src/browser/proxy.ts
interface ProxyConfig {
  host: string;
  port: number;
  username?: string;
  password?: string;
}

export class ProxyRotator {
  private proxies: ProxyConfig[] = [];
  private currentIndex: number = 0;

  constructor(proxies: ProxyConfig[]) {
    this.proxies = proxies;
  }

  getNext(): ProxyConfig {
    const proxy = this.proxies[this.currentIndex];
    this.currentIndex = (this.currentIndex + 1) % this.proxies.length;
    return proxy;
  }

  getArgs(): string[] {
    const proxy = this.getNext();
    const args = [`--proxy-server=${proxy.host}:${proxy.port}`];

    if (proxy.username) {
      // Note: proxy authentication is handled separately
      // in page.authenticate()
    }

    return args;
  }
}

// Usage
const proxyRotator = new ProxyRotator([
  { host: "proxy1.example.com", port: 8080 },
  { host: "proxy2.example.com", port: 8080 },
  { host: "proxy3.example.com", port: 8080, username: "user", password: "pass" },
]);

const browser = await launchBrowser({
  args: proxyRotator.getArgs(),
});

Implement rate limiting with exponential backoff:

// src/scrapers/rate-limiter.ts
export class RateLimiter {
  private requestCount: number = 0;
  private lastReset: number = Date.now();
  private maxRequests: number;
  private windowMs: number;

  constructor(maxRequests: number, windowMs: number) {
    this.maxRequests = maxRequests;
    this.windowMs = windowMs;
  }

  async waitIfNeeded(): Promise {
    this.requestCount++;

    if (this.requestCount >= this.maxRequests) {
      const elapsed = Date.now() - this.lastReset;
      const waitTime = this.windowMs - elapsed;

      if (waitTime > 0) {
        console.log(`Rate limit reached. Waiting ${waitTime}ms...`);
        await new Promise((resolve) => setTimeout(resolve, waitTime));
      }

      this.requestCount = 0;
      this.lastReset = Date.now();
    }
  }

  async handleError(statusCode: number): Promise {
    if (statusCode === 429) {
      const waitTime = 60000; // Wait 1 minute
      console.log(`Rate limited. Waiting ${waitTime}ms...`);
      await new Promise((resolve) => setTimeout(resolve, waitTime));
    }
  }
}

Handle CAPTCHA detection:

// src/anti-detection/captcha.ts
export async function detectCaptcha(page: Page): Promise {
  // Check for common CAPTCHA elements
  const captchaSelectors = [
    'iframe[src*="recaptcha"]',
    'iframe[src*="hcaptcha"]',
    '[data-captcha]',
    ".g-recaptcha",
    ".h-captcha",
  ];

  for (const selector of captchaSelectors) {
    const element = await page.$(selector);
    if (element) {
      return true;
    }
  }

  // Check for CAPTCHA text
  const bodyText = await page.evaluate(() => document.body.innerText);
  const captchaKeywords = ["verify you are human", "prove you're not a robot", "captcha"];
  return captchaKeywords.some((keyword) =>
    bodyText.toLowerCase().includes(keyword)
  );
}

Lessons Learned

  1. The stealth plugin is necessary but not sufficient. Advanced anti-bot systems detect automation through behavioral analysis, not just browser fingerprinting.

  2. Always implement human-like delays between actions. Random delays between 500ms and 2000ms mimic natural browsing behavior.

  3. Respect rate limits and robots.txt. Scraping should not disrupt the target site's normal operation. Implement aggressive rate limiting to be a good citizen.

  4. Cache aggressively to minimize requests. If the same data is needed multiple times, store it locally instead of re-fetching.

  5. Monitor for CAPTCHA challenges and handle them gracefully. If you encounter CAPTCHAs regularly, your scraping pattern needs adjustment.

This blog does not accept any external sponsorships, affiliate marketing, or ad revenue.