Published on

A WordPress plugin that throttles Facebook's link-preview crawler

A WordPress plugin that throttles Facebook's link-preview crawler
Authors
On this page

Your WordPress site is suddenly slow. You open the access logs and there it is: thousands of requests from facebookexternalhit. Every time someone shares a link, Facebook's crawler comes knocking, and it doesn't knock once.

That's exactly what happened to a friend's site. My quick fix for them turned into an open-source plugin that now helps WordPress sites keep aggressive crawler traffic in check. Here's how it works, and what its users taught me along the way.

TL;DR: what this plugin does

ProblemSolution
Facebook crawler overwhelming your serverConfigurable request throttling
No visibility into crawler behaviorBuilt-in logging system
Images getting blocked unexpectedlySmart request filtering
Other aggressive botsMulti-bot protection

The problem: a crawler that won't slow down

When someone shares a URL on Facebook, its crawler (facebookexternalhit/1.1) fetches the page to build the preview card. Harmless, right? Until you add it up:

  • Facebook re-crawls pages frequently to keep previews fresh
  • Multiple shares trigger multiple crawl requests
  • Viral content can mean hundreds of requests per minute
  • Shared hosting plans can't handle the load
Cartoon of a worried little server surrounded by dozens of small robots, each holding up a link-preview card.

My friend's site was living this. Their shared hosting provider was threatening to suspend the account over resource usage. The culprit was Facebook's crawler, asking for pages faster than the server could serve them.

The fix: make the crawler wait its turn

Remember when Facebook's crawler last fetched a URL. If it's back too soon, answer with a quick "try later" instead of rendering the whole page.

Here's the core logic:

function check_facebook_throttle() {
    $user_agent = $_SERVER['HTTP_USER_AGENT'] ?? '';

    // Detect Facebook's crawler
    if (strpos($user_agent, 'facebookexternalhit') === false) {
        return; // Not Facebook, proceed normally
    }

    // Get the URL-specific cache key to allow multiple pages to be crawled
    $request_uri = $_SERVER['REQUEST_URI'] ?? '/';
    $cache_key = 'fb_throttle_' . md5($request_uri);

    $throttle_seconds = get_option('fb_throttle_seconds', 60);
    $last_access = get_transient($cache_key);

    if ($last_access && (time() - $last_access) < $throttle_seconds) {
        // Too soon. Tell Facebook to retry later.
        // Use 429 (Too Many Requests) which is the proper rate-limiting status
        header('HTTP/1.1 429 Too Many Requests');
        header('Retry-After: ' . $throttle_seconds);
        exit;
    }

    // Record this access
    set_transient($cache_key, time(), $throttle_seconds * 2);
}

add_action('init', 'check_facebook_throttle', 1);

What to notice:

  • Cache keys are per URL, so crawling page A doesn't block page B.
  • It answers with HTTP 429 (Too Many Requests) instead of 503. 429 is the standard rate-limiting response.
  • Access times live in WordPress's Transient API. No extra database tables, and it works with object caching if you have it.
Flowchart showing request filtering, bot detection, and per-URL transient rate limiting in WordPress.

What users taught me

Issue #1: the previews lost their images

A user messaged me on Facebook (yes, the irony) to say their images weren't showing up in Facebook previews. The plugin was throttling every Facebook crawler request, including the ones fetching og:image assets.

The fix was to spot image requests:

function is_image_request() {
    $request_uri = $_SERVER['REQUEST_URI'] ?? '';

    // Parse the path without query strings
    $path = parse_url($request_uri, PHP_URL_PATH) ?? '';

    // Check for image extensions
    $image_extensions = ['jpg', 'jpeg', 'png', 'gif', 'webp', 'svg'];
    $extension = strtolower(pathinfo($path, PATHINFO_EXTENSION));

    return in_array($extension, $image_extensions, true);
}

Now image requests skip the throttle, and previews show their pictures again.

Issue #2: "What's actually happening?"

The same user asked how to see what the plugin was doing. Fair question. Throttling is invisible by default, so I added logging:

function log_throttle_event($action, $user_agent) {
    if (!get_option('fb_throttle_logging', false)) {
        return;
    }

    // Sanitize user agent to prevent log injection
    $safe_user_agent = preg_replace('/[^\x20-\x7E]/', '', $user_agent);
    $safe_user_agent = substr($safe_user_agent, 0, 100);

    // Get real IP (check forwarded headers for proxies/CDNs)
    $ip = $_SERVER['HTTP_X_FORWARDED_FOR'] ?? $_SERVER['REMOTE_ADDR'] ?? 'unknown';
    // Take only the first IP if multiple are present
    $ip = explode(',', $ip)[0];
    $ip = filter_var(trim($ip), FILTER_VALIDATE_IP) ?: 'invalid';

    $log_entry = sprintf(
        "[%s] %s | UA: %s | IP: %s\n",
        gmdate('Y-m-d H:i:s'),
        $action,
        $safe_user_agent,
        $ip
    );

    error_log($log_entry, 3, WP_CONTENT_DIR . '/fb-throttle.log');
}

Security note: user agents and IPs are sanitized before they're logged, to prevent log injection attacks.

Pull request: it's not just Facebook

A contributor on GitHub pointed out that Facebook isn't the only pushy crawler. Their PR expanded detection to cover:

  • facebookexternalhit - Facebook link previews
  • Facebot - Facebook's general crawler
  • LinkedInBot - LinkedIn preview fetcher
  • Twitterbot - Twitter/X card generator
  • WhatsApp - WhatsApp link previews

You can turn each bot on or off on its own in the settings.

What it does now

After months of community-driven development, here's the feature set:

Throttling

  • Configurable delay between requests (default: 60 seconds)
  • Per-URL throttling (crawling page A doesn't block page B)
  • Per-bot throttle settings
  • HTTP 429 response with Retry-After header (the standard rate-limiting response)

Visibility

  • Optional logging to wp-content/fb-throttle.log
  • Log rotation to prevent disk bloat
  • Request details: timestamp, user agent, IP, action taken

Flexibility

  • Whitelist specific paths (e.g., /wp-admin/)
  • Image request bypass
  • WP-CLI commands for cache clearing
Component diagram showing rate limiting, multi-bot detection, logging, and WP-CLI features of the plugin.

Installation

  1. Download from GitHub
  2. Upload to wp-content/plugins/
  3. Activate in WordPress admin
  4. Configure under Settings -> FB Throttle

Or do it with WP-CLI (download first, then install):

# Download the plugin
curl -L -o fb-throttle.zip https://github.com/nadimtuhin/Facebook-Request-Throttle-WordPress-Plugin/archive/main.zip

# Install and activate
wp plugin install fb-throttle.zip --activate

Do you need it?

Good fit:

  • Shared hosting with limited resources
  • Sites that frequently go viral on social media
  • WordPress installations seeing high crawler traffic in access logs

Not needed:

  • Sites on dedicated servers with resources to spare
  • Low-traffic blogs with occasional social shares
  • If you're already using Cloudflare or similar with bot management

Lessons learned

Maintaining this plugin taught me a few things about open source:

  1. Real users find real bugs. I'd never have hit the image bug in my own testing. I don't share enough content on Facebook.

  2. Logging isn't optional. People need to see what's happening. "Trust me, it's working" isn't good enough.

  3. Start simple, grow on demand. The first version was 30 lines. Now it's ~300, and every addition came from a real use case.

Security considerations

If you throttle crawlers, keep these in mind:

  • User-agent spoofing: any bot can claim to be facebookexternalhit. This plugin is for rate limiting, not security.
  • Log injection: always sanitize data before writing it to logs. The code above shows how.
  • Race conditions: under very high load, several requests can pass the throttle check at the same time. For most sites, that's acceptable.

Real-world crawler behavior

Aggressive social crawlers don't care about your server's capacity. Without rate limiting at the application or edge level, a single shared post can swamp your upstream PHP workers with simultaneous scrape requests.

Rate limiting is one layer. On a busy site, a persistent object cache such as Redis keeps the plugin's transients out of the database, and a page cache can serve your OpenGraph tags without running full WordPress page queries.

Get involved

The plugin is MIT-licensed and contributions are welcome:


Resources