- Published on
A WordPress plugin that throttles Facebook's link-preview crawler

- Authors

- Name
- Nadim Tuhin
- @nadimtuhin
On this page
Your WordPress site is suddenly slow. You open the access logs and there it is: thousands of requests from facebookexternalhit. Every time someone shares a link, Facebook's crawler comes knocking, and it doesn't knock once.
That's exactly what happened to a friend's site. My quick fix for them turned into an open-source plugin that now helps WordPress sites keep aggressive crawler traffic in check. Here's how it works, and what its users taught me along the way.
TL;DR: what this plugin does
| Problem | Solution |
|---|---|
| Facebook crawler overwhelming your server | Configurable request throttling |
| No visibility into crawler behavior | Built-in logging system |
| Images getting blocked unexpectedly | Smart request filtering |
| Other aggressive bots | Multi-bot protection |
The problem: a crawler that won't slow down
When someone shares a URL on Facebook, its crawler (facebookexternalhit/1.1) fetches the page to build the preview card. Harmless, right? Until you add it up:
- Facebook re-crawls pages frequently to keep previews fresh
- Multiple shares trigger multiple crawl requests
- Viral content can mean hundreds of requests per minute
- Shared hosting plans can't handle the load

My friend's site was living this. Their shared hosting provider was threatening to suspend the account over resource usage. The culprit was Facebook's crawler, asking for pages faster than the server could serve them.
The fix: make the crawler wait its turn
Remember when Facebook's crawler last fetched a URL. If it's back too soon, answer with a quick "try later" instead of rendering the whole page.
Here's the core logic:
function check_facebook_throttle() {
$user_agent = $_SERVER['HTTP_USER_AGENT'] ?? '';
// Detect Facebook's crawler
if (strpos($user_agent, 'facebookexternalhit') === false) {
return; // Not Facebook, proceed normally
}
// Get the URL-specific cache key to allow multiple pages to be crawled
$request_uri = $_SERVER['REQUEST_URI'] ?? '/';
$cache_key = 'fb_throttle_' . md5($request_uri);
$throttle_seconds = get_option('fb_throttle_seconds', 60);
$last_access = get_transient($cache_key);
if ($last_access && (time() - $last_access) < $throttle_seconds) {
// Too soon. Tell Facebook to retry later.
// Use 429 (Too Many Requests) which is the proper rate-limiting status
header('HTTP/1.1 429 Too Many Requests');
header('Retry-After: ' . $throttle_seconds);
exit;
}
// Record this access
set_transient($cache_key, time(), $throttle_seconds * 2);
}
add_action('init', 'check_facebook_throttle', 1);
What to notice:
- Cache keys are per URL, so crawling page A doesn't block page B.
- It answers with HTTP 429 (Too Many Requests) instead of 503. 429 is the standard rate-limiting response.
- Access times live in WordPress's Transient API. No extra database tables, and it works with object caching if you have it.

What users taught me
Issue #1: the previews lost their images
A user messaged me on Facebook (yes, the irony) to say their images weren't showing up in Facebook previews. The plugin was throttling every Facebook crawler request, including the ones fetching og:image assets.
The fix was to spot image requests:
function is_image_request() {
$request_uri = $_SERVER['REQUEST_URI'] ?? '';
// Parse the path without query strings
$path = parse_url($request_uri, PHP_URL_PATH) ?? '';
// Check for image extensions
$image_extensions = ['jpg', 'jpeg', 'png', 'gif', 'webp', 'svg'];
$extension = strtolower(pathinfo($path, PATHINFO_EXTENSION));
return in_array($extension, $image_extensions, true);
}
Now image requests skip the throttle, and previews show their pictures again.
Issue #2: "What's actually happening?"
The same user asked how to see what the plugin was doing. Fair question. Throttling is invisible by default, so I added logging:
function log_throttle_event($action, $user_agent) {
if (!get_option('fb_throttle_logging', false)) {
return;
}
// Sanitize user agent to prevent log injection
$safe_user_agent = preg_replace('/[^\x20-\x7E]/', '', $user_agent);
$safe_user_agent = substr($safe_user_agent, 0, 100);
// Get real IP (check forwarded headers for proxies/CDNs)
$ip = $_SERVER['HTTP_X_FORWARDED_FOR'] ?? $_SERVER['REMOTE_ADDR'] ?? 'unknown';
// Take only the first IP if multiple are present
$ip = explode(',', $ip)[0];
$ip = filter_var(trim($ip), FILTER_VALIDATE_IP) ?: 'invalid';
$log_entry = sprintf(
"[%s] %s | UA: %s | IP: %s\n",
gmdate('Y-m-d H:i:s'),
$action,
$safe_user_agent,
$ip
);
error_log($log_entry, 3, WP_CONTENT_DIR . '/fb-throttle.log');
}
Security note: user agents and IPs are sanitized before they're logged, to prevent log injection attacks.
Pull request: it's not just Facebook
A contributor on GitHub pointed out that Facebook isn't the only pushy crawler. Their PR expanded detection to cover:
facebookexternalhit- Facebook link previewsFacebot- Facebook's general crawlerLinkedInBot- LinkedIn preview fetcherTwitterbot- Twitter/X card generatorWhatsApp- WhatsApp link previews
You can turn each bot on or off on its own in the settings.
What it does now
After months of community-driven development, here's the feature set:
Throttling
- Configurable delay between requests (default: 60 seconds)
- Per-URL throttling (crawling page A doesn't block page B)
- Per-bot throttle settings
- HTTP 429 response with
Retry-Afterheader (the standard rate-limiting response)
Visibility
- Optional logging to
wp-content/fb-throttle.log - Log rotation to prevent disk bloat
- Request details: timestamp, user agent, IP, action taken
Flexibility
- Whitelist specific paths (e.g.,
/wp-admin/) - Image request bypass
- WP-CLI commands for cache clearing

Installation
- Download from GitHub
- Upload to
wp-content/plugins/ - Activate in WordPress admin
- Configure under Settings -> FB Throttle
Or do it with WP-CLI (download first, then install):
# Download the plugin
curl -L -o fb-throttle.zip https://github.com/nadimtuhin/Facebook-Request-Throttle-WordPress-Plugin/archive/main.zip
# Install and activate
wp plugin install fb-throttle.zip --activate
Do you need it?
Good fit:
- Shared hosting with limited resources
- Sites that frequently go viral on social media
- WordPress installations seeing high crawler traffic in access logs
Not needed:
- Sites on dedicated servers with resources to spare
- Low-traffic blogs with occasional social shares
- If you're already using Cloudflare or similar with bot management
Lessons learned
Maintaining this plugin taught me a few things about open source:
Real users find real bugs. I'd never have hit the image bug in my own testing. I don't share enough content on Facebook.
Logging isn't optional. People need to see what's happening. "Trust me, it's working" isn't good enough.
Start simple, grow on demand. The first version was 30 lines. Now it's ~300, and every addition came from a real use case.
Security considerations
If you throttle crawlers, keep these in mind:
- User-agent spoofing: any bot can claim to be
facebookexternalhit. This plugin is for rate limiting, not security. - Log injection: always sanitize data before writing it to logs. The code above shows how.
- Race conditions: under very high load, several requests can pass the throttle check at the same time. For most sites, that's acceptable.
Real-world crawler behavior
Aggressive social crawlers don't care about your server's capacity. Without rate limiting at the application or edge level, a single shared post can swamp your upstream PHP workers with simultaneous scrape requests.
Rate limiting is one layer. On a busy site, a persistent object cache such as Redis keeps the plugin's transients out of the database, and a page cache can serve your OpenGraph tags without running full WordPress page queries.
Get involved
The plugin is MIT-licensed and contributions are welcome:
- Report issues: found a bug or edge case? Open an issue
- Contribute code: PRs welcome for new features or fixes
- Join the discussion: Discord Community
Resources
- GitHub Repository
- Facebook Sharing Debugger - Test how Facebook sees your URLs
- WordPress Transients API - How the plugin stores access times
Related posts
- opencode-9router: model discovery for OpenCode from a 9Router gateway
An OpenCode plugin that asks a 9Router gateway for its models at startup, fills in limits and capabilities, and caches the list. v1.0.0, 26 tests after a six-fix PR.
- express-simple-proxy: an Express proxy with hooks and typed errors
express-simple-proxy forwards Express routes to an upstream API with error codes, a beforeRequest hook and an onResponse callback. 183 tests, two runtime dependencies, and rough edges I found.
- devutils: 33 browser-only dev tools and what the README overstated
devutils runs JSON, JWT, SQL and Base64 tools in the browser with no network calls. I checked the README against the code: the privacy claim holds, the 35+ tool count did not.