
Parse Leniently, Normalise Early
You will meet RSS 2.0, RSS 1.0, Atom and JSON Feed, plus extensions such as content:encoded, media: and dc: (see RSS namespaces). Do not let the rest of your code know about any of that. Parse each format once into a single internal model and work only with that:
{
"id": "stable identifier, never empty",
"url": "https://example.com/post",
"title": "Plain text title",
"published": "2026-11-10T08:30:00Z",
"updated": null,
"author": "Name",
"summary": "Plain text, short",
"content_html": "<p>Sanitised HTML</p>",
"image": "https://example.com/cover.jpg",
"enclosures": [{"url": "...", "type": "audio/mpeg", "length": 123}],
"source": {"feed": "https://example.com/feed.xml", "title": "Example"}
}Real-world feeds are often not well-formed. A single unescaped & breaks a strict XML parser. Try strict parsing first and fall back to a recovering parser (libxml's recover mode, or an HTML5 parser for the worst cases), and log which sources needed the fallback. Convert the body to UTF-8 using the encoding from the XML declaration or the Content-Type header before parsing.
Give Every Item a Stable ID
Use the feed's own ID first: guid, Atom id or JSON Feed id. When it is missing, fall back to the normalised link, and only then to a hash of title and date. Store IDs per source, because two feeds may use the same short IDs like 1842. Normalise links before comparing: lowercase the host, drop www., a trailing slash, fragments and tracking parameters such as utm_*. Readers that skip this step are the reason users see duplicate posts.
Treat Dates With Suspicion
- RSS uses RFC 822 (
Tue, 10 Nov 2026 08:30:00 +0000), Atom and JSON Feed use RFC 3339. Expect both in either format, plus local-language month names and missing time zones. - Assume UTC when no zone is given, and cap dates in the future at the moment you first saw the item.
- When an item has no date at all, use the time you first saw it, and keep that value: do not let undated items jump to the top on every fetch.
Poll Politely and Efficiently
Polling thousands of feeds is mostly about not downloading the same bytes again. Store the ETag and Last-Modified headers of every response and send them back as If-None-Match and If-Modified-Since. A 304 Not Modified answer means there is nothing to parse.
$h = ["User-Agent: MyReader/1.2 (+https://example.com/bot; 42 subscribers)"];
if ($feed->etag) $h[] = "If-None-Match: " . $feed->etag;
if ($feed->lastModified) $h[] = "If-Modified-Since: " . $feed->lastModified;
curl_setopt_array($ch, [CURLOPT_HTTPHEADER => $h, CURLOPT_ENCODING => "", CURLOPT_TIMEOUT => 20]);
$body = curl_exec($ch);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
if ($status === 304) { $feed->touch(); return; }
if ($status === 410) { $feed->disable(); return; }
if ($status === 429) { $feed->backoff($retryAfter); return; }Beyond that:
- Follow permanent redirects for real. After a
301or308, update the stored feed URL. Keep following302and307without updating. - Adapt the interval. A feed that publishes ten times a day deserves a check every 15 to 30 minutes; one that publishes monthly, a few times a day at most. Back off exponentially after errors and respect
Retry-After. - Identify yourself. A User-Agent with a contact URL, and for readers the number of subscribers, lets publishers see who you are instead of blocking an unknown bot.
- Prefer push when offered. If a feed links a WebSub hub, subscribe to it and poll rarely as a safety net.
- Accept compression and set sensible limits: a timeout, a maximum body size and a maximum decompressed size.
Security: Two Things You Must Get Right
Server-side request forgery
If users can submit feed URLs, your server will fetch whatever they type, including http://169.254.169.254/ or http://localhost:6379/. Allow only http and https on standard ports, resolve the host name yourself and refuse private, loopback and reserved addresses, connect to the IP you checked so DNS cannot change in between, and repeat the check for every redirect. Disable external entities in your XML parser to prevent XXE.
Untrusted HTML
Feed content is HTML written by strangers. Sanitise it with an allowlist of tags and attributes before showing it: no <script>, no event handlers, no javascript: URLs, no inline styles that can overlay your page. Make relative links absolute using the item or feed URL as base, add rel="noopener" to links, and consider proxying images so readers do not leak their IP address to every publisher.
Turning New Items Into Events
Many integrations only need "something new appeared": a webhook, a chat message, a database row. After each fetch, compare the item IDs with the ones you stored, emit an event for each unseen ID and store it in the same transaction. Include the stable ID in the webhook payload so the receiver can ignore repeats, and never fire events for the first fetch of a new source, or users get fifty notifications at once.
Don't Want to Build All This?
Our RSS to JSON API fetches any feed, handles all four formats and returns normalised JSON, with CORS for use in the browser. For combining sources there is the feed merger, and the health check shows how a server behaves for conditional requests before you add it to your poller.
Questions
Can an RSS feed trigger a webhook?
Not by itself: a feed is a file that clients fetch, so something has to watch it. A small poller, an automation service or a WebSub subscription can detect new items and call your webhook. Use the item's stable ID to make sure each item triggers only once, also after restarts.
How can I avoid unsafe HTML in feed entries?
Never insert feed HTML into your pages as it is. Run it through an HTML sanitiser with an allowlist, such as DOMPurify in the browser or HTML Purifier in PHP, strip scripts, event handlers and dangerous URLs, and show plain text where you do not need formatting. Titles in particular should always be treated as plain text.
How often should I poll a feed?
Base it on the feed's own rhythm: roughly as often as it publishes, within limits of every 15 minutes and once a day. Conditional requests make frequent checks cheap for both sides. Respect the ttl element in RSS and Cache-Control headers as lower bounds when they are present.
Sources and Further Reading
- feedparserfeedparser documentation
- Server Side Request ForgeryOWASP
- RFC 9110: HTTP SemanticsIETF
- WebSubW3C Recommendation