- Treat a feed as untrusted input
- Build tests from the boundary outward
Treat a feed as untrusted input
A source may be useful and still return a malformed, unexpectedly large or hostile document. A feed collector therefore needs a boundary between receiving bytes and accepting records. The boundary should be explicit, tested and independent of how familiar the publisher’s name looks.
The goal is not to reject every unusual document. It is to define the resources the parser may consume and the behaviors it may perform before the application treats the result as a candidate for further processing.
Bound the transfer and the expansion
A compressed response can be small on the network and much larger after decompression. Measure both quantities. Set a maximum for received bytes and another for the decoded document, then fail with a distinct error when either boundary is exceeded.
Choose limits from an operational budget rather than increasing them without end. A larger legitimate feed may need a different adapter, pagination or a scoped exception. It should not automatically cause every source to receive an unlimited parser budget.
Know what the XML library permits
Python’s XML security guidance describes risks associated with untrusted XML, including entity expansion and external-resource behavior. Use a parser configuration or library that rejects dangerous features for this use case. Do not assume that detecting the letters XML is a security validation.
The transport boundary also matters: require the intended secure scheme, reject credentials embedded in URLs and avoid following redirects into private destinations. A public feed URL is not permission for the collector to reach unrelated internal services.
Validate records after parsing
Successful parsing establishes that a document could be interpreted. It does not establish that every item has a usable article URL, a sensible timestamp or a meaningful title. Validate those fields separately. Reject a missing article link instead of silently substituting the feed’s own URL.
Normalize carefully. Removing known tracking parameters can help recognize the same article, but deleting every query parameter can collapse different pages into one. Preserve the original source identity and record how canonicalization was performed.
Make refusal a visible result
Use separate outcomes for oversized responses, unsafe XML, invalid records, TLS failures, rate limits and timeouts. An operator should not have to infer the cause from a generic red indicator. Each outcome should have an appropriate next action and retry policy.
A size refusal does not mean the publisher is malicious. An unsafe-XML refusal does not establish that the feed’s articles are false. These are statements about a processing boundary, not editorial judgments about the publisher.
Build tests from the boundary outward
Start with a small valid feed. Add fixtures for missing links, invalid timestamps, duplicate URLs, a forbidden entity, a compressed oversized response and a redirect to a non-public address. Test that refusal happens before any unwanted resource is fetched or stored.
Finally, run the collector against a representative real source set and report both accepted and rejected outcomes. A larger successful count is useful only when the protections and the meaning of success remain intact.
Read beyond this page.
Recorded source-check date: 10 Sep 2026. A link is not, by itself, evidence that every claim has been independently verified.
Changes & version history
Version 3 · 15 Sep 2026
Scheduled release of checksum-bound AI-assisted editorial review
Version 2 · 10 Sep 2026
Checksum-bound editorial review scheduled for release
Version 1 · 10 Sep 2026
Source-linked private review edition