Separate source time from discovery time
Your objectiveExplain what an ingestion timestamp actually proves.
Understand the idea.
A source-supplied time and the time your collector first sees a record answer different questions. The first is a statement by the source; the second is your system’s observation. Neither automatically tells you when the underlying event happened. Preserve unknown times as unknown. Substituting a convenient time can make a stale article look like a new event.
A worked example.
Your collector discovers an article on Thursday. Its publication date is Monday, and it discusses a launch that occurred the previous month. Thursday is a valid discovery time. It is not a valid replacement for either the source date or event date.
Try the reasoning.
Make a table with event time, source publication time, discovery time and TLB release time. Fill only the values established by this scenario. Mark the rest unknown rather than guessing.
Carry it into practice.
Use separate fields in an incident or editorial record and retain the original source value.
Read the reference: IETF · HTTP semantics ↗