Alt text:
It's not just time zones and leap seconds. SI seconds on Earth are slower because of relativity, so there are time standards for space stuff (TCB, TGC) that use faster SI seconds than UTC/Unix time. T2 - T1 = [God doesn't know and the Devil isn't telling.]
You should care if it's wrong.
at the resolution of clock drift in milliseconds when I'm running reports that are, at most, only specific to the day?
Clock drift? No. Time zones? Probably.
not really time zones either outside the edge case where a data point exists within delta of midnight so that the time zone drift would result in a date change
Time zones change. Relative times without time zones don't make sense.