test: characterization suite for timestamps with time zone - #25175
Open
adriangb wants to merge 4 commits into
Open
test: characterization suite for timestamps with time zone#25175adriangb wants to merge 4 commits into
adriangb wants to merge 4 commits into
Conversation
Adds `test_files/datetime/timestamps_timezone.slt`, ~1500 lines in 18 labelled sections that record what DataFusion actually does today with timezone-aware timestamps. DataFusion has a long tail of timezone correctness bugs, and the reason they keep recurring is structural: almost nothing in the test suite pins the behaviour down, so a change to timezone semantics can land without producing a single test diff. `SET datafusion.execution.time_zone` appears roughly 40 times in the whole sqllogictest corpus, nearly all of it in two files, and DST-boundary dates are essentially absent. This file does not change any behaviour. It writes the current behaviour down, including the parts that are known wrong or known to disagree with PostgreSQL -- those carry a comment saying so and a link to the issue, rather than being "fixed" here. When a fix lands, the diff is the specification of what changed. Every case is exercised as a literal and, where it matters, as a real CTAS column, since several of these behaviours only reproduce on columns. Sections cover literal typing under five session zones, `AT TIME ZONE`, casts in all four directions, round trips, tz-aware/tz-naive comparison (with `EXPLAIN` of the rewritten filter), `date_bin`/`date_trunc`/`date_part`, `to_char` / `to_local_time` / `to_timestamp*` / `from_unixtime`, `now`/`current_date`, interval arithmetic across both DST transitions, aggregates, joins and coercion, the US and EU DST boundaries by all four routes into a named zone, and a no-DST zone plus a half-hour zone. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 10, 2026
This was referenced Sep 10, 2026
Section 9 already had a `date_bin(..., origin)` case, but all of its data sits inside a single July day, so it never crosses a transition and the drift is invisible. Adds a case whose two rows are the same local time of day on either side of the America/Denver spring-forward transition, with the origin chosen so bins land on local midnight. The first row does; the second lands on 01:00 local. That is the bug in apache#25168 -- `date_bin` steps a fixed number of nanoseconds from a fixed instant, so it cannot track a local day that is 23 or 25 hours long. `date_trunc` is asserted alongside as a control: it gets both rows right, so the drift is a property of the origin arithmetic rather than of the data. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Upstream apache#24920 (merged 2026-09-06) adds field context to cast errors. The DST-boundary section pinned the full error chain, so 16 expectations started to fail once CI merged this PR onto current main -- 10 for America/Denver and 6 for Europe/Brussels: expected: DataFusion error: Arrow error: Cast error: Cannot cast timezone ... got: DataFusion error: Failed to cast field 'column1' from Timestamp(ns) to Timestamp(ns, "America/Denver") caused by Arrow error: Cast error: Cannot cast timezone to different timezone The wrapper chain is not what these cases pin. They pin which error class each path raises: the column path raises a Cast error, and the literal path raises a Parser error. So each expectation now matches only the `Arrow error: ...` tail. This uses the single-line `query error <regex>` form. In sqllogictest 0.29.1 an inline pattern goes through `Regex::is_match`, which is an unanchored search, while the multiline `----` form requires an exact match. The tail appears verbatim in both the old and the new message, so these expectations hold on either side of apache#24920 and will not break on the next wrapper change. Expectations and SQL are otherwise unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Author
|
Self-review (QA pass) of our own PR, at head The main hazard of this file: a pinned bug without a comment reads as an intended contract. So I checked each expectation that records a bug or a divergence from PostgreSQL. For each one I asked two questions: does a comment name the problem, and does it link the specific issue? Result. The file passes at this head, alone and merged with current PostgreSQL answers below come from Findings (most important first)1. CI: "Check License Header" fails at this head
Fix: put 2. Section 8 links the wrong issue, and several of its pins have no commentThe Section 8 header says: "This is the area corrupted by unwrap_cast", and links #25095. But no query in Section 8 has the shape of 25095. That issue is about a naive column compared with an aware literal under a non-UTC session time zone. Section 8 pins a different behaviour: a naive literal compared with an aware column is read in the time zone of the column, not in the session time zone. These expectations record that behaviour:
No issue in our list covers this exactly. The closest open issue is #13212 (" 3. Two
|
| Engine | Bin for 2024-03-11T12:30:00Z |
|---|---|
| DataFusion (this file) | 2024-03-11T01:00:00-06:00 |
PostgreSQL 15, date_bin |
2024-03-11 01:00:00-06 |
DuckDB 1.5.2, time_bucket |
2024-03-11 01:00:00-06 |
The pin is correct, and #25168 is the right link. But the comment must say that PostgreSQL and DuckDB give the same answer. Otherwise a fix can claim PostgreSQL parity that does not exist.
8. Low-priority notes
- The header "Related issues" list does not include Document that
date_binwith an explicit origin drifts an hour across a DST transition #25168. - Line 1181 says: "the naive side is treated as UTC". That is not true in general. I joined
cmp_denverwithnaive_col, with the session time zone unset. The naive2024-07-01 12:00:00matches2024-07-01T12:00:00-06:00, which is 18:00 UTC. So the naive side is read in the zone of the column. PostgreSQL 15 underUTCmatches the 12:00 UTC row. - The PR body says that the 40
SET datafusion.execution.time_zonestatements onmainare "nearly all" in two files. Onmainthey are in 9 files.set_variable.slthas 1 of them. - docs: document the
TIMESTAMP WITH TIME ZONEtype mapping and fix a stale comment #25171 adds docs that say DataFusion "discards theZ" in'...Z'::timestamptz. Lines 52-56 of this file say that the offset "is still honoured". Lines 90-95 confirm this:+05:30becomes06:30:00. The two texts must agree. The offset is applied, and only the time zone label is lost.
Merge-order collisions, checked against the file
I merged each PR into this branch at bafa661263 and ran this file, except where the diff shows no possible effect.
| PR (head) | Change | Result of the merged run | Expectations that change |
|---|---|---|---|
#25165 (3e42f8b102) |
AT TIME ZONE on an aware value returns a naive value |
2 failures | lines 323 and 331 (5b): Timestamp(ns, "Europe/Brussels") 2024-01-15T13:00:00+01:00 becomes Timestamp(ns) 2024-01-15T13:00:00, and the Denver query changes the same way |
#25163 (bdbfe1f8cd) |
date_part accepts timezone, timezone_hour and timezone_minute |
2 failures | lines 800 and 803: the two query error blocks now succeed |
#25161 (fb72121aef) |
from_unixtime uses the session time zone |
pass | none. The comment at lines 835-838 becomes stale |
#25173 (aeb4122371) |
generate_series precision |
not run | none: this file has no generate_series, and the PR changes only generate_series.rs and table_functions.slt |
#25171 (095e6d3a31) |
a code comment and docs | not run | none: no behaviour change. See the Z note in finding 8 |
current main |
includes #24920 and #25182 | pass | none. At a32121369b, #24920 broke at least 10 Section 16 error expectations. bafa661263 fixes them |
The updated #25165 adds a CASE limitation test in timestamps.slt. This file applies AT TIME ZONE to no CASE expression, so that test adds no collision.
What I checked and found correct
- The file passes at
bafa661263(cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests -- timestamps_timezone), and also with currentmainmerged in. - These comments name the problem and link the right issue: Sections 0-4 (TIMESTAMP WITH TIME ZONE can resolve to a timezone-naive type, and casting to it discards an existing timezone #25166), the
'+05:30'literal in 5a (AT TIME ZONE '+05:30'uses the opposite sign convention from PostgreSQL #25170), 5b, 6b and the first round trip in 7 (Casting existing timestamp to timestamp again strips timezone information #12218), the Denver and Kolkatadate_bin/date_trunccases (date_binanddate_truncdisagree on timezone-aware timestamps #25167), the origin drift (Document thatdate_binwith an explicit origin drifts an hour across a DST transition #25168),from_unixtime(Make from_unixtime aware of execution timezone #12892), and all of Section 16 (Cast fromTimestamp(_, None)to a named timezone errors on DST boundaries #25084). - I measured the PostgreSQL answers that the comments quote. They are correct for 5a, 5b, the composed idiom, 6b, the round trips, the Section 8 row counts, the
168 dayscase,to_timestamp,current_time, theUNIONtype, and all four Section 16 DST values. - The new Section 16 expectations match the stable
Arrow error: ...tail, so they still separate the column path (Cannot cast timezone to different timezone) from the literal path (error computing timezone offset).
🤖 Generated with Claude Code
…ization suite Upstream apache#25182 now checks license headers in `.slt` files, and main uses `#` on header lines 8 and 10. This file had empty lines there, so the license check fails. A self-review of this PR found annotations that misdescribe what is pinned: - Section 8 and two later comments linked 25095 (the unwrap_cast bug), but no query here has that shape. What they pin is 13212, the zone a tz-naive side is read in, and 25166, a tz-naive `'...Z'::timestamptz` literal. Links re-pointed; the header now says 25095 is not exercised. - The UNION comment said "the instants are preserved", but its second row is the 12:00Z row where 06:00Z was requested -- the Section 8 problem again. - The explicit-origin `date_bin` case was labelled KNOWN BUG. PostgreSQL 15.19 (`date_bin`) and DuckDB 1.5.2 (`time_bucket`) return the same `01:00-06`, so it is inherent to instant-based binning, not a divergence. Reframed. - Added missing links or divergence notes for `::timestamp` giving the UTC wall clock (12218), the `'+05:30'` sign convention (25170), `date_bin` versus `date_trunc` (25167), and the `timezone_hour` gap that PR 25163 fills. Comments and header only. No query or expected output changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #25175 +/- ##
==========================================
+ Coverage 81.60% 81.91% +0.31%
==========================================
Files 1123 1132 +9
Lines 408898 421314 +12416
Branches 408898 421314 +12416
==========================================
+ Hits 333670 345135 +11465
- Misses 55625 55771 +146
- Partials 19603 20408 +805 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
pull Bot
pushed a commit
to TCeason/arrow-datafusion
that referenced
this pull request
Sep 11, 2026
…pache#25161) ## Which issue does this PR close? - Closes apache#12892 ## Rationale for this change ### The bug, in SQL Set a session time zone. Every other date and time function honours it. `from_unixtime` does not. ```sql SET datafusion.execution.time_zone = 'America/Denver'; -- now() and to_timestamp() honour the session time zone SELECT arrow_typeof(now()), arrow_typeof(to_timestamp(1704110400)); -- Timestamp(ns, "America/Denver") | Timestamp(ns, "America/Denver") -- from_unixtime() does not SELECT arrow_typeof(from_unixtime(1704110400)), from_unixtime(1704110400); -- Timestamp(s) | 2024-01-01T12:00:00 ``` The last row is the defect. The user asks for a Denver session, but the value comes back as a bare wall clock with no zone. `2024-01-01T12:00:00` is the UTC clock, not the Denver clock. After this PR the same query returns the session time zone: ```sql SET datafusion.execution.time_zone = 'America/Denver'; SELECT arrow_typeof(from_unixtime(1704110400)), from_unixtime(1704110400); -- Timestamp(s, "America/Denver") | 2024-01-01T05:00:00-07:00 ``` The instant is the same. Only the declared type and the rendered offset change. ### What changes for a user - `from_unixtime(expr)` returns a value in `datafusion.execution.time_zone`. - `from_unixtime(expr, 'tz')` is unchanged. An explicit argument still wins. - With the session time zone unset, which is the default, nothing changes at all. - A value that no time zone can render is now an error, not a process panic. ### Field research: PostgreSQL and DuckDB Neither engine has a function named `from_unixtime`. The closest equivalent in both is `to_timestamp(<epoch>)`, so that is the comparison I ran. Both engines are timezone-aware there, and both render in the session time zone. **PostgreSQL 17.11** ```sql SHOW TimeZone; -- Etc/UTC SELECT to_timestamp(1704110400), pg_typeof(to_timestamp(1704110400)); -- 2024-01-01 12:00:00+00 | timestamp with time zone SET TimeZone = 'America/Denver'; SELECT to_timestamp(1704110400); -- 2024-01-01 05:00:00-07 SELECT extract(epoch from to_timestamp(1704110400)); -- 1704110400.000000 ``` PostgreSQL has no timezone-naive form of `to_timestamp`. A user must ask for one: ```sql SELECT to_timestamp(1704110400) AT TIME ZONE 'UTC'; -- 2024-01-01 12:00:00 | timestamp without time zone ``` **DuckDB v1.5.2** ```sql SELECT current_setting('TimeZone'), to_timestamp(1704110400), typeof(to_timestamp(1704110400)); -- America/Chicago | 2024-01-01 06:00:00-06 | TIMESTAMP WITH TIME ZONE SET TimeZone = 'America/Denver'; SELECT to_timestamp(1704110400); -- 2024-01-01 05:00:00-07 SELECT epoch(to_timestamp(1704110400)); -- 1704110400.0 ``` DuckDB defaults its session time zone to the operating system zone, so it is never naive. The naive form is `make_timestamp(<microseconds>)`, which returns `TIMESTAMP`. **What the comparison supports, and what it does not** - The type and the render rule of this PR agree with both engines. A one-argument epoch conversion is timezone-aware and follows the session time zone. - The default differs, and this PR keeps DataFusion's default. DataFusion leaves the session time zone unset, so `from_unixtime` stays naive out of the box. PostgreSQL and DuckDB always have a session zone. - The fixed-offset string form is not a meaningful comparison against PostgreSQL. `SET TimeZone = '+08:00'` in PostgreSQL means UTC-08:00, because PostgreSQL reads the string POSIX-style. DataFusion reads `'+08:00'` as UTC+08:00. That divergence is separate and is tracked in apache#25170. - The out-of-range comparison is not meaningful either. The three engines have three different timestamp ranges. PostgreSQL raises `ERROR: timestamp out of range` at `to_timestamp(-8334601211039)`. DuckDB renders the same value as `262145-12-31 (BC) 23:59:59.000448-04:56`. DataFusion sits between the two, and this PR makes it raise an error rather than panic. ## What changes are included in this PR? ### The fix - `FromUnixtimeFunc` carries an `Option<Arc<str>> timezone` taken from `config.execution.time_zone`, and implements `ScalarUDFImpl::with_updated_config`. `NowFunc` and the `to_timestamp*` functions already work this way. The single-argument form reports and produces `Timestamp(Second, <session tz>)` from both `return_field_from_args` and `invoke_with_args`. - `from_unixtime` moves to `make_udf_function_with_config!`, and its `expr_fn` gains the `@config` marker. The session `ConfigOptions` now reach the function at registration time, and again on every `SET` or `RESET`. - `FromUnixtimeFunc::new()` is deprecated since `56.0.0` in favour of `FromUnixtimeFunc::new_with_config()`. This mirrors the `to_timestamp*` constructors. `Default` still yields the previous timezone-naive behaviour. - The `from_unixtime` documentation description was wrong before this PR. It claimed an RFC3339 string with nanosecond precision and a `Z` suffix, but the function returns an Arrow `Timestamp(Second, ...)`, and the example directly below it shows a `-04:00` offset. The description now states what the function returns, and `docs/source/user-guide/sql/scalar_functions.md` is regenerated with `dev/update_function_docs.sh`. ### The guard Arrow renders a `Timestamp(Second, Some(tz))` through `chrono`'s `DateTime::naive_local`, which panics when the shifted value leaves `NaiveDateTime`'s range. The panic fires at render time, far from `from_unixtime`. The two-argument form can always reach it. Once the one-argument form applies a session zone, it can reach it too. So `invoke_with_args` now checks the bound and returns a `DataFusionError`: ``` Execution error: Cannot convert -8334601211039 to a timestamp in timezone "America/New_York" for function from_unixtime: the local date and time is outside the supported range ``` The check has three parts: 1. A timezone-naive result is never shifted, so the check returns at once. The full `NaiveDateTime` range stays usable, exactly as on `main`. 2. A fast path reads only the array's `min` and `max`. Inside `[NaiveDateTime::MIN + 86400, NaiveDateTime::MAX - 86400]` no zone offset can push a value out of range, so no per-value work happens. 3. Outside that window the check resolves the zone offset per value. `86400` is a deliberate over-bound. The largest offset in the whole IANA database is `Asia/Manila` at `-15:56:08` local mean time, and the largest positive one is `America/Metlakatla` at `+15:13:42`. The largest fixed offset Arrow's `Tz` parser accepts is `+23:59`, which is `86340` seconds. `+24:00` is rejected by the parser. ## What is the testing strategy for this PR? **New sqllogictest file** `datafusion/sqllogictest/test_files/from_unixtime_timezone.slt`, modelled on `to_timestamp_timezone.slt`. It asserts both `arrow_typeof` and the value for: - the session time zone unset, - a fixed offset (`+08:00`), - a named IANA zone (`America/Denver`), - the explicit two-argument form under a set session time zone, which must ignore it, - array (not constant folded) input, - `NULL` input, - equality of one instant across two time zones, - `RESET`, which must restore the timezone-naive behaviour. The `to_unixtime` / `from_unixtime` round trip from the issue is in the file. So are the out-of-range bounds, in both directions, for a named zone and a fixed offset zone, for the one-argument and the two-argument forms, and for array input. **New unit tests** in `datafusion/functions/src/datetime/from_unixtime.rs` cover `return_field_from_args` and `invoke_with_args` with a session time zone set, an explicit time zone that overrides it, and the out-of-range bound. Every bounds test renders its result. The unit tests go through `arrow::util::display::ArrayFormatter`, and the sqllogictest cases use `query P`. This is deliberate: the panic they guard against fires when the value is formatted, not when it is produced, so a test that only computes the value would pass either way. **Verified locally** - `cargo test -p datafusion-sqllogictest --test sqllogictests` — 506 files pass - `cargo test -p datafusion-functions --lib datetime::from_unixtime` — 10 tests pass - `cargo clippy -p datafusion-functions -p datafusion-sql --all-targets -- -D warnings` — clean - `cargo fmt --all --check` and `./ci/scripts/doc_prettier_check.sh` — clean **Verified against a `main` build.** I built `main` at the merge base and this branch as two `datafusion-cli` binaries and diffed their output. With the session time zone unset, eleven probes are byte-identical, including the error paths. The two-argument form is byte-identical across 140 pairs of value and zone. The full detail is in a separate comment on this PR. **Cost of the guard.** `SELECT count(from_unixtime(v)) FROM generate_series(1, 100000000) t(v)`, three runs each, `ci` profile: 3.95 s / 3.65 s / 3.07 s with the zone unset, and 3.09 s / 3.78 s / 3.33 s with `America/Denver`. The two extra kernel passes sit inside the noise. ## Are there any user-facing changes? Yes, four. **1. The behaviour change, which is the bug fix.** When `datafusion.execution.time_zone` is set, `from_unixtime(expr)` returns `Timestamp(Second, <session tz>)` instead of `Timestamp(Second, None)`. The epoch value is unchanged. Only the declared type and the rendered offset differ. With the default unset session time zone nothing changes. **2. An API deprecation.** `FromUnixtimeFunc::new()` is deprecated since `56.0.0` in favour of `FromUnixtimeFunc::new_with_config()`. `Default::default()` stays available and behaves like the old `new()`. **3. A public factory signature change.** `datafusion::functions::datetime::from_unixtime()` becomes `from_unixtime(&ConfigOptions)`, because it now uses `make_udf_function_with_config!`. This breaks callers of that factory. It is the established convention for config-aware functions in this module: `now` and all five `to_timestamp*` functions already have this signature on `main`, with no zero-argument shim. The `expr_fn` wrapper `datafusion::functions::expr_fn::from_unixtime(expr)` keeps its signature. **4. A new error in place of a panic.** See below. ### Out of range values Arrow renders a `Timestamp(Second, Some(tz))` value by a shift of the UTC instant by the zone's offset. It does that with `chrono`'s `DateTime::naive_local`, which **panics** when the shifted value leaves `NaiveDateTime`'s range. The panic fires when the value is formatted, not when it is cast, so it surfaces far from where it starts: ```sql SELECT from_unixtime(-8334601211039, 'America/New_York'); -- thread 'main' panicked at chrono-0.4.45/src/datetime/mod.rs:579:14: -- Local time out of range for `NaiveDateTime` ``` This PR replaces that panic with an error. The check applies to both forms. Its deliberate limits: - **Timezone-naive results are unaffected.** They are never shifted, so the full `NaiveDateTime` range stays usable, exactly as on `main`. - **A value that is out of range in UTC as well is left to the cast**, which already reports it (`Cast error: Failed to convert 8210266876800 to datetime for ...`). No error message changes. - **A query that consumed an out-of-range value without a render now returns an error.** On `main` such a value is only fatal at render time, so `SELECT to_unixtime(from_unixtime(-8334601211039, 'America/New_York'))` returns `-8334601211039` today and errors after this PR. Every query that produced a rendered result keeps its behaviour. ### Known interaction with issue 16594 This removes the panic reproducer in apache#16594, which I verified directly. `from_unixtime(-8334601211038 - 1, 'America/New_York')` errors instead of a panic, and `from_unixtime(8210266876799 + 1, 'America/New_York')` keeps its existing cast error. It does **not** close that issue. The issue asks that *all* `Int64` values convert, and that is not reachable while `chrono::NaiveDateTime` renders the value. The underlying defect is an unguarded `naive_local()` in the Arrow timestamp-with-timezone formatter. It stays reachable with no `from_unixtime` at all: ```sql SELECT arrow_cast(-8334601211039, 'Timestamp(Second, Some("America/New_York"))'); -- same panic SET datafusion.execution.time_zone = 'America/New_York'; SELECT to_timestamp_seconds(-8334601211039); -- same panic, on main and on this branch ``` A proper guard belongs in Arrow. The check here only stops `from_unixtime` from handing the formatter a value the formatter cannot render. ### Merge order with PR 25175 apache#25175 pins today's `from_unixtime` behaviour in a characterization test file. **No assertion in it breaks.** Its SECTION 10 cases run with the session time zone unset, and this PR keeps `Timestamp(Second, None)` in that case. I replayed both assertions on this branch and they still pass: ``` Timestamp(s) 2024-07-01T00:00:00 Timestamp(s, "America/Denver") 2024-06-30T18:00:00-06:00 ``` The comment above them says `from_unixtime` "ignores `datafusion.execution.time_zone`". That comment goes stale when this PR lands, so whichever merges second must reword it. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
This PR closes no issue and fixes no bug. It adds tests only.
Related to these issues. For each issue, the file records what DataFusion does today, so a fix shows up as a test diff:
AT TIME ZONEand::timestampon aware valuesTimestamp(_, None)to a named timezone errors on DST boundaries #25084: casts into a named zone fail at a DST boundaryTIMESTAMP WITH TIME ZONEresolves to a naive type (first part only, see below)date_binanddate_truncdisagree on timezone-aware timestamps #25167:date_binanddate_truncdisagreedate_binwith an explicit origin drifts an hour across a DST transition #25168:date_binwith an explicit origin drifts across DSTAT TIME ZONE '+05:30'uses the opposite sign convention from PostgreSQL #25170: the sign ofAT TIME ZONE '+05:30'These issues are also related, but no expectation in the file exercises them yet. The self-review comment on this PR gives a query for each:
unwrap_castdrops the time zone shift. Not exercised by this file; the Section 8 header explains why.from_unixtimeignores the session time zonetimestamptzreplaces the zone of an aware valueBackground: #10368, #10602, #13212.
Rationale for this change
Here is one behaviour that this file pins.
day_denverhas four rows at 00:00, 06:00, 12:00 and 18:00 UTC on 2024-07-01, with the typeTimestamp(ns, "America/Denver"):tsdate_bin(INTERVAL '1 day', ts)date_trunc('day', ts)2024-06-30T18:00:00-06:002024-06-30T18:00:00-06:002024-06-30T00:00:00-06:002024-07-01T00:00:00-06:002024-06-30T18:00:00-06:002024-07-01T00:00:00-06:002024-07-01T06:00:00-06:002024-06-30T18:00:00-06:002024-07-01T00:00:00-06:002024-07-01T12:00:00-06:002024-06-30T18:00:00-06:002024-07-01T00:00:00-06:00A user asks both functions for the day of each row, and gets two different answers.
date_binreturns the UTC day, shown as 18:00 local time.date_truncreturns the local Denver day. So aGROUP BYon each function gives different groups. The file pins this and links #25167. When a fix lands, these rows change, and the diff shows the new contract.What this file is for. It records what DataFusion does today with timestamps that carry a time zone, bugs included. A change to that behaviour then makes a test fail, so a reviewer sees the change. Where today's answer is a bug or differs from PostgreSQL, a comment above the query says so and links the issue.
Why time zone bugs recur
mainbefore this PR, the sqllogictest corpus has 40SET datafusion.execution.time_zonestatements, in 9 files. The largest file has 10.test_files/pg_compat/) has no time zone coverage.The companion PR #25164 checks the subset where DataFusion and PostgreSQL agree, against a real PostgreSQL. This PR pins everything else. So this file changes when a fix lands, by design.
What changes are included in this PR?
This PR adds one test file and changes no production code:
datafusion/sqllogictest/test_files/datetime/timestamps_timezone.slt(1553 lines, 18 sections).Many cases run on a literal and also on a real CTAS column, because some behaviours occur only on columns.
arrow_typeofand the value of::timestamp,::timestamptz,TIMESTAMP '...'andTIMESTAMP WITH TIME ZONE '...', with the session time zone unset,+00:00,+05:30,America/DenverandEurope/BrusselsAT TIME ZONEon naive and aware values, and the sign of'+05:30'EXPLAINof the filterdate_bin,date_trunc,date_partandextracton aware values, and thedate_binorigin driftto_char,from_unixtime,to_unixtime,to_timestamp*andto_local_timenow,current_date,current_timeandmake_dateunder a session time zone'1 day'against'24 hours'MIN,MAX,GROUP BY,ORDER BY,DISTINCT; joins on aware keys, mixed zones included;UNION,CASE,COALESCEandgreatestacross mixed zonesAmerica/DenverandEurope/Brussels, by column cast,AT TIME ZONE, string literal and::timestamptzAmerica/Phoenix(no DST) andAsia/Kolkata(half-hour offset)How we made the expected output
--completemode.Harness notes
Arrow error: ...tail of each message. The wrapper text above that tail changed in feat: include source field context in cast errors #24920, and these cases do not pin it.Merge order: merge this PR after the fixes that it pins
We recommend that this PR merges after the fix PRs below. Then each fix PR stays small, and this PR records the fixed behaviour in one rebase. If this PR merges first, each fix PR must also edit this file.
We merged each PR into this branch at
bafa661263and ran this file:3e42f8b102)AT TIME ZONEon an aware value returns a naive valueTimestamp(ns, "Europe/Brussels") 2024-01-15T13:00:00+01:00becomesTimestamp(ns) 2024-01-15T13:00:00, and the Denver query changes the same way. Delete theDIVERGES FROM POSTGRESQL (type, not instant)comment.bdbfe1f8cd)date_partacceptstimezone,timezone_hourandtimezone_minutequery errorblocks succeed. Make them value queries, and delete the "GAP vs PostgreSQL" comment.fb72121aef)from_unixtimeuses the session time zonefrom_unixtimeignores the session time zone.aeb4122371)generate_seriesprecisiongenerate_series.095e6d3a31)mainIn Section 5, only 5b applies
AT TIME ZONEto an aware value. 5a, 5c, the composed idiom, Section 12 and Section 16 apply it to naive values, and #25165 does not change those.License header. #25182 (merged 2026-09-11) adds
*.sltto the license header check. The file now has#on header lines 8 and 10, asmainexpects (fixed in6df7863bed).What is the testing strategy for this PR?
This PR adds tests only. Checks at head
bafa661263:cargo test --profile ci -p datafusion-sqllogictest --test sqllogictests -- timestamps_timezone: pass.mainmerged in: pass.Field research: measured answers where a pinned behaviour diverges
DataFusion answers are the pinned expectations, except where the row says
datafusion-cli. PostgreSQL answers come frompostgres:15. DuckDB answers come from DuckDB 1.5.2. "Unset /UTC" means: the DataFusion session time zone is unset, and PostgreSQL and DuckDB useUTC.'2024-07-01 12:00:00Z'::timestamptz(0)UTCTimestamp(ns)(naive)timestamp with time zoneTIMESTAMP WITH TIME ZONETIMESTAMP '2024-07-01 12:00:00' AT TIME ZONE '+05:30'(5a)UTC2024-07-01T12:00:00+05:30(06:30 UTC)2024-07-01 17:30:00+00Unknown TimeZone '+05:30'2024-07-01 12:00:00ZAT TIME ZONE 'Europe/Brussels'(5b)UTCTimestamp(ns, "Europe/Brussels")2024-07-01T14:00:00+02:00timestamp without time zone2024-07-01 14:00:00TIMESTAMP2024-07-01 14:00:00('2024-07-01T18:00:00Z'::timestamptz AT TIME ZONE 'America/Denver')::timestamp(5)UTC2024-07-02T00:00:002024-07-01 12:00:002024-07-01 12:00:00::timestamp(6b)America/Denver2024-07-01T18:00:002024-07-01 12:00:002024-07-01 12:00:00WHERE ts = '2024-07-01 12:00:00'on rows at 00, 06, 12, 18 UTC (8)UTCWHERE ts > '2024-07-01 06:00:00'on the same rows (8)UTCWHERE ts = '2024-07-01T06:00:00Z'::timestamptzon the same rows (8)UTCdate_trunc('day', ts)for 00:00 UTC on a Denver value (9)UTC2024-06-30T00:00:00-06:002024-07-01 00:00:00+002024-07-01 00:00:00+00date_bin(INTERVAL '1 day', ts)for the four Denver rows (9); PostgreSQL and DuckDB with origin1970-01-01 00:00:00+00America/Denver2024-06-30T18:00:00-06:002024-06-30 18:00:00-062024-06-30 18:00:00-06(time_bucket)date_trunc('day', ...)anddate_bin(INTERVAL '1 day', ...)for2024-01-01 12:00:00+05:30(9)Asia/Kolkata2024-01-01T00:00:00+05:30and2024-01-01T05:30:00+05:302024-01-01 00:00:00+05:30and2024-01-01 05:30:00+05:302024-01-01 00:00:00+05:30and2024-01-01 05:30:00+05:30date_binwith origin2024-03-01T07:00:00Z, for2024-03-11T12:30:00Z(9)America/Denver2024-03-11T01:00:00-06:002024-03-11 01:00:00-062024-03-11 01:00:00-06date_binwith a1 monthstride (9)UTCtimestamps cannot be binned into intervals containing months or yearsdate_part('timezone_hour', ts)in July (9)America/DenverDate part 'timezone_hour' not supported-6-6from_unixtime(1719792000); PostgreSQL and DuckDBto_timestamp(1719792000)(10)UTCTimestamp(s)2024-07-01T00:00:00timestamp with time zone2024-07-01 00:00:00+00TIMESTAMP WITH TIME ZONE2024-07-01 00:00:00+00current_time(11)America/Denver/UTCTime64(ns)time with time zone(TIMESTAMP '2024-01-15 12:00:00' AT TIME ZONE 'America/Denver') = (TIMESTAMP '2024-07-01 12:00:00' AT TIME ZONE 'America/Denver') - INTERVAL '168 days'(12)UTCtrueffalseAmerica/Denvertrue(datafusion-cli)ttrue'2024-11-03 01:30:00'::timestamptz, a repeated local time (16)America/Denvererror computing timezone offset2024-11-03 01:30:00-072024-11-03 01:30:00-07'2024-03-10 02:30:00'::timestamptz, a local time that does not exist (16)America/Denvererror computing timezone offset2024-03-10 03:30:00-062024-03-10 03:30:00-06'2024-10-27 02:30:00'::timestamptz(16)Europe/Brusselserror computing timezone offset2024-10-27 02:30:00+012024-10-27 02:30:00+01'2024-03-31 02:30:00'::timestamptz(16)Europe/Brusselserror computing timezone offset2024-03-31 03:30:00+022024-03-31 03:30:00+022024-07-01T12:00:00Z::timestampon a Kolkata value (17)Asia/Kolkata2024-07-01T12:00:002024-07-01 17:30:00What the measurements show:
date_binagrees with PostgreSQL and DuckDB.date_truncdiffers only when the session time zone is not the zone of the value.date_bin/date_truncdisagreement also exists in PostgreSQL and DuckDB underAmerica/Denver: all three engines return 18:00 local time fordate_bin, and local midnight fordate_trunc.date_binorigin drift is the same in all three engines. A fix for Document thatdate_binwith an explicit origin drifts an hour across a DST transition #25168 moves DataFusion away from PostgreSQL.168 dayscase depends on the scope of the day interval. DataFusion uses the zone of the value, and PostgreSQL and DuckDB use the session time zone. The engines agree when the two zones are the same.AT TIME ZONE, string literal and::timestamptz. PostgreSQL and DuckDB resolve all four values the same way. The file cites Resolve DST-ambiguous and nonexistent local times when casting timestamps to a named timezone arrow-rs#11038 as the arrow-rs fix for Cast fromTimestamp(_, None)to a named timezone errors on DST boundaries #25084.Are there any user-facing changes?
No. This PR adds one test file. It changes no production code, no public API and no behaviour.
🤖 Generated with Claude Code