Skip to content

add tumble feature - #794

Open
ericyuanhui wants to merge 1 commit into
LadybugDB:mainfrom
ericyuanhui:tumble_feature
Open

add tumble feature#794
ericyuanhui wants to merge 1 commit into
LadybugDB:mainfrom
ericyuanhui:tumble_feature

Conversation

@ericyuanhui

Copy link
Copy Markdown
Contributor

Goal

Add a Cypher-native TUMBLE reading clause that assigns every input row to a
fixed, non-overlapping event-time window. The clause exposes the window start
and end timestamps so that existing WITH, GROUP BY-equivalent aggregation,
and CREATE ... TABLE AS facilities can consume them.

The compatibility target is the row-assignment portion of Feldera SQL
TUMBLE(TABLE input, DESCRIPTOR(time_col), interval), not its continuous
incremental execution model.

Proposed Syntax

TUMBLE ON <timestamp-expression>
       EVERY <fixed-interval-expression>
       YIELD <window-start-name>, <window-end-name>

Example:

MATCH (s:CampusUser)-[:CampusUserHasOnlineSession]->(e:EventNetworkOnline)
TUMBLE ON e.login_time EVERY interval('1 day')
YIELD window_start, window_end
WITH s.id AS entity_id, window_start, window_end,
     count(DISTINCT e.user_ipv4) AS daily_ipv4_count
RETURN entity_id, window_start, window_end, daily_ipv4_count;

TUMBLE consumes the rows produced by the preceding reading clauses and keeps
all bindings in scope. YIELD introduces exactly two new timestamp bindings.
The yielded names must be distinct and must not shadow an in-scope name.

The grammar makes whitespace around the comma optional, so both forms are
equivalent:

YIELD window_start, window_end
YIELD window_start , window_end

Required Semantics

  • The time expression is evaluated once for each input row and must have one
    of Ladybug's timestamp types: TIMESTAMP, TIMESTAMP_SEC, TIMESTAMP_MS,
    TIMESTAMP_NS, or TIMESTAMP_TZ. The yielded columns have the same logical
    type as the input expression.
  • TIMESTAMP_TZ uses the represented instant and Unix-epoch alignment;
    interval('1 day') means exactly 24 hours, not a session-local civil day.
  • DATE is not accepted as a time expression in Phase 1 and must be cast
    explicitly. INTERVAL is accepted only as EVERY.
  • The interval expression must be a non-null, compile-time constant positive
    INTERVAL of a fixed duration. Phase 1 supports microseconds through days;
    month and year components are rejected because they are calendar-dependent.
  • To preserve the timestamp logical type without silently losing a window
    boundary, TIMESTAMP_SEC requires a whole-second interval and
    TIMESTAMP_MS requires a whole-millisecond interval. TIMESTAMP,
    TIMESTAMP_NS, and TIMESTAMP_TZ accept any fixed microsecond interval.
  • The alignment origin is Unix epoch 1970-01-01 00:00:00 UTC.
  • For an input timestamp t and window size w, the window start is the
    greatest origin + k*w not later than t; the end is start + w.
  • Membership uses a half-open range: [window_start, window_end). A timestamp
    exactly at a boundary belongs to the later window.
  • A NULL time expression produces NULL for both yielded columns. It does
    not remove the row.
  • Output order is unchanged. This operator neither aggregates nor deduplicates
    rows.
  • The result is deterministic for a fixed database snapshot and query.

The implementation normalizes timestamp input to the existing canonical
microsecond representation for bucket arithmetic, then converts both results
back to the input logical timestamp type. Existing timestamp conversion rules,
including the existing precision behavior of TIMESTAMP_NS, remain unchanged.
For this operator's floor-bucket calculation, a pre-epoch TIMESTAMP_NS value
with a sub-microsecond remainder is normalized toward negative infinity.

Explicit Non-goals

  • A stream source, watermark, event-time progress, late-event policy, or
    processing-time trigger.
  • Sliding, hopping, session, or calendar-month windows.
  • Feldera TABLE(TUMBLE(...)) SQL syntax.
  • Analytical windows: OVER, PARTITION BY, ORDER BY, ROWS, and RANGE.
  • Automatic refresh of CREATE NODE TABLE AS / CREATE REL TABLE AS results,
    triggers, subscriptions, or change-data capture.
  • Automatic rolling P95 maintenance.

Expected Usage with Existing Ladybug Features

The daily aggregate can be snapshotted with existing CTAS support:

CREATE NODE TABLE DailyFeature AS
MATCH (s:CampusUser)-[:CampusUserHasOnlineSession]->(e:EventNetworkOnline)
TUMBLE ON e.login_time EVERY interval('1 day')
YIELD window_start, window_end
RETURN concat(string(s.id), ':', string(window_start)) AS id,
       s.id AS entity_id,
       window_start,
       window_end,
       count(DISTINCT e.user_ipv4) AS f_v;

This is a one-time snapshot. Future writes to EventNetworkOnline do not
automatically refresh DailyFeature; an application or scheduled job must
rebuild or explicitly update affected daily rows.

Acceptance Criteria

  • The example parses, binds, plans, and returns a start/end pair for every
    input row.
  • Daily grouping produces one group per entity and day, including boundary
    timestamps at midnight.
  • Tests cover sub-day intervals, multi-day intervals, timestamps before the
    epoch, null timestamps, invalid types, zero/negative intervals, calendar
    intervals, alias conflicts, and chained MATCH/WITH scopes.
  • EXPLAIN identifies the logical and physical TUMBLE operator.
  • Existing queries and table functions retain their current behavior.
  • Existing queries that use tumble as an unescaped identifier continue to
    parse. TUMBLE and EVERY remain contextual/non-reserved keywords outside
    the new clause syntax.

Signed-off-by: ericyuanhui <285521263@qq.com>
@ericyuanhui

Copy link
Copy Markdown
Contributor Author

This is a proposal to implement a window function similar to time_bucket. From my preliminary understanding, only TigerGraph supports this feature at present. Discussion for development is welcome. Thank you.

@adsharma

Copy link
Copy Markdown
Contributor

TUMBLE is not a part of open cypher. When we add non-cypher extensions, we generally follow duckdb.

FROM TUMBLE(table, ts, INTERVAL '10' MINUTE). // streaming engines
GROUP BY time_bucket(INTERVAL '10 minutes', ts) // duckdb

Will the duckdb variant work for you?

@ericyuanhui

Copy link
Copy Markdown
Contributor Author

TUMBLE is not a part of open cypher. When we add non-cypher extensions, we generally follow duckdb.

FROM TUMBLE(table, ts, INTERVAL '10' MINUTE). // streaming engines
GROUP BY time_bucket(INTERVAL '10 minutes', ts) // duckdb

Will the duckdb variant work for you?

Either the streaming engine style or the DuckDB style works for me. What matters is that Ladybug supports this capability. So I need to implement this feature following either the DuckDB syntax or the streaming style, right?

@adsharma

Copy link
Copy Markdown
Contributor

Yes, the feature looks valuable. Let's use the duckdb style for consistency with the rest of UDFs.

@ericyuanhui

Copy link
Copy Markdown
Contributor Author

Yes, the feature looks valuable. Let's use the duckdb style for consistency with the rest of UDFs.

Are there any existing syntax implementations aligned with DuckDB or streaming engines in the current project? I'd like to refer to them. Could you provide an example?

@adsharma

Copy link
Copy Markdown
Contributor

#694 (comment)

is an example where we implemented functionality to be compatible with duckdb. It was more semantics than syntax.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants