Repository navigation
Commit a720ff0
LAV-4306: redact customer identifiers and storage paths from analytics (#4502)
Anonymize every identifier and plain string field in the cloned telemetry AST,
including quoted/qualified names, aliases, columns, session-variable references,
storage URLs, stage paths, comments and secrets. Keep typed literal markers and
safe structural option keys, and anonymize unknown option keys. Only exact,
fully redacted error templates survive; other free-form messages become the
safe category "SQL error". Preserve wire-derived codes, SQLSTATE and all existing
categorical dimensions, the event name, payload schema, and batch cardinality.
Update ADR 045 and receiver tests. Identify batch events through non-sensitive
query IDs, explicitly wait for the asynchronous flush before counting events,
and verify neither successful batch children nor a successful pre-failure child
produce independent events. Execution results, returned column names and
customer-bearing wire errors retain their operational semantics. No operational
credential-masking path, intercepted-command recording, parser-fallback path or
Python telemetry producer changes.
test_edit_waiver: tests/telemetry/test_telemetry_events.py
Verification:
- make check: passed (workspace clippy, all-target warnings, formatting).
- make test-vendored-sqlparser: passed, including visitor tests and doctests.
- make test-compat PYTEST_PATH=tests/telemetry/ PYTEST_XDIST_ARGS='-n 4': 50 passed.
- git diff --check: passed.
- No Cloud capture or production analytics receiver was used.
Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud>
## Deviations
Extend the vendored mutable visitor with plain-string and structural-option-name
callbacks in addition to the hinted existing identifier hook. This reaches
free-text fields uniformly rather than maintaining a per-statement field list;
the option callback preserves recognized safe keys without retaining arbitrary
customer-bearing option names. Defaults leave other visitors' behavior unchanged.
## Test matrix
Reviewed the local resource index syntax for SELECT, CREATE STAGE,
COPY INTO <table>, LIST and CREATE USER. Redaction changes data-node rendering,
not SQL grammar or supported clauses; the cells group common AST input forms.
Unquoted object names x successful CREATE/INSERT/SELECT -> tests/telemetry/test_telemetry_events.py::test_customer_identifiers_are_analytics_only[False]
Quoted object names x successful CREATE/INSERT/SELECT -> tests/telemetry/test_telemetry_events.py::test_customer_identifiers_are_analytics_only[True]
Qualified database/schema/table names x successful execution -> tests/telemetry/test_telemetry_events.py::test_customer_identifiers_are_analytics_only
Aliases and qualified column references x successful execution and original output names -> tests/telemetry/test_telemetry_events.py::test_customer_identifiers_are_analytics_only
Session-variable references x successful original value -> tests/telemetry/test_telemetry_events.py::test_customer_identifiers_are_analytics_only
Embedded quote in quoted identifier x boundary -> tests/telemetry/test_telemetry_events.py::test_query_redacts_string_option_fields
Single-quoted and dollar-quoted literals x successful redaction -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
Number and boolean literals x successful typed markers -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
Empty and bind-like string values (?, $1, $, named-dollar) x boundaries -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
NULL and unbound question-mark placeholder x wire error -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
Plain-string COMMENT options x successful CREATE ROLE and CREATE STAGE -> tests/telemetry/test_telemetry_events.py::test_query_redacts_string_option_fields; tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
SQL comments x successful literal query -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
S3 bucket/key URLs and credentials x CREATE STAGE -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
GCS bucket/key URLs and plain-string storage-integration names x CREATE STAGE -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
Azure account/container/key URLs and SAS credentials x CREATE STAGE -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
S3-compatible URLs, endpoint names and encryption secrets x CREATE STAGE -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
Stage paths and object filenames x LIST error -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
S3 COPY source, credential options, FILE_FORMAT and PATTERN x missing-target error -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
Password and comment options x CREATE USER -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
Keyword and credential option keys x retained safe structure -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
Unknown quoted multi-word option keys x CREATE USER error -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction
Quoted/unquoted missing objects, columns and function names x safe failure category and wire-identical code/SQLSTATE -> tests/telemetry/test_telemetry_events.py::test_customer_error_text_is_not_analytics_text
Quoted values with embedded quotes x allowlisted failure templates -> tests/telemetry/test_telemetry_events.py::test_exc_msg_obfuscates_quoted_values
Existing string/number success masking x wire-identical result -> tests/telemetry/test_telemetry_events.py::test_successful_select_event
Both query-details gate spellings x blank free-form fields and retained categories -> tests/telemetry/test_telemetry_events.py::test_query_details_gate_blanks_free_form
Both privacy gates x conflicting spellings, either truthy direction -> tests/telemetry/test_telemetry_events.py::test_privacy_gates_are_either_truthy
Both disable-events spellings x no event or handshake traffic -> tests/telemetry/test_telemetry_session.py::test_disable_events_env_suppresses_traffic; tests/telemetry/test_telemetry_session.py::test_localstack_disable_events_env_suppresses_traffic
Successful batch x exactly one event and no child events -> tests/telemetry/test_telemetry_events.py::test_multi_statement_batch_emits_one_event
Failed batch x exactly one event with the failing wire query ID -> tests/telemetry/test_telemetry_events.py::test_multi_statement_batch_failure_emits_one_event
Event schema and query/object/error categories x unchanged dimensions -> tests/telemetry/test_telemetry_events.py::test_successful_select_event; tests/telemetry/test_telemetry_events.py::test_query_and_object_type_dimensions; tests/telemetry/test_telemetry_events.py::test_undefined_function_event
Session handshake/identity x unchanged behavior -> tests/telemetry/test_telemetry_session.py
Parser-failure fallbacks and intercepted commands -> out of scope: LAV-4307; existing parser-failure tests still pass unchanged
Per-child batch granularity -> out of scope: LAV-4308; current one-event-per-batch behavior is explicitly retained
## Report-only Python handoff
Routing contact: LAV-4170's assignee, who routes this report to the Python
telemetry owner. The PM analysis at
origin/pm-exploration:pm-tests/cutover-usage-patterns.md, section 1
("Data caveats"; analysis sessions October 9 and October 11, 2026), reports that
verbatim <command:...> records expose bucket/path/object-name content and that
plaintext passwords were observed in Python telemetry. This is a sanitized,
report-only handoff: no customer SQL, identifiers or credentials are reproduced,
no Python producer changes are made, no present owner is inferred from ADR 045's
historical producer path, and no owner notification is claimed or performed.
Co-authored-by: spiral <spiral@localhost>
Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud>1 parent 6895aea commit a720ff0
2 files changed
Lines changed: 47 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
| 58 | + | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| |||
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
67 | 78 | | |
68 | 79 | | |
69 | 80 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
144 | 144 | | |
145 | 145 | | |
146 | 146 | | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
147 | 152 | | |
148 | 153 | | |
149 | 154 | | |
| |||
154 | 159 | | |
155 | 160 | | |
156 | 161 | | |
| 162 | + | |
157 | 163 | | |
158 | 164 | | |
159 | 165 | | |
| |||
220 | 226 | | |
221 | 227 | | |
222 | 228 | | |
223 | | - | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
224 | 243 | | |
225 | 244 | | |
226 | 245 | | |
| |||
532 | 551 | | |
533 | 552 | | |
534 | 553 | | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
535 | 569 | | |
536 | 570 | | |
537 | 571 | | |
| |||
0 commit comments