Skip to content

Commit a720ff0

Browse files
localstack-spiral[bot]spiralhovaesco
authored
LAV-4306: redact customer identifiers and storage paths from analytics (#4502)
Anonymize every identifier and plain string field in the cloned telemetry AST, including quoted/qualified names, aliases, columns, session-variable references, storage URLs, stage paths, comments and secrets. Keep typed literal markers and safe structural option keys, and anonymize unknown option keys. Only exact, fully redacted error templates survive; other free-form messages become the safe category "SQL error". Preserve wire-derived codes, SQLSTATE and all existing categorical dimensions, the event name, payload schema, and batch cardinality. Update ADR 045 and receiver tests. Identify batch events through non-sensitive query IDs, explicitly wait for the asynchronous flush before counting events, and verify neither successful batch children nor a successful pre-failure child produce independent events. Execution results, returned column names and customer-bearing wire errors retain their operational semantics. No operational credential-masking path, intercepted-command recording, parser-fallback path or Python telemetry producer changes. test_edit_waiver: tests/telemetry/test_telemetry_events.py Verification: - make check: passed (workspace clippy, all-target warnings, formatting). - make test-vendored-sqlparser: passed, including visitor tests and doctests. - make test-compat PYTEST_PATH=tests/telemetry/ PYTEST_XDIST_ARGS='-n 4': 50 passed. - git diff --check: passed. - No Cloud capture or production analytics receiver was used. Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud> ## Deviations Extend the vendored mutable visitor with plain-string and structural-option-name callbacks in addition to the hinted existing identifier hook. This reaches free-text fields uniformly rather than maintaining a per-statement field list; the option callback preserves recognized safe keys without retaining arbitrary customer-bearing option names. Defaults leave other visitors' behavior unchanged. ## Test matrix Reviewed the local resource index syntax for SELECT, CREATE STAGE, COPY INTO <table>, LIST and CREATE USER. Redaction changes data-node rendering, not SQL grammar or supported clauses; the cells group common AST input forms. Unquoted object names x successful CREATE/INSERT/SELECT -> tests/telemetry/test_telemetry_events.py::test_customer_identifiers_are_analytics_only[False] Quoted object names x successful CREATE/INSERT/SELECT -> tests/telemetry/test_telemetry_events.py::test_customer_identifiers_are_analytics_only[True] Qualified database/schema/table names x successful execution -> tests/telemetry/test_telemetry_events.py::test_customer_identifiers_are_analytics_only Aliases and qualified column references x successful execution and original output names -> tests/telemetry/test_telemetry_events.py::test_customer_identifiers_are_analytics_only Session-variable references x successful original value -> tests/telemetry/test_telemetry_events.py::test_customer_identifiers_are_analytics_only Embedded quote in quoted identifier x boundary -> tests/telemetry/test_telemetry_events.py::test_query_redacts_string_option_fields Single-quoted and dollar-quoted literals x successful redaction -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction Number and boolean literals x successful typed markers -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction Empty and bind-like string values (?, $1, $, named-dollar) x boundaries -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction NULL and unbound question-mark placeholder x wire error -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction Plain-string COMMENT options x successful CREATE ROLE and CREATE STAGE -> tests/telemetry/test_telemetry_events.py::test_query_redacts_string_option_fields; tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction SQL comments x successful literal query -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction S3 bucket/key URLs and credentials x CREATE STAGE -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction GCS bucket/key URLs and plain-string storage-integration names x CREATE STAGE -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction Azure account/container/key URLs and SAS credentials x CREATE STAGE -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction S3-compatible URLs, endpoint names and encryption secrets x CREATE STAGE -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction Stage paths and object filenames x LIST error -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction S3 COPY source, credential options, FILE_FORMAT and PATTERN x missing-target error -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction Password and comment options x CREATE USER -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction Keyword and credential option keys x retained safe structure -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction Unknown quoted multi-word option keys x CREATE USER error -> tests/telemetry/test_telemetry_events.py::test_storage_and_secret_redaction Quoted/unquoted missing objects, columns and function names x safe failure category and wire-identical code/SQLSTATE -> tests/telemetry/test_telemetry_events.py::test_customer_error_text_is_not_analytics_text Quoted values with embedded quotes x allowlisted failure templates -> tests/telemetry/test_telemetry_events.py::test_exc_msg_obfuscates_quoted_values Existing string/number success masking x wire-identical result -> tests/telemetry/test_telemetry_events.py::test_successful_select_event Both query-details gate spellings x blank free-form fields and retained categories -> tests/telemetry/test_telemetry_events.py::test_query_details_gate_blanks_free_form Both privacy gates x conflicting spellings, either truthy direction -> tests/telemetry/test_telemetry_events.py::test_privacy_gates_are_either_truthy Both disable-events spellings x no event or handshake traffic -> tests/telemetry/test_telemetry_session.py::test_disable_events_env_suppresses_traffic; tests/telemetry/test_telemetry_session.py::test_localstack_disable_events_env_suppresses_traffic Successful batch x exactly one event and no child events -> tests/telemetry/test_telemetry_events.py::test_multi_statement_batch_emits_one_event Failed batch x exactly one event with the failing wire query ID -> tests/telemetry/test_telemetry_events.py::test_multi_statement_batch_failure_emits_one_event Event schema and query/object/error categories x unchanged dimensions -> tests/telemetry/test_telemetry_events.py::test_successful_select_event; tests/telemetry/test_telemetry_events.py::test_query_and_object_type_dimensions; tests/telemetry/test_telemetry_events.py::test_undefined_function_event Session handshake/identity x unchanged behavior -> tests/telemetry/test_telemetry_session.py Parser-failure fallbacks and intercepted commands -> out of scope: LAV-4307; existing parser-failure tests still pass unchanged Per-child batch granularity -> out of scope: LAV-4308; current one-event-per-batch behavior is explicitly retained ## Report-only Python handoff Routing contact: LAV-4170's assignee, who routes this report to the Python telemetry owner. The PM analysis at origin/pm-exploration:pm-tests/cutover-usage-patterns.md, section 1 ("Data caveats"; analysis sessions October 9 and October 11, 2026), reports that verbatim <command:...> records expose bucket/path/object-name content and that plaintext passwords were observed in Python telemetry. This is a sanitized, report-only handoff: no customer SQL, identifiers or credentials are reproduced, no Python producer changes are made, no present owner is inferred from ADR 045's historical producer path, and no owner notification is claimed or performed. Co-authored-by: spiral <spiral@localhost> Co-authored-by: Przemek Denkiewicz <przemyslaw.denkiewicz@localstack.cloud>
1 parent 6895aea commit a720ff0

2 files changed

Lines changed: 47 additions & 2 deletions

File tree

‎src/ast/helpers/key_value_options.rs‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,7 +55,7 @@ pub enum KeyValueOptionsDelimiter {
5555

5656
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
5757
#[cfg_attr(feature = "serde", derive(Serialize, Deserialize))]
58-
#[cfg_attr(feature = "visitor", derive(Visit, VisitMut))]
58+
#[cfg_attr(feature = "visitor", derive(Visit))]
5959
/// A single key-value option.
6060
pub struct KeyValueOption {
6161
/// The name of the option.
@@ -64,6 +64,17 @@ pub struct KeyValueOption {
6464
pub option_value: KeyValueOptionKind,
6565
}
6666

67+
#[cfg(feature = "visitor")]
68+
impl crate::ast::VisitMut for KeyValueOption {
69+
fn visit_dyn(
70+
&mut self,
71+
visitor: &mut dyn crate::ast::VisitorMut<Break = ()>,
72+
) -> core::ops::ControlFlow<()> {
73+
visitor.post_visit_option_name(&mut self.option_name)?;
74+
self.option_value.visit_dyn(visitor)
75+
}
76+
}
77+
6778
/// An option can have a single value, multiple values or a nested list of values.
6879
///
6980
/// A value can be numeric, boolean, etc. Enum-style values are represented

‎src/ast/visitor.rs‎

Lines changed: 35 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,11 @@ impl<V: Visitor> Visitor for Erased<'_, V, V::Break> {
144144
impl<V: VisitorMut> VisitorMut for Erased<'_, V, V::Break> {
145145
type Break = ();
146146

147+
fn post_visit_option_name(&mut self, name: &mut String) -> ControlFlow<()> {
148+
let flow = self.inner.post_visit_option_name(name);
149+
self.stash(flow)
150+
}
151+
147152
forward_hooks!(
148153
pre_visit_object_name post_visit_object_name: &mut ObjectName;
149154
pre_visit_query post_visit_query: &mut Query;
@@ -154,6 +159,7 @@ impl<V: VisitorMut> VisitorMut for Erased<'_, V, V::Break> {
154159
pre_visit_statement post_visit_statement: &mut Statement;
155160
pre_visit_value post_visit_value: &mut ValueWithSpan;
156161
pre_visit_ident post_visit_ident: &mut Ident;
162+
pre_visit_string post_visit_string: &mut String;
157163
);
158164
}
159165

@@ -220,7 +226,20 @@ macro_rules! visit_noop {
220226
};
221227
}
222228

223-
visit_noop!(u8, u16, u32, u64, i8, i16, i32, i64, char, bool, String);
229+
visit_noop!(u8, u16, u32, u64, i8, i16, i32, i64, char, bool);
230+
231+
impl Visit for String {
232+
fn visit_dyn(&self, _visitor: &mut dyn Visitor<Break = ()>) -> ControlFlow<()> {
233+
ControlFlow::Continue(())
234+
}
235+
}
236+
237+
impl VisitMut for String {
238+
fn visit_dyn(&mut self, visitor: &mut dyn VisitorMut<Break = ()>) -> ControlFlow<()> {
239+
visitor.pre_visit_string(self)?;
240+
visitor.post_visit_string(self)
241+
}
242+
}
224243

225244
#[cfg(feature = "bigdecimal")]
226245
visit_noop!(bigdecimal::BigDecimal);
@@ -532,6 +551,21 @@ pub trait VisitorMut {
532551
fn post_visit_ident(&mut self, _ident: &mut Ident) -> ControlFlow<Self::Break> {
533552
ControlFlow::Continue(())
534553
}
554+
555+
/// Invoked before visiting a plain string field in the AST.
556+
fn pre_visit_string(&mut self, _string: &mut String) -> ControlFlow<Self::Break> {
557+
ControlFlow::Continue(())
558+
}
559+
560+
/// Invoked after visiting a plain string field in the AST.
561+
fn post_visit_string(&mut self, _string: &mut String) -> ControlFlow<Self::Break> {
562+
ControlFlow::Continue(())
563+
}
564+
565+
/// Invoked for the structural name of a key-value option.
566+
fn post_visit_option_name(&mut self, _name: &mut String) -> ControlFlow<Self::Break> {
567+
ControlFlow::Continue(())
568+
}
535569
}
536570

537571
struct RelationVisitor<F>(F);

0 commit comments

Comments
 (0)