Skip to content

[BUG] tokenizer.Reset() does not clear dialect state → pooled tokenizers leak dialect/keywords across callers #524

Description

@mem

Bug Description

Tokenizer.Reset() (called by PutTokenizer before returning an instance to the sync.Pool) resets input, position, line tracking, and the logger, but not the dialect field or the keywords table. Because a Tokenizer obtained from the pool can have had its dialect changed via SetDialect, returning it to the pool leaves a non-default dialect in a shared, reusable instance. The next GetTokenizer() caller can receive a tokenizer configured for the wrong dialect (e.g. MySQL when it expected the PostgreSQL default), causing silently incorrect tokenization/parsing.

To Reproduce

  1. Get a tokenizer from the pool
  2. Set Dialect
  3. Put it back
  4. Get a tokenizer from the pool
  5. Validate that the dialect is set to the default value

Steps 4. and 5. might need to be repeated on a loop.

Code Example

func TestPoolDialectLeak(t *testing.T) {
    a := GetTokenizer()
    a.SetDialect(keywords.DialectMySQL)
    PutTokenizer(a)

    b := GetTokenizer()
    defer PutTokenizer(b)
    if b.Dialect() != keywords.DialectPostgreSQL {
        t.Fatalf("LEAK: got dialect %q, want default %q", b.Dialect(), keywords.DialectPostgreSQL)
    }
}

SQL Query

N/A

Expected Behavior

The GetTokenizer behavior is consistent, or there's a explit note in the docs about having to set the dialect each time a tokenizer is retrieved from the pool.

Actual Behavior

You might get a tokenizer with a different dialect than the default.

Environment

  • OS: Linux
  • Go Version: 1.26
  • GoSQLX Version: 1.14.0

Additional Context

N/A

Possible Solution

  1. Set the tokenizer back to the default when Reset() is called.
  2. Require setting the tokenizer explicitly after GetTokenizer()

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions