Skip to content

Exercise PostgreSQL ClickPipe creation through the live CLI - #541

Open
sdairs wants to merge 3 commits into
stack/446-clickpipe-tls-guidancefrom
stack/447-live-cli-postgres-clickpipe
Open

Exercise PostgreSQL ClickPipe creation through the live CLI#541
sdairs wants to merge 3 commits into
stack/446-clickpipe-tls-guidancefrom
stack/447-live-cli-postgres-clickpipe

Conversation

@sdairs

@sdairs sdairs commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • rename the required managed-Postgres CDC integration target and create its ClickPipe through the built clickhousectl binary
  • verify the private-CA API failure and CLI guidance without --ca-certificate, then write the fetched CA bundle and verify successful creation, Running state, snapshot rows, and ongoing CDC
  • build the CLI in the ClickPipes workflow and explicitly classify both the historical and renamed test paths
  • document the renamed target, CLI build, and CLICKHOUSE_CLOUD_TEST_CLICKHOUSECTL_BIN requirement
  • query CLI-created SharedMergeTree-backed tables without unsupported FINAL

Tests

  • cargo test -p clickhouse-cloud-api --test clickpipe_postgres_cli_cdc_test
  • cargo test -p clickhousectl --bin clickhousectl postgres_
  • cargo test -p clickhousectl --test cli_request_shape_test postgres_
  • python3 -m unittest discover -s scripts/tests -p "test_*.py"
  • cargo fmt --all --check
  • cargo clippy -p clickhouse-cloud-api -p clickhousectl --all-targets -- -D warnings
  • Cloud Integration, clickpipes scope on 300a284d10856697058afdf041bd3ef5840d81d7

The credentialed live run passed. The managed Postgres fixture uses a private CA, so success without an uploaded CA remains a residual gap until CI provides a publicly trusted PostgreSQL source.

Closes #447

@sdairs

sdairs commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator Author

Carry-over review from PR #503 (old PR for #447). Design-wise this PR is better than #503 (self-describing rename, extracted PostgresCliCreate arg builder with an offline unit test pinning the exact arg vector, defensive cleanup registration on unexpected success, error-before-hint ordering assertion, explicit --auth basic/--replication-mode cdc, corrected EC2-stage comment, no unexplained FINAL removal). Two required items that #503 already had:

  1. No live Cloud CI execution exists for this PR. The entire point of issue Exercise PostgreSQL ClickPipe creation through the CLI in live CI #447 is live CLI coverage. Exercise PostgreSQL ClickPipe creation through the CLI in live CI #503 has a green credentialed Cloud Integration run (run 32751122188) and carries the run-cloud-integration label; this PR has neither, and its description admits the live test was not executed locally. Add the run-cloud-integration label and land a green Cloud Integration run on this branch before merging.

  2. Documentation regression vs Exercise PostgreSQL ClickPipe creation through the CLI in live CI #503. Exercise PostgreSQL ClickPipe creation through the CLI in live CI #503 updated AGENTS.md and the clickhouse-cloud-api README; this PR touches neither, and after the test rename crates/clickhouse-cloud-api/README.md:~83 still documents the deleted clickpipe_postgres_cdc_test target and never mentions the required CLI build or the CLICKHOUSE_CLOUD_TEST_CLICKHOUSECTL_BIN env var; AGENTS.md's ClickPipes bullet does not mention the CLI dependency. Update both docs to the renamed target and the CLI-build requirement.

  3. Minor (optional): inside the async test body, prefer the file's existing return Err(...) / TestResult convention over panicking assert!/assert_eq! for more actionable failure messages; and consider whether always passing --url (with a hardcoded production default) slightly reduces fidelity to the default CLI path.

@sdairs
sdairs force-pushed the stack/447-live-cli-postgres-clickpipe branch from 032c3b3 to 70fa877 Compare August 26, 2026 18:58
@sdairs
sdairs force-pushed the stack/447-live-cli-postgres-clickpipe branch from 70fa877 to 3af64db Compare August 26, 2026 19:01
@sdairs
sdairs force-pushed the stack/447-live-cli-postgres-clickpipe branch 2 times, most recently from 54910a5 to f00c6fd Compare August 26, 2026 19:07
@sdairs
sdairs force-pushed the stack/447-live-cli-postgres-clickpipe branch from f00c6fd to 41b403e Compare August 26, 2026 19:10
@sdairs
sdairs force-pushed the stack/447-live-cli-postgres-clickpipe branch from 41b403e to 420426a Compare August 26, 2026 19:12
Comment thread .github/workflows/cloud-integration.yml
@sdairs
sdairs force-pushed the stack/447-live-cli-postgres-clickpipe branch 2 times, most recently from 77e258e to ce1179f Compare August 26, 2026 19:24
@sdairs
sdairs force-pushed the stack/447-live-cli-postgres-clickpipe branch from ce1179f to 01059f7 Compare August 26, 2026 19:28
@sdairs
sdairs force-pushed the stack/447-live-cli-postgres-clickpipe branch from 01059f7 to cabfa1b Compare August 26, 2026 19:31
@sdairs
sdairs force-pushed the stack/447-live-cli-postgres-clickpipe branch from cabfa1b to b532b38 Compare August 26, 2026 19:34
@sdairs sdairs added the run-cloud-integration Run live Cloud integration tests for this PR head label Aug 26, 2026
@sdairs
sdairs had a problem deploying to cloud-integration August 26, 2026 19:37 — with GitHub Actions Failure

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit e43ad29. Configure here.

.into());
}
Ok(path)
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Relative CLI path fails locally

Low Severity

clickhousectl_binary checks CLICKHOUSE_CLOUD_TEST_CLICKHOUSECTL_BIN with is_file() against the test process working directory. Cargo sets that to the clickhouse-cloud-api package root, not the workspace. The documented target/debug/clickhousectl path is workspace-relative, so the live test rejects a valid binary. CI is unaffected because the workflow uses an absolute path.

Additional Locations (1)
Fix in Cursor Fix in Web

Reviewed by Cursor Bugbot for commit e43ad29. Configure here.

@sdairs
sdairs deployed to cloud-integration August 26, 2026 20:11 — with GitHub Actions Active
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

run-cloud-integration Run live Cloud integration tests for this PR head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Exercise PostgreSQL ClickPipe creation through the CLI in live CI

1 participant