feat(postgres): support PostgreSQL in ODC backend (issue #850)#19
Open
LordofAvernus wants to merge 24 commits into
Open
feat(postgres): support PostgreSQL in ODC backend (issue #850)#19LordofAvernus wants to merge 24 commits into
LordofAvernus wants to merge 24 commits into
Conversation
Add PostgresJdbcUrlParser that implements JdbcUrlParser interface to parse PostgreSQL JDBC URLs. Uses string splitting (not regex) for parsing to reduce complexity. Features: - Parse standard URL: jdbc:postgresql://host:port/database - Extract schema from currentSchema parameter - Support default port 5432 when not specified - Parse multiple query parameters Includes comprehensive unit tests covering: - Standard URL parsing - URL without port (default 5432) - URL without parameters - URL with multiple parameters - currentSchema extraction - Special character handling - Null/invalid input handling Ref: AC-001.4, AC-001.5
…arameters for PostgresConnectionExtension - Add getConnectionInfo() method using PostgresJdbcUrlParser - Add appendDefaultJdbcUrlParameters() method with ApplicationName=ODC default - Update generateJdbcUrl() to support JDBC parameters properly - Add comprehensive unit tests for all new methods This completes AC-001.1 and AC-001.4 requirements for PG connection extension.
…in PostgresSessionExtension Implement PostgreSQL-specific session management: - switchSchema(): use SET search_path TO for schema switching - getCurrentSchema(): query current_schema() function - getCurrentDatabase(): query current_database() function - getConnectionId(): use pg_backend_pid() for process ID - getKillQuerySql(): generate SELECT pg_cancel_backend(pid) - getKillSessionSql(): generate SELECT pg_terminate_backend(pid) - getVariable(): use current_setting() for session variables Unit tests verify SQL generation correctness for all methods.
…nto service layer Integrate PostgreSQL SQL processing components into ODC service layer: - Add PostgresSqlBuilder for PostgreSQL identifier/value quoting - Modify SqlUtils.split() to use PostgreSqlSplitter for PostgreSQL dialect - Modify SqlUtils.iterator() to support PostgreSQL streaming SQL parsing - Modify ConnectConsoleService.queryTableOrViewData() to use PostgresSqlBuilder - Modify ConnectConsoleService.streamExecute() to enable PostgreSQL SQL splitting PostgreSQL requires special SQL splitting due to: - Dollar-quoting ($$...$$, $tag$...$tag$) in function bodies - E-string (E'...') with backslash escapes - Nested block comments support Covers: AC-003.1, AC-003.5, AC-003.7
Implement PostgreSQLFeatures class to define PostgreSQL-specific feature support in ODC: - Create PostgreSQLFeatures extending DefaultFeatures - Override 8 feature methods for PostgreSQL: - supportsShowTrace(): false (PG has no show trace) - supportsViewObject(): true - supportsOBTenant(): false (PG has no tenant concept) - supportsShowTenant(): false - supportsProcedure(): true (PG 11+ supports procedures) - supportsSchemaPrefixInSql(): true - supportsExplain(): true - supportsAutoIncrement(): false (PG uses SERIAL/IDENTITY) - Register POSTGRESQL constant in AllFeatures.getByConnectType() - Add unit tests with 11 test cases covering all methods Covers requirements: AC-005.8, NFR-001.2
…TraceExtension - Add PostgresDiagnoseExtensionPoint implementing SqlDiagnoseExtensionPoint - getExplain() executes EXPLAIN <sql> and returns execution plan text - Other methods (getPhysicalPlanBySqlId, etc.) throw UnsupportedOperationException - Add PostgresTraceExtension implementing TraceExtensionPoint - getExecuteDetail() returns empty SqlExecTime as placeholder - PostgreSQL has no built-in trace mechanism like OceanBase - Add unit tests for both extensions (34 test cases total) - Add mockito-core test dependency to connect-plugin-postgres Reference: SQLServer implementation pattern for consistency Requirement: FR-011, AC-011.1, AC-011.3, NFR-003.2
…cessor Implement PostgreSQL-specific table metadata query methods: Table listing: - listTables(schema, like): query pg_class + pg_namespace Column operations: - listBasicTableColumns(schema): batch query pg_attribute + pg_type - listBasicTableColumns(schema, table): single table column info - listTableColumns(schema, table): full column info with default/comment - listTableColumns(schema, tableNames): batch multi-table query Table options and DDL: - getTableOptions(schema, table): query table comment - listTableOptions(schema): batch table options - getTableDDL(schema, table): programmatic DDL assembly (columns -> PK -> comments -> indexes -> constraints) Partition: - getPartition(schema, table): detect partitioned tables and query partition info (RANGE/LIST/HASH) Indexes: - listTableIndexes(schema): batch table index query - listTableIndexes(schema, table): single table index (btree/hash/gin/gist) Constraints: - listTableConstraints(schema): batch constraint query - listTableConstraints(schema, table): single table constraint (PK/UNIQUE/FK/CHECK) Unit tests: PostgresSchemaAccessorTest with 13 test cases
…emaAccessorTest - Add independent tableName field to DBTrigger model to properly support SQL Server trigger template which needs both tableName and schemaName - Update getTableName() to return tableName field if set, fallback to schemaName for backward compatibility - Fix PostgresSchemaAccessorTest mock patterns to handle multiple SQL queries for different constraint types (PK/Unique, FK, CHECK) properly This change ensures SqlServerTriggerTemplateTest passes correctly while maintaining backward compatibility with existing code that uses getTableName() as an alias for schemaName.
…ethods in PostgresSchemaAccessor - Implement listViews(), listAllViews(), listAllUserViews(), listAllSystemViews(), showSystemViews() - Implement listFunctions(), listProcedures(), listTriggers(), listTypes(), listSequences() - Implement showVariables(), showSessionVariables(), showGlobalVariables() - Implement getView(), getFunction(), getProcedure(), getTrigger(), getType(), getSequence() - Implement listBasicViewColumns() for view column queries - Add comprehensive unit tests for all new methods (23 tests total)
…and session queries - Implement getTableStats() using pg_stat_user_tables and pg_total_relation_size() - Implement listAllSessions() querying pg_stat_activity - Implement currentSession() using pg_backend_pid() - Add custom PostgresDBSessionRowMapper with proper state mapping - Update DBStatsAccessorFactory.buildForPostgres() to return new accessor - Add 13 unit tests with mock-based testing
…PostgresConstraintEditor, PostgresPartitionEditor, PostgresObjectOperator and PostgresTableEditor with factory registration
…ate and PostgresProcedureTemplate - Add PostgresViewTemplate for generating CREATE VIEW statements with schema prefix support - Add PostgresFunctionTemplate for generating CREATE FUNCTION statements with dollar-quoting - Add PostgresProcedureTemplate for generating CREATE PROCEDURE statements (PG 11+) - Update DBViewTemplateFactory, DBFunctionTemplateFactory, DBProcedureTemplateFactory to register PostgreSQL templates - Add comprehensive unit tests for all three templates Template features: - View: Supports schema.viewName format, CREATE OR REPLACE VIEW syntax - Function: Supports IN/OUT/INOUT parameters, DEFAULT values, RETURNS clause, PL/pgSQL dollar-quoting - Procedure: Similar to function but without RETURNS clause, supports all parameter modes All 41 unit tests pass.
…r/operator access Add 3 new factory methods to DBAccessorUtil for PostgreSQL: - getStatsAccessor(Connection) - provides DBStatsAccessor instance - getTableEditor(Connection) - provides DBTableEditor instance - getObjectOperator(Connection) - provides DBObjectOperator instance These methods enable schema-plugin-postgres to access db-browser's PostgreSQL-specific implementations for statistics, table editing, and database object operations. Also add unit tests (10 test cases) covering all factory methods.
…on, PostgresFunctionExtension and PostgresProcedureExtension Complete schema-plugin-postgres module with 4 Extension implementations: 1. PostgresTableExtension enhancements: - Override getDetail() - use schemaAccessor directly instead of OBMySQLGetDBTableByParser - Override generateCreateDDL() - delegate to DBAccessorUtil.getTableEditor() - Override generateUpdateDDL() - delegate to DBAccessorUtil.getTableEditor() - Override getStatsAccessor() - return DBAccessorUtil.getStatsAccessor() - Override getTableStats() - format table size with BinarySizeUnit - Override getTableEditor() - return DBAccessorUtil.getTableEditor() 2. PostgresViewExtension (new): - Extend OBMySQLViewExtension - Override getSchemaAccessor() - return DBAccessorUtil.getSchemaAccessor() - Override getOperator() - return PostgresObjectOperator instance - Override generateCreateTemplate() - use PostgresViewTemplate - Override getTemplate() - return DBBrowser view template for PostgreSQL 3. PostgresFunctionExtension (new): - Extend OBMySQLFunctionExtension - Override getSchemaAccessor() - return DBAccessorUtil.getSchemaAccessor() - Override getOperator() - return PostgresObjectOperator instance - Override generateCreateTemplate() - use PostgresFunctionTemplate - Override getTemplate() - return DBBrowser function template for PostgreSQL 4. PostgresProcedureExtension (new): - Extend OBMySQLProcedureExtension - Override getSchemaAccessor() - return DBAccessorUtil.getSchemaAccessor() - Override getOperator() - return PostgresObjectOperator instance - Override generateCreateTemplate() - use PostgresProcedureTemplate - Override getTemplate() - return DBBrowser procedure template for PostgreSQL Unit tests: 33 test cases all passed
… data types - Add PGBooleanMapper: maps boolean/bool type to "true"/"false" string - Add PGNumericMapper: ensures NUMERIC/DECIMAL precision with BigDecimal.toPlainString() - Add PGByteaMapper: handles bytea binary data with size truncation display - Add PGArrayMapper: handles PostgreSQL array types (_int4, _text, etc.) - Add PGTimestampTZMapper: handles timestamptz type with timezone info - Modify DefaultJdbcRowMapper: add PostgreSQL dialect branch with 5 mappers - Add PGTestCellData test helper class - Add unit tests for all 5 mappers (29 test cases total)
Disable support_trigger, support_trigger_ddl, support_sequence, and support_type configs for PostgreSQL because the corresponding ExtensionPoints (TriggerExtensionPoint, SequenceExtensionPoint, TypeExtensionPoint) are not implemented in schema-plugin-postgres module. This fixes the error "Feature extension point is not supported for POSTGRESQL" when users try to use trigger/sequence/type features in ODC. Keep these features disabled until plugin extensions are implemented, same as SQL Server which also doesn't implement these extension points.
…croseconds is null When PostgreSQL/SQLServer/Doris execute SQL, the TraceExtension.getExecuteDetail() returns empty SqlExecTime object (executeMicroseconds=null) because these databases don't have built-in trace mechanism like OceanBase/MySQL's show profile. Original implementation skipped creating DB_SERVER_EXECUTE_SQL stage when executeMicroseconds is null, causing frontend to display '-' for DB duration. This fix adds fallback logic to use Execute stage time as approximate DB execution time, ensuring frontend can always display DB duration correctly. Fixes: E-002 PostgreSQL DB duration shows '-' Also benefits: SQLServer, Doris, MySQL(profiling=OFF), Oracle(exception)
oceanbase#850, compat-RISK-1) Refs: dms-ee#850 Risk: compat-RISK-1 data-upgrade: B-V_4_3_4_14__fix_postgresql_version_diff_config_idempotent 存量 ODC 部署因 V_4_3_4_13 末尾 `ON DUPLICATE KEY UPDATE config_key=config_key` （自我赋值即 NOOP），无法把已存在的 PostgreSQL 三项配置 support_trigger / support_sequence / support_type 从 'true' 覆盖为 'false'，导致升级后 PG 后端 PostgreSQLFeatures 仍按 true 加载，与前端收敛不一致（compat_risks.md §1 compat-RISK-1 / design.md §8.4）。 新增独立 Flyway DML 脚本 V_4_3_4_14，对存量行执行 UPDATE： - WHERE `db_mode` = 'POSTGRESQL'：仅命中 PG，不影响 MySQL/Oracle 等其它 db_mode - WHERE `config_key` IN ('support_trigger','support_sequence','support_type')： 仅修复三项 bug 配置，不影响 PG 其它 config_key（support_view / support_function 等） - WHERE `config_value` = 'true'：仅修复老错误状态，二次执行 / 全新部署均零变更 （幂等），不抹掉运维已手工修正的状态 V_4_3_4_13 文本未改动（Flyway checksum 约束）；本次新增 SQL 与 V_4_3_4_13 同 目录，Flyway 同 location 扫描，按版本号顺序执行。
…ithout catalog_name (issue oceanbase#850) Bug: DMS 创建 PG 数据源时仅传 default_schema（典型值 public）而不传 catalogName， ODC 后端 PostgresConnectionExtension#generateJdbcUrl 抛 `IllegalArgumentException: catalog name can not be null`，导致 DatabaseService.syncDataSourceSchemas 100% 失败、前端资源树 PG 节点没有 caret-down， 无法展开到 schema 层（dms-ee#850 / Task-001 烟雾测试 EXC-2）。 Root cause: PG 的 JDBC URL 形如 jdbc:postgresql://host:port/<catalog>?currentSchema=<schema>， catalog 必须非空，但 ODC `ConnectionConfig` 把 catalogName / defaultSchema 分离持久化， 上游（DMS）侧只填了 default_schema，未提供 database name；ODC 当前没有兜底。 Fix: 在 OBConsoleDataSourceFactory 新增 `resolveEffectiveCatalogName(dialectType, catalogName, defaultSchema)` 静态方法做 PG 专属兜底——仅当 dialectType=POSTGRESQL 且 catalogName 为空时生效： 1. defaultSchema 非空且 !equalsIgnoreCase("public") → 用 defaultSchema 兜底（兼容 用户在 default_schema 字段中实际填了 database 名的场景）； 2. 否则使用 PG 标准内置数据库 `postgres`（POSTGRESQL_DEFAULT_DATABASE， 在所有 PG 标准安装中默认存在）。 不直接用 defaultSchema=public 作 catalog 兜底，因为 PG 中 public 是 schema 名而不是 database 名，强行用之会抛 FATAL: database "public" does not exist。 对其他数据源类型（MySQL/Oracle/SQLServer/OceanBase 等）保持原行为不变：catalogName 原样透传，不影响其既有 JDBC URL 生成逻辑（这些 dialect 的 ConnectionExtension 未对 catalogName 做非空校验）。 Files: - odc-core/OdcConstants: 新增 POSTGRESQL_DEFAULT_DATABASE = "postgres" 常量 - odc-service/OBConsoleDataSourceFactory: getJdbcUrlProperties() 通过新增的 resolveEffectiveCatalogName 做兜底；保留原 connectionConfig 持久化字段不变 - odc-service/ConnectionTesting#getJdbcUrlProperties: 同步走 resolveEffectiveCatalogName 兜底，让测试连接（POST /api/v2/datasource/.../connection_test）也修复 - odc-service/DataSourceInfoMapper#getJdbcUrl: DLM 数据迁移任务路径同步兜底 - odc-service test: 新增 OBConsoleDataSourceFactoryTest 12 条用例，覆盖 PG/MySQL/Oracle/OBMysql/SqlServer 各种 catalog/schema 组合，含 issue oceanbase#850 现场复现 Validation: - mvn -pl server/odc-core install + mvn -pl server/odc-service compile 全通过 - 单测 OBConsoleDataSourceFactoryTest 12/12 OK（独立 JUnit runner） - ODC 重启后日志 `Create datasource success, jdbcUrl: jdbc:postgresql://10.186.16.126:5433/postgres?...¤tSchema=public&...` 证明 jdbc URL 生成成功；之前 `catalog name can not be null` 错误**不再出现** Refs: dms-ee#850, EXC-2 (Task-001 烟雾测试)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ODC 后端 4.3.4 分支接入 PostgreSQL 支持，覆盖 issue oceanbase#850 的 ODC 侧改动：
关联 issue: actiontech/dms-ee#850
关联文档: docs/spec/design.md §3.4 / §8.4，docs/dev/compat_risks.md §compat-RISK-1，docs/dev/fix-task-002-odc-pg-catalog-null.md
Test plan