[doc](lance) support lance catalog - #4024
Conversation
Gabriel39
left a comment
Gabriel39
left a comment
There was a problem hiding this comment.
Please also update docs/lakehouse/file-formats/lance.md in this PR. That existing page still says that Catalog access and vector/predicate pushdown are unsupported, and it instructs users to pass data/*.lance fragment globs. The new page says the opposite and requires a dataset-root path. Leaving both pages unchanged would publish conflicting official guidance.
| ); | ||
| ``` | ||
|
|
||
| `file_path` is relative to `user_files_secure_path` on the target BE. For a Local TVF, the BE obtains the Schema and reads the latest version available when execution begins. The current Local Lance TVF uses one Scanner and does not support reading multiple Lance datasets through a Glob pattern. |
There was a problem hiding this comment.
[Blocking documentation/implementation mismatch] This statement is not true with the current code: the Lance local TVF skips getFileListFromBackend()/safe_glob() and passes file_path directly to lance_dataset_open(). It therefore does not currently resolve relative to user_files_secure_path. Please land the secure dataset-directory resolution fix first, or revise this statement, and cover absolute, .., and symlink-escape paths.
| - Only `s3()` and `local()` are supported. Other file TVFs, such as HDFS and HTTP, are not currently supported. | ||
| - `path_partition_keys` is not supported. | ||
| - One TVF path can represent only one Lance dataset. | ||
| - `DESC FUNCTION` can display a Schema containing unsupported types, but SQL cannot project unsupported columns. |
There was a problem hiding this comment.
[Documentation/implementation mismatch] This is not currently true for the Local TVF path. BE schema discovery returns an error as soon as arrow_type_to_doris_type() encounters any unsupported field, so DESC FUNCTION cannot expose that schema and a query cannot proceed by selecting only supported fields. Please fix the BE schema representation or narrow this claim to the Catalog/S3 paths that actually provide this behavior.
| ORDER BY _distance ASC, row_id; | ||
| ``` | ||
|
|
||
| The result contains all columns from the Lance source table plus a Doris-generated `_distance` column of type `FLOAT`. The source table must not already contain a column named `_distance`. A SQL relation does not guarantee output order, so explicitly specify `ORDER BY _distance ASC` when deterministic nearest-neighbor ordering is required. Adding a unique column as a tie-breaker is recommended for rows with the same distance. |
There was a problem hiding this comment.
The _distance column is generated by the Lance Scanner, not by Doris. Doris configures the nearest query through lance-c and deserializes the _distance Arrow column that Lance auto-projects. Please describe it as Lance-generated and Doris-exposed as FLOAT; it would also help to state clearly that this is a distance (lower is closer), not a generic similarity score.
2 participants
Versions
Languages
Docs Checklist