feat: support pylance 4.0.0

[XuQianJin-Stars]

Is your feature request related to a problem?

The current Daft project pins pylance to <0.40.0 (locked at 0.39.0), which prevents users from
leveraging features and improvements in newer lance versions (1.0.0 through 4.0.0). Since pylance
0.39.0 was released on 2025-11-04 and pylance 4.0.0 was released on 2026-03-30, there have been
4 major version upgrades with potential performance improvements, new index types (e.g. RTREE),
and bug fixes that Daft users cannot benefit from.

Describe the solution you'd like

Upgrade pylance dependency to support version 4.0.0 with the following changes:

1. Update dependency constraints in pyproject.toml:

   • Change optional dependency from pylance<0.40.0 to pylance>=0.39.0,<5.0.0
   • Update dev dependency from pylance==0.39.0 to pylance==4.0.0

2. Adapt to pylance 4.0.0 API changes in daft/io/lance/lance_scalar_index.py:

   • Rename fragment_uuid parameter to index_uuid in create_scalar_index() calls
   • Fix replace logic in create_scalar_index_internal() to include old indices
     in removed_indices when committing, preventing panic in pylance 4.0.0

3. Handle error message changes in daft/io/lance/lance_data_sink.py:

   • Add handling for "not a directory" error in _load_existing_dataset(),
     as pylance 4.0.0 changed the error message format when target path is a file

4. Update test expectations in tests/io/lancedb/test_lancedb_scalar_index.py:

   • Update error message match for invalid index type test to include the new RTREE
     index type added in pylance 4.0.0

All 189 lance/lancedb tests pass (189 passed, 2 skipped, 0 failed) after these changes.

Describe alternatives you've considered

1. Pin to a specific intermediate version (e.g. pylance 1.0.0 or 2.0.0): This would reduce
   the scope of breaking changes but would still miss out on the latest features and fixes in 4.0.0.

2. Support multiple pylance version ranges with conditional code paths: This would maintain
   backward compatibility with 0.39.0 while also supporting 4.0.0, but adds complexity and
   maintenance burden. The current approach of setting a broad range (>=0.39.0,<5.0.0) is simpler.

3. Wait for upstream stabilization: Since lance has gone through 4 major versions in ~5 months,
   we could wait for the API to stabilize. However, the current changes are minimal and well-tested.

Additional Context

No response

Would you like to implement a fix?

Yes