Repository navigation
Python: add Milvus connector for the VectorStore abstraction - #14577
Nantha kumar (Nanduu24) wants to merge 1 commit into
Conversation
Adds MilvusStore, MilvusCollection and MilvusSettings implementing the modern VectorStore/VectorStoreCollection/VectorSearch interfaces, so Milvus is available as an out-of-the-box vector store alongside Postgres, Qdrant, Redis, Pinecone, Weaviate, etc. Previously Milvus was only available through the deprecated MemoryStore API. - Schema, typed scalar fields, FLOAT_VECTOR fields and per-vector indexes are derived from the data model definition; the dynamic field is enabled so properties outside the definition are preserved. - Distance functions and index kinds are mapped to Milvus metric/index types. - Vector search with lambda/string filters translated to Milvus boolean expressions, plus upsert, get, delete, and collection lifecycle. - Registered in connectors.memory for lazy import and the .pyi stub. - Unit tests covering ingestion, search, filter translation, serialization, component lifecycle and schema creation. Resolves microsoft#13391
|
Well done! Thx Is this working with the new Microsoft Agent Framework? |
|
Thanks Allyson (@allyson-tcesp)! This PR targets Semantic Kernel's |
|
Hi Eduard van Valkenburg (@eavanvalkenburg) — since you own the Python memory connectors, could you review this when you get a moment? It adds a Milvus |
Motivation and Context
Milvus is one of the most widely used open-source vector databases, but on the Python side it was only available through the deprecated
MemoryStoreAPI (connectors/memory_stores/milvus). Every other major backend — Postgres, Qdrant, Redis, Pinecone, Weaviate, Chroma, Faiss, SQL Server, Azure AI Search — has been ported to the modernVectorStore/VectorStoreCollectionabstraction, but Milvus had not.This has been an open, unaddressed request: #13391 ("Why is the Milvus connector not impl
VectorStore?"), where another user reported trying to build their own and finding it non-trivial. This PR closes that gap.Resolves #13391
Description
Adds a new
semantic_kernel/connectors/milvus.pywithMilvusStore,MilvusCollectionandMilvusSettings, following the same structure as the existingqdrant.py/postgres.pyconnectors:INT64/VARCHAR), data fields become typed scalar fields, and vector fields becomeFLOAT_VECTORfields, each with an index. The Milvus dynamic field is enabled so properties outside the definition are preserved on round-trip.DistanceFunction→ Milvus metric types (COSINE/L2/IP),IndexKind→ Milvus index types (HNSW/FLAT/IVF_FLAT/DISKANN/AUTOINDEX).ensure_collection_exists/collection_exists/ensure_collection_deleted,upsert,get,delete, and vectorsearch. Lambda/string filters are translated into Milvus boolean expressions via_lambda_parser(==,!=,<,<=,>,>=,in,not in,and/or/not, chained comparisons).MilvusClientwrapped withasyncio.to_threadso it works consistently across the supportedpymilvusrange (theMilvusClientAPI requirespymilvus >= 2.4, which the existing sharedmilvusextra already resolves to — happy to bump the extra floor from2.3to2.4if you'd prefer it made explicit).connectors/memory.py(lazy import) andconnectors/memory.pyi(type stub), mirroring the other connectors.Testing
tests/unit/connectors/memory/test_milvus.py(28 tests): ingestion, semantic search, filter translation, serialization/deserialization, component lifecycle, and schema creation (hermetic — fake client, no live DB).ruff format --check,ruff check, andmypyare clean on the changed files;uv lock --lockedis unaffected.I'm also happy to follow up with a concept sample under
samples/concepts/memory/and a docs entry if that would help.Contribution Checklist