Repository navigation
Python: add Milvus vector store (agent-framework-milvus) #9346
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Nantha kumar (Nanduu24)
wants to merge
11
commits into
microsoft:main
Choose a base branch
from
Nanduu24:python-milvus-vector-store
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,865
−1
Open
Changes from all commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
d121cfe
Python: add Milvus vector store (agent-framework-milvus)
Nanduu24 73796ab
Python: address review feedback on the Milvus vector store
Nanduu24 78c5046
Python: validate existing Milvus collections and filter operand types
Nanduu24 b2e1bd9
Python: only swallow genuine already-exists errors in ensure_collecti…
Nanduu24 26e60d5
Merge remote-tracking branch 'upstream/main' into python-milvus-vecto…
Nanduu24 6061318
Python: use the native async Milvus client and simplify client handling
Nanduu24 117ab0e
Python: declare Python 3.15 support for the Milvus package
Nanduu24 c2976c4
Merge remote-tracking branch 'upstream/main' into python-milvus-vecto…
Nanduu24 1a94e5c
Merge remote-tracking branch 'upstream/main' into python-milvus-vecto…
Nanduu24 0a8f6d5
Python: strengthen Milvus schema/vector validation and fix Lite docs
Nanduu24 834298c
Merge remote-tracking branch 'upstream/main' into python-milvus-vecto…
Nanduu24 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| MIT License | ||
|
|
||
| Copyright (c) Microsoft Corporation. | ||
|
|
||
| Permission is hereby granted, free of charge, to any person obtaining a copy | ||
| of this software and associated documentation files (the "Software"), to deal | ||
| in the Software without restriction, including without limitation the rights | ||
| to use, copy, modify, merge, publish, distribute, sublicense, and/or sell | ||
| copies of the Software, and to permit persons to whom the Software is | ||
| furnished to do so, subject to the following conditions: | ||
|
|
||
| The above copyright notice and this permission notice shall be included in all | ||
| copies or substantial portions of the Software. | ||
|
|
||
| THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR | ||
| IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, | ||
| FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE | ||
| AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER | ||
| LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, | ||
| OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE | ||
| SOFTWARE. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,69 @@ | ||
| # Milvus for Microsoft Agent Framework | ||
|
|
||
| `agent-framework-milvus` provides a [Milvus](https://milvus.io/) vector store for the | ||
| Microsoft Agent Framework, implementing the framework's `BaseVectorStore` / | ||
| `BaseVectorCollection` interfaces. It works alongside the Postgres, Qdrant, Redis and | ||
| other vector-store connectors. | ||
|
|
||
| ## Installation | ||
|
|
||
| ```bash | ||
| pip install agent-framework-milvus | ||
| ``` | ||
|
|
||
| This connects to a Milvus server. To use a local `.db` file (Milvus Lite) instead, also | ||
| install the Milvus Lite extra, which is published for Linux and macOS only: | ||
|
|
||
| ```bash | ||
| pip install "pymilvus[milvus_lite]" | ||
| ``` | ||
|
|
||
| ## Usage | ||
|
|
||
| ```python | ||
| from agent_framework import VectorStoreCollectionDefinition, VectorStoreField | ||
| from agent_framework_milvus import MilvusStore | ||
|
|
||
| definition = VectorStoreCollectionDefinition([ | ||
| VectorStoreField("key", name="id", type_="int"), | ||
| VectorStoreField("data", name="text", type_="str"), | ||
| VectorStoreField("vector", name="embedding", dimensions=3, distance_function="cosine_similarity"), | ||
| ]) | ||
|
|
||
| # Connects to a Milvus server; set the MILVUS_URI env var or pass uri=... | ||
| # For Milvus Lite, pass a local ".db" path as the uri argument, e.g. | ||
| # MilvusStore(uri="milvus.db") (requires the pymilvus[milvus_lite] extra). The | ||
| # MILVUS_URI env var must be a server address, not a file path. | ||
| async with MilvusStore(uri="http://localhost:19530") as store: | ||
| collection = store.get_collection(dict, definition=definition, collection_name="docs") | ||
| await collection.ensure_collection_exists() | ||
| # upsert takes a sequence; pass generate_vectors=False when vectors are supplied. | ||
| await collection.upsert([{"id": 1, "text": "hello", "embedding": [0.1, 0.2, 0.3]}], generate_vectors=False) | ||
| results = await collection.search(vector=[0.1, 0.2, 0.3], top=5) | ||
| async for hit in results: # each hit is a SearchResponse mapping | ||
| print(hit["record"], hit["score"]) | ||
| ``` | ||
|
|
||
| ## Configuration | ||
|
|
||
| Settings are resolved from explicit arguments, then an optional `.env` file, then | ||
| environment variables prefixed with `MILVUS_`: | ||
|
|
||
| | Setting | Environment variable | Notes | | ||
| | ---------- | -------------------- | -------------------------------------------------- | | ||
| | `uri` | `MILVUS_URI` | Server URI (e.g. `http://localhost:19530`). For Milvus Lite, pass a local `.db` path as the `uri=` argument instead — the `MILVUS_URI` env var must be a server address. | | ||
| | `token` | `MILVUS_TOKEN` | Access token (API key or `user:password`). | | ||
| | `user` | `MILVUS_USER` | Username. | | ||
| | `password` | `MILVUS_PASSWORD` | Password. | | ||
| | `db_name` | `MILVUS_DB_NAME` | Database name. | | ||
|
|
||
| ## Notes | ||
|
|
||
| - The key field becomes the Milvus primary key (`INT64` for `int` keys, `VARCHAR` | ||
| otherwise). Data fields map to typed scalar fields and vector fields to | ||
| `FLOAT_VECTOR` fields, each with its own index. | ||
| - Distance functions map to Milvus metrics (`cosine_similarity`/`cosine_distance` → | ||
| `COSINE`, `dot_prod` → `IP`, `euclidean_distance` → `L2`) and index kinds to Milvus | ||
| index types (`hnsw`, `flat`, `ivf_flat`, `disk_ann`, and `default` → `AUTOINDEX`). | ||
| - Portable filters support `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `between`, `in`, | ||
| `not_in`, and `and`/`or`/`not` groups, translated to Milvus boolean expressions. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,14 @@ | ||
| # Copyright (c) Microsoft. All rights reserved. | ||
|
|
||
| """Milvus vector stores for Microsoft Agent Framework.""" | ||
|
|
||
| import importlib.metadata | ||
|
|
||
| from ._vector_store import MilvusCollection, MilvusSettings, MilvusStore | ||
|
|
||
| try: | ||
| __version__ = importlib.metadata.version(__name__) | ||
| except importlib.metadata.PackageNotFoundError: | ||
| __version__ = "0.0.0" | ||
|
|
||
| __all__ = ["MilvusCollection", "MilvusSettings", "MilvusStore", "__version__"] |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This job runs only
poe test, which excludes the newly added live integration tests. The workflow header requires parallel jobs to stay in sync withpython-integration-tests.yml, but that workflow has no Milvus service/Lite job and Milvus is absent from its report/gate dependencies. Please schedulepoe test-integrationagainst a disposable Milvus server or Milvus Lite in the integration workflow and wire it into the aggregate report/check.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looking into this, I hit two concrete constraints and want your steer before wiring it:
MILVUS_URI. pymilvus reads a globalMILVUS_URIat import and validates it ashttp(s)://..., so pointing it at a local.dbcrashesimport pymilvus. (I verifiedAsyncMilvusClientitself works fine against Lite when the path is passed as theuri=argument, not the env var — I've fixed the docs accordingly.)milvus run standalone), and GitHubservices:can't override the image command, so it can't be a service container like qdrant/redis/postgres — it needs adocker runstep.Proposed: add
python-tests-milvustopython-integration-tests.ymlthatdocker runs a disposable Milvus standalone (embedded etcd) withMILVUS_URI=http://localhost:19530, waits on/healthz, runspoe test-integration, and wire it into the report +python-integration-tests-checkgate. Happy to push that — just confirming you'd prefer the standalone-server route over Milvus Lite (which the reserved env var rules out for theMILVUS_URI-gated tests).