S3 Service Adapter
| crate | version | docs |
|---|---|---|
| s3s |  |
 |
| s3s-aws |  |
 |
| s3s-sigv2 |  |
 |
| s3s-sigv4 |  |
 |
| s3s-fs |  |
 |
📚 Development documentation for the main branch is available on GitHub Pages.
This experimental project intends to offer an ergonomic adapter for building S3-compatible services.
s3s implements Amazon S3 REST API in the form of a generic hyper service. S3-compatible services can focus on the S3 API itself and don't have to care about the HTTP layer.
s3s-aws provides useful types and integration with aws-sdk-s3.
s3s-fs implements the S3 API based on file system, as a sample implementation. It is designed for integration testing, which can be used to mock an S3 client. It also provides a binary for debugging. Play it!
The diagram above shows how s3s works.
s3s converts HTTP requests to operation inputs before calling the user-defined service.
s3s converts operation outputs or errors to HTTP responses after calling the user-defined service.
The data types, serialization and deserialization are generated from the smithy model in aws-sdk-rust repository. We apply manual hacks to fix some problems in smithy server codegen and make s3s ready to use now.
S3Service and other adapters in this project are not a complete security boundary. If they are exposed to the Internet directly, they may be attacked.
It is up to the user to implement security enhancements such as HTTP body length limits, object-size limits, rate limits and back pressure.
Authentication is required for production deployments. Without calling set_auth, the service accepts anonymous (unsigned) requests and skips authorization entirely: every S3 operation is open to any client that can reach the service, and signed requests fail with NotImplemented because no authentication provider is configured. A forgotten set_auth turns the service into a publicly readable and writable endpoint.
For streaming uploads (PUT Object, UploadPart), s3s applies a default 5 GiB object-size limit matching the AWS single-PUT limit; set S3Config::put_object_max_size to None to disable it and enforce deployment-specific caps in the S3 implementation. For production, set it explicitly even though the default is already 5 GiB. POST Object keeps using S3Config::post_object_max_file_size.
List-type responses (ListObjects, ListBuckets, ...) are serialized in full by s3s: their memory usage grows with the number of entries the S3 implementation returns. Implementations should paginate (max-keys / continuation tokens) and deployments should bound response sizes.
Docker images are available at GitHub Container Registry (GHCR).
See Docker documentation for usage details.
We have a reward funds pool for contributors: s3s-project#174
If my open-source work has been helpful to you, please sponsor me.
Every little bit helps. Thank you!