Analyzing data spread across many computers is inherently difficult. AnalyticsMesh solves this problem by providing real-time insights using a local summary of your distributed data, which eliminates network round-trips and enables offline operations.
With AnalyticsMesh, nodes analyze data independently and seamlessly merge their insights into a unified view. This collaborative approach enables analysis of massive datasets and data streams while ensuring data remains accessible even when networks or devices fail.
A distinctive aspect of AnalyticsMesh is its novel enhancement of HyperLogLog, a probabilistic data structure, into a CRDT (Conflict-Free Replicated Data Type). This transformation enables efficient and scalable approximate count-distinct operations, which are essential for big data analytics.
Key features of AnalyticsMesh include:
-
Horizontally Scalable Architecture: AnalyticsMesh allows for easy scaling by adding more nodes, enhancing capacity and performance without disrupting existing operations.
-
Fault Tolerance: AnalyticsMesh withstands network partitions, disruptions, delays, and node failures, ensuring continuous data processing without data loss.
-
Strong Eventual Consistency: Leveraging HyperLogLog's CRDT properties, AnalyticsMesh guarantees eventual consistency in data across all distributed nodes.
-
Tunable Durability and Atomicity: AnalyticsMesh allows tuning data durability and atomicity guarantees, adapting to various operational requirements.
-
Containerization Support: Containerization simplifies scalable deployment by providing consistent, isolated environments across platforms.
- Git
- Apache Thrift compiler
- Make
- Python >= 3.12
- Install prerequisites (Ubuntu/Debian):
sudo apt update
sudo apt install -y git thrift-compiler make python3.12- Clone the repository and navigate to the project directory:
git clone https://github.com/asjadsyed/AnalyticsMesh
cd AnalyticsMesh- Install Python dependencies:
python3 -m pip install -r requirements.txt- Build the project:
make- Run the application:
python3 ./src/main.py --help- Git
- Docker
- Docker Compose
- Install prerequisites (Ubuntu/Debian):
sudo apt update
sudo apt install -y git
sudo snap install docker- Clone the repository and navigate to the project directory:
git clone https://github.com/asjadsyed/AnalyticsMesh
cd AnalyticsMesh- Build Docker containers:
docker-compose build- Run the application:
docker-compose run analytics-mesh --helpAnalyticsMesh is licensed under the Apache License 2.0. Refer to the LICENSE file for details.