Skip to content

Latest commit

 

History

History
61 lines (48 loc) · 1.89 KB

README.md

File metadata and controls

61 lines (48 loc) · 1.89 KB

🚀 Distributed Task Manager

A scalable and distributed task manager built with Node.js, Express, MongoDB, and Kafka. This project handles task execution with a worker-based architecture and event-driven messaging.

🛠 Tech Stack

Component Technology Description
Backend Node.js + Express REST API for task management
Database MongoDB Stores tasks and execution details
Queue System Kafka Manages task distribution across workers
Containerization Docker Manages services as containers
Service Communication REST API Communicates between microservices

🔄 Flow of Code Execution

  1. Task Submission → API Service receives a task submission request (POST /tasks).
  2. Queue Processing → API Service publishes the task to Kafka.
  3. Worker Execution → Workers consume tasks from Kafka and process them.
  4. Task Completion & Storage → Worker updates MongoDB with task results.
  5. Centralized DB Service → Centralized MongoDB service for task storage and retrieval.
  6. Retries & Failure Handling → If a task fails, it is retried based on an exponential backoff strategy. Persistent failures go to the Dead Letter Queue (DLQ).

🚀 Getting Started

1️⃣ Clone the Repository

git clone https://github.com/vr-varad/distributed-task-manager.git
cd distributed-task-manager

2️⃣ Install Dependencies

pnpm install

3️⃣ Start Kafka (Docker-based setup)

cd apps/queue-service
docker compose up -d

4️⃣ Start Services

Start API Service:

pnpm run start:api

Start Worker Service:

pnpm run start:worker

Start Scheduler Service:

pnpm run start:scheduler