SignConnect - Real-time Sign Language Recognition System

SignConnect is an AI-powered project designed for real-time recognition and translation of Vietnamese Sign Language (VSL). The system features an interactive web interface built with Next.js and an AI engine that processes streaming data via WebSocket, powered by Python (FastAPI, PyTorch, MediaPipe).

---

🏗 Project Structure

The project is divided into two main components:

• . (Root directory): The Frontend web application using Next.js.
• /server: The Backend API & AI Engine responsible for gesture recognition.

---

🛠 Prerequisites

To set up and run the entire project, your machine needs to have:

1. Node.js (v18.x or above recommended) & npm / pnpm / yarn.
2. Python 3.12 (Required for the Backend and running AI models).
3. PostgreSQL & Redis (Running them via Docker is highly recommended).
4. Docker & Docker Compose (For quick environment setup).

---

🚀 Installation & Getting Started

Step 1: Initialize Infrastructure (Database & Redis)

If you have Docker installed, you can quickly spin up PostgreSQL and Redis using the provided configuration file in the server directory:

cd server
docker-compose up -d

(Note: Based on the configuration, the project uses Redis on port 6388 for the Backend and port 6379 for the Frontend. Please ensure these are configured correctly).

---

Step 2: Setup and Run the Backend (AI Server)

The Backend is responsible for receiving video/landmark streams from users, processing them through the AI inference model, and returning the results via WebSocket.

1. Navigate to the server directory (if not already there):

   cd server

2. Create and activate a Virtual Environment:

   python3.12 -m venv .venv
   
   # Activate on macOS/Linux:
   source .venv/bin/activate  
   
   # Activate on Windows:
   .venv\Scripts\activate

3. Install dependencies:

   pip install --upgrade pip
   pip install -r requirements.txt

4. Configure environment variables (.env): Create a .env file inside the server/ directory and populate it with the following configuration (you can copy the snippet below):

   DATABASE_URL="postgresql+asyncpg://postgres:Lewdko@localhost:5432/signconnect"
   JWT_SECRET_KEY="super_secret_jwt_key_for_signconnect"
   REDIS_PASSWORD=""
   HOST="0.0.0.0"
   PORT=8765

5. Start the WebSocket Server:

   python websocket_main.py

   (The server will run on port 8765, ready to accept connections).

---

Step 3: Setup and Run the Frontend (Next.js)

1. Open a new Terminal and navigate to the root directory of the project (/).

2. Install Node packages:

   npm install
   # Or if you use pnpm: pnpm install

3. Configure environment variables (.env): Copy the available configuration template:

   cp .env.example .env

   Note: Open the newly created .env file and fill in your API Keys for Clerk (authentication service) and Deepgram (if utilizing Text-to-Speech/Speech-to-Text features).

4. Start the Web application:

   npm run dev
   # Or: pnpm dev

5. Open your browser and navigate to http://localhost:3000 to experience the app.

---

🧠 For AI Developers / Contributors

This project integrates a complete AI pipeline, from collecting raw data to deploying an optimized model:

• Data Collection (Crawling): An automated tool to download VSL videos via Selenium (server/app/data/collectors/download_data.py).
• Preprocessing & Feature Extraction: Uses MediaPipe to extract coordinates of hand/body joints (Landmarks), standardizing them for the model (server/app/pipelines/data_processer.py).
• Training: Utilizes Metric Learning techniques combined with a Neural Network (Sign Encoder) and FAISS (Vector Database) to resolve class confusion and dramatically reduce Inference Time.

👉 To gain an in-depth understanding of the data flow architecture for both the AI and Server, please read the Server Technical Documentation here.