NetraSathi — Screenless Exam Assistance System

Voice-first platform enabling visually impaired students to independently navigate and attempt digital exams — no screen, no sighted assistance required.

---

The Problem

Over 8 million visually impaired students in India have no accessible way to attempt digital examinations independently. Existing systems require sighted assistance, removing autonomy from students who are otherwise fully capable.

NetraSathi solves this by replacing the screen entirely with voice.

---

Demo Flow

1. Admin uploads exam paper (PDF or image)
2. OCR extracts and parses all questions automatically
3. BERT classifies each question as MCQ or Descriptive
4. Admin reviews questions, corrects OCR errors, and sets the answer key
5. Student navigates the entire exam by voice:
   • Questions are read aloud via TTS
   • Student speaks answers → captured by Whisper STT
   • Voice commands handle navigation (next, repeat, skip, submit)
   • Answers are evaluated in real-time using BERT similarity

---

Architecture

Backend — FastAPI + Python

OCR (Tesseract / PyMuPDF)
    → Question Parser (regex, multilingual)
    → BERT Classifier (MCQ vs Descriptive)
    → Voice Interface (Whisper STT + gTTS / pyttsx3)
    → Intent Pipeline (Rule-based → Fuzzy → BERT)
    → Answer Evaluator (regex for MCQ, BERT cosine for Descriptive)
    → Encrypted Audit Log (Fernet)

Frontend — React Native + Expo

• Admin Portal — 3-step flow: Upload → Review Questions → Set Answer Key
• Student Exam Screen — Voice-only interface with animated status indicators, dual timer (exam + per-question), and live progress tracking

---

Tech Stack

Layer	Technologies
Backend	FastAPI, Python, Pydantic
ML / NLP	HuggingFace Transformers, Sentence-Transformers, RapidFuzz
STT	OpenAI Whisper (offline) + Google STT (online fallback)
TTS	gTTS (online) + pyttsx3 (offline fallback)
OCR	Tesseract + PyMuPDF
Frontend	React Native, Expo, React Navigation
Security	Fernet encryption, audit logging

---

Key Features

• Fully offline capable — Whisper STT + pyttsx3 TTS fallback when no internet
• Multilingual — English, Hindi, and Marathi supported
• 3-tier intent detection — Rule-based → Fuzzy matching → BERT zero-shot fallback
• BERT semantic evaluation — cosine similarity for descriptive answers with configurable thresholds
• Haptic feedback — iOS haptic patterns for answer saved, error, and submission
• Encrypted audit trail — every exam event logged and answers encrypted with Fernet
• Real-time dual timer — full exam countdown + per-question response bar

---

Project Structure

netrasathi/
├── backend/
│   ├── routers/          # FastAPI route handlers
│   ├── services/         # Business logic (OCR, STT, intent, evaluation)
│   ├── models/           # Pydantic schemas
│   ├── config/           # Environment settings
│   ├── data/             # Intent patterns JSON
│   ├── main.py           # FastAPI app entry point
│   ├── requirements.txt
│   └── .env.example
│
└── frontend/
    ├── screens/          # AdminHome, ReviewQuestions, AnswerKey, StudentExam
    ├── components/       # QuestionCard, VoiceStatusIndicator, ExamTimer, ProgressBar
    ├── hooks/            # useExamFlow, useVoiceCommand
    ├── services/         # examService.js, api.js
    ├── App.js
    └── package.json

---

Setup

Backend

cd backend
pip install -r requirements.txt
cp .env.example .env
# Edit .env and set your ENCRYPTION_SECRET
uvicorn main:app --reload

Frontend

cd frontend
npm install
# Open config.js and set BASE_URL to your machine's local IP address
# Example: export const BASE_URL = 'http://192.168.1.5:8000';
npx expo start

Note: 127.0.0.1 only works on emulators. For a physical device, use your machine's local network IP.

---

API Reference

Method	Endpoint	Description
POST	/ocr/upload	Upload exam paper (PDF/image)
POST	/classify/{exam_id}	Classify questions as MCQ/Descriptive
POST	/admin/correct	Save reviewed questions
POST	/evaluation/answer-key	Upload correct answers
POST	/voice/speak	Read text aloud via TTS
POST	/voice/listen	Listen for student speech via STT
POST	/intent/detect	Detect voice command intent
POST	/evaluation/evaluate/single	Evaluate one student answer
POST	/evaluation/submit/{exam_id}	Finalize and lock exam submission

---

Environment Variables

Create a .env file in the backend/ folder based on .env.example:

ENCRYPTION_SECRET=your-secret-key-here
UPLOAD_DIR=uploads
CORRECTED_DIR=corrected
RESULTS_DIR=results
CONFIDENCE_THRESHOLD=0.80
AUDIT_LOG_DIR=audit_logs

---

Built For

Hackathon project — assistive technology for accessible education in India.