Intelligent Audio-Visual CC Suggestion System (Goals 1 & 2)

[Siddharth-732]

closes #2

Summary

This PR implements in the middle phase of the Intelligent CC Suggestion Tool. The goal is to automatically identify meaningful non-speech audio events along with the visual in videos to determine whether a non-speech event is important enough to require a caption or not.

Key Features

1. Robust Audio Classification (Goal 1)

• Semantic Label Grouping: Maps 500+ YAMNet classes into 20+ canonical CC categories (e.g., [Gunshot/Explosion], [Animal Sound]).
• Multi-Label Probability Summing: Aggregates scores across similar acoustic classes to provide stable, confident labels.
• Ambient Noise Filtering: Automatically suppresses low-value background sounds (e.g., "Inside, small room").

2. Intelligent Reaction Detection (Goal 2)

• Hybrid Vision Tracking: Uses MediaPipe Pose for full-body tracking and automatically falls back to FaceMesh for close-up head/shoulder shots.
• Scale-Invariant Movement: Calculates movement relative to the person's height, ensuring accuracy regardless of distance from the camera.
• Baseline Comparison: Compares visual state against a pre-event baseline to catch subtle flinches, head-turns, or "startle" reactions.

How to Verify

To ensure a clean run, please follow these steps to set up a local environment:

1. System Requirements
   Python 3.10 or 3.11
   FFmpeg: Must be available in your system path.
   Windows: winget install ffmpeg
   Linux: sudo apt install ffmpeg

2. Environment Setup
   # Clone and enter the directory
git clone [your-repo-link]
cd Intelligent-cc-generation

   # Create a fresh virtual environment
python -m venv .venv

   # Activate the environment
# Windows:
.\.venv\Scripts\activate

   # Linux/Mac:
source .venv/bin/activate

   # Install dependencies (includes specific numpy version fix)
pip install -r requirements.txt

3. Run the Detection Modules

Test Sound Event Detection (Goal 1): python test_goal_1.py "path/to/your/video.mp4"
Test Full Audio-Visual Reaction Sync (Goal 2): python test_goal_2.py "path/to/your/video.mp4"

Note to Maintainer: The test_goal_2.py script automatically utilizes the output from Goal 1 and analyzes the visual frames around those specific timestamps.

ScreenShot of Results

Screenshot explanation - after the first test, the pipeline will create a output folder that will contain the audio of the video as sample.wav, this output will then be analyzed and the output can be seen in the terminal, i.e it detects 3 events.

[Copilot]

Pull request overview

This PR introduces the “Goal 1” sound event detection pipeline for the Intelligent CC Suggestion Tool, adding YAMNet-based audio event detection plus basic audio utilities, extraction, documentation, and initial tests.

Changes:

• Added cc_tool.audio package: FFmpeg-based audio extraction, waveform chunking/normalization, YAMNet inference, and simple adjacent-event merging.
• Added a manual run script (test_goal_1.py), README setup/usage guidance, and Python dependencies.
• Added initial pytest suites for audio utilities/extractor, plus decision/export tests that currently reference missing modules.

Reviewed changes

Copilot reviewed 14 out of 16 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
cc_tool/audio/detector.py	Implements YAMNet loading, inference over chunks, and event merging.
cc_tool/audio/extractor.py	Adds FFmpeg subprocess wrapper for extracting 16kHz mono WAV.
cc_tool/audio/models.py	Adds AudioEvent dataclass and serialization helper.
cc_tool/audio/utils.py	Adds chunking and waveform normalization helpers.
cc_tool/audio/__init__.py	Exposes audio API via package exports.
cc_tool/__init__.py	Marks cc_tool as a package.
tests/test_audio/test_detector.py	Adds tests for chunking and normalization utilities.
tests/test_audio/test_extractor.py	Adds a basic missing-input test for extractor.
tests/test_audio/__init__.py	Marks audio tests as a package.
tests/test_decision/test_combiner.py	Adds tests for a decision combiner that is not present in this PR/repo.
tests/test_decision/test_srt_writer.py	Adds tests for an SRT writer/models that are not present in this PR/repo.
tests/test_decision/__init__.py	Marks decision tests as a package.
test_goal_1.py	Adds a manual run script, but is currently named like a pytest test module.
requirements.txt	Adds TensorFlow/TFHub and other dependencies (includes currently-unused pydantic).
README.md	Documents setup and example usage for Goal 1.
.gitignore	Ignores Python artifacts, venvs, and outputs/ directory.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.