Add comprehensive security scanning module with SAST, dependency, and configuration analysis

[Copilot]

Implements a cybersecurity analysis module that monitors code, dependencies, and configurations for vulnerabilities using static analysis, CVE detection, and security best practices aligned with OWASP/SANS/MITRE standards.

Implementation

Core Security Scanners

• sast_scanner.py - Detects code injection (eval/exec), SQL/command injection, hardcoded secrets, weak crypto
• dependency_scanner.py - Scans requirements.txt/pyproject.toml for known CVEs and unpinned versions
• config_scanner.py - Analyzes .env files, Dockerfiles, GitHub Actions workflows for exposed secrets and misconfigurations
• security_reporter.py - Generates reports with 5-tier severity ratings, remediation guidance, CVE/CWE/OWASP references
• security_validator.py - Orchestrates scanners and produces comprehensive reports
• knowledge_base.py - Integrates OWASP Top 10 2021, SANS Top 25 CWE, MITRE ATT&CK patterns

CLI Integration

# Run security scan
agent-tester security --path . --severity high --format json

# Generates detailed report with file locations and fix recommendations

API Usage

from agent_tester.security import SecurityValidator

validator = SecurityValidator()
report = validator.validate_repository("./project")

for issue in report.critical_issues:
    print(f"{issue.title} at {issue.file_path}:{issue.line_number}")
    print(f"Fix: {issue.recommendation}")

CI/CD Integration

• GitHub Actions workflow for automated scanning on push/PR
• Daily scheduled scans with artifact uploads
• PR comments with security summaries

Testing & Validation

• 24 comprehensive tests covering all scanners and report generation
• CodeQL scan clean (0 alerts)
• Example scripts demonstrating usage
• Documentation: SECURITY_MODULE.md, CYBERSECURITY_ROLE.md

Performance

Scans ~30 Python files in ~0.05s with comprehensive error logging for diagnostics.

Original prompt

AI Agent Role: Cybersecurity & Secure-Code Contributor

1. Role Definition

Primary Role: Act as a cybersecurity reviewer and security-focused contributor in an open-source software repository.

Scope of Work:

Monitor all existing code,

Monitor all incoming or future code additions,

Perform continuous defensive security analysis,

Identify, report, and recommend fixes for vulnerabilities,

Provide insights based on latest cyber-security best practices.

2. Core Responsibilities
   2.1 Continuous Repository Monitoring

Track all files, commits, branches, pull requests, and new contributions.

Maintain awareness of changes in:

Application logic

Dependencies

Infrastructure-as-code

Configuration files

Secrets or credentials

Detect anomalies, insecure patterns, or high-risk changes.

2.2 Security Testing & Analysis

Perform non-destructive, ethical security evaluations, including:

Static Application Security Testing (SAST):

Scan source code continuously for vulnerabilities.

Detect insecure functions, tainted input, unsafe patterns.

Dynamic Analysis (DAST):

Safe, controlled execution-based security testing.

Identify misconfigurations, unsafe endpoints, or runtime issues.

Dependency & Supply-Chain Security:

Analyze third-party libraries for vulnerabilities (CVEs).

Recommend updates or safer alternatives.

Configuration Security Checks:

Identify insecure defaults.

Detect exposed secrets, tokens, API keys, passwords.

2.3 Penetration-Testing Simulations

Perform defensive pen-testing simulations, within ethical limits:

Attempt safe exploitation of discovered vulnerabilities.

Validate severity and reproducibility.

Confirm whether issues are exploitable and require immediate fixes.

3. Reporting & Recommendations
   3.1 Issue Detection

For every vulnerability found:

Provide a clear description.

Identify file(s), function(s), commit(s) where it occurs.

Rate severity (Low / Medium / High / Critical).

Explain potential impact and attack vector.

3.2 Proposed Fixes

For each issue:

Suggest secure coding practices.

Provide corrected code samples when appropriate.

Recommend configuration changes, dependency upgrades, or architecture improvements.

3.3 Continuous Summaries

Create periodic security reports.

Maintain a changelog of:

Detected issues

Actions taken

Remaining risks

Improvement recommendations

4. Knowledge Requirements

The agent must stay updated with the latest cybersecurity information, including:

New CVEs, exploits, and patch information.

State-of-the-art defensive measures.

Secure coding standards:

OWASP ASVS

OWASP Top 10

SANS Top 25

MITRE ATT&CK knowledge

Best practices for:

Web security

API security

Cloud security

Authentication/authorization

Cryptography

CI/CD pipeline security

5. Operational Constraints

All testing and exploitation simulations must be ethical, non-destructive, and limited to the repository itself.

The agent must not generate, use, or promote harmful or illegal attack tools.

The agent operates strictly in a defensive security capacity.

[Chronological Review: The conversation began with the user requesting a review of the code and guidance on running it. The user encountered issues with pytest and Python errors, prompting further inquiries about testing agents and creating agents in Azure AI Foundry. The user expressed a desire to clean up the codebase for production readiness, followed by a request to check for security best practices. The user then initiated the creation of a GitHub repository and aimed to develop an open-source agent testing framework. The conversation progressed with the user defining a cybersecurity-focused AI agent role, detailing responsibilities, reporting, knowledge requirements, and operational constraints.]

[Intent Mapping:

1. "review the code and let me know how to run it and what's missing?" - User seeks guidance on code execution.
2. "It's unable to recognize pytest and python is also giving some error" - User reports issues with Python and pytest.
3. "how would I test some real life agents using this script?" - User wants to understand practical applications of the script.
4. "help me create agents in azure AI foundry" - User requests assistance in agent creation.
5. "what's this error" - User seeks clarification on an error message.
6. "Let's do a thorough cleanup to make it production ready" - User aims to prepare the codebase for production.
7. "Check for security and production best practice risk" - User requests a security review of the project.
8. "How is this thing, I mean how our code is testing the agent's responses" - User inquires about the framework's capabilities.
9. "Do a deep internet search and find out if our tool/framework is actually a unique one" - User wants to assess the uniqueness and necessity of the framework.
10. "Let's imp...

Created from VS Code via the GitHub Pull Request extension.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.