Initial commit: ROA2WEB - FastAPI + Vue.js + Telegram Bot
Modern ERP Reports Application with microservices architecture Tech Stack: - Backend: FastAPI + python-oracledb (Oracle DB integration) - Frontend: Vue.js 3 + PrimeVue + Vite - Telegram Bot: python-telegram-bot + SQLite - Infrastructure: Shared database pool, JWT authentication, SSH tunnel Features: - FastAPI backend with async Oracle connection pool - Vue.js 3 responsive frontend with PrimeVue components - Telegram bot alternative interface - Microservices architecture with shared components - Complete deployment support (Linux Docker + Windows IIS) - Comprehensive testing (Playwright E2E + pytest) Repository Structure: - reports-app/ - Main application (backend, frontend, telegram-bot) - shared/ - Shared components (database pool, auth, utils) - deployment/ - Deployment scripts (Linux & Windows) - docs/ - Project documentation - security/ - Security scanning and git hooks
This commit is contained in:
277
security/README.md
Normal file
277
security/README.md
Normal file
@@ -0,0 +1,277 @@
|
||||
# 🔒 ROA2WEB Security Audit Implementation
|
||||
|
||||
## 📋 Overview
|
||||
|
||||
This directory contains comprehensive security tools for the ROA2WEB project, implemented based on the critical findings in `SECURITY_AUDIT_CONTEXT.md`. The implementation addresses the discovered secrets in git history and provides ongoing protection against future security violations.
|
||||
|
||||
## 🚨 Critical Issues Addressed
|
||||
|
||||
### Secrets Found in Repository:
|
||||
- **Oracle Password**: `ROMFASTSOFT` (in multiple .env files)
|
||||
- **User Passwords**: `{"marius": "Parola81", "eli": "eli"}`
|
||||
- **SSH Private Key**: `roa_oracle_server`
|
||||
- **Environment Files**: Multiple .env files with production credentials
|
||||
|
||||
## 🛠️ Security Tools Implemented
|
||||
|
||||
### 1. 🔍 `secrets_scanner.py`
|
||||
Advanced secrets detection tool with pattern-based scanning.
|
||||
|
||||
**Features:**
|
||||
- Scans current files for secrets and credentials
|
||||
- Optional git history scanning
|
||||
- Pattern-based detection with high accuracy
|
||||
- JSON report generation
|
||||
- Integration ready for CI/CD
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Basic scan
|
||||
python security/secrets_scanner.py
|
||||
|
||||
# Scan with git history (slow but thorough)
|
||||
python security/secrets_scanner.py --scan-git-history
|
||||
|
||||
# Save detailed report
|
||||
python security/secrets_scanner.py --save-report security_report.json
|
||||
```
|
||||
|
||||
### 2. 🧹 `git_cleanup.py`
|
||||
Git history cleanup tool for removing secrets from repository history.
|
||||
|
||||
**Features:**
|
||||
- Complete repository backup before cleanup
|
||||
- Removes sensitive files from git history
|
||||
- Replaces secret patterns in commits
|
||||
- Verification of cleanup completion
|
||||
- Detailed logging of all actions
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Create backup only
|
||||
python security/git_cleanup.py --backup
|
||||
|
||||
# Scan for secrets in history
|
||||
python security/git_cleanup.py --scan
|
||||
|
||||
# Run complete cleanup (DANGEROUS - rewrites history)
|
||||
python security/git_cleanup.py --cleanup
|
||||
|
||||
# Force cleanup without prompts
|
||||
python security/git_cleanup.py --cleanup --force
|
||||
```
|
||||
|
||||
### 3. 🪝 Git Hooks
|
||||
Pre-commit and commit-msg hooks to prevent future secrets commits.
|
||||
|
||||
**Installation:**
|
||||
```bash
|
||||
# Install all security hooks
|
||||
./security/install_hooks.sh
|
||||
```
|
||||
|
||||
**Features:**
|
||||
- **pre-commit**: Scans staged files for secrets before commit
|
||||
- **commit-msg**: Validates commit messages for suspicious keywords
|
||||
- Blocks commits containing credentials
|
||||
- Provides actionable remediation guidance
|
||||
|
||||
### 4. 🛡️ Enhanced .gitignore
|
||||
Comprehensive patterns to prevent committing sensitive files.
|
||||
|
||||
**Added Protections:**
|
||||
- All environment files (except .example)
|
||||
- SSH keys and certificates
|
||||
- Secrets and credentials files
|
||||
- Database connection files
|
||||
- Production configurations
|
||||
- Development tool caches
|
||||
|
||||
## 📊 Security Scanning Patterns
|
||||
|
||||
### Critical Patterns Detected:
|
||||
- `ORACLE_PASSWORD=*`
|
||||
- `VALID_USERS=*`
|
||||
- SSH private key headers
|
||||
- AWS access keys
|
||||
- Bearer tokens
|
||||
- Generic password patterns
|
||||
- Connection strings
|
||||
|
||||
### Suspicious File Patterns:
|
||||
- `*.env` (except .example)
|
||||
- `*_rsa`, `*.key`, `*.pem`
|
||||
- `*secret*`, `*credential*`, `*password*`
|
||||
- `config.prod.*`
|
||||
|
||||
## 🚀 Quick Start Guide
|
||||
|
||||
### 1. Immediate Security Scan
|
||||
```bash
|
||||
# Run comprehensive security scan
|
||||
python security/secrets_scanner.py --save-report current_security_status.json
|
||||
```
|
||||
|
||||
### 2. Install Git Hooks
|
||||
```bash
|
||||
# Prevent future secrets commits
|
||||
./security/install_hooks.sh
|
||||
```
|
||||
|
||||
### 3. (CRITICAL) Git History Cleanup
|
||||
⚠️ **WARNING**: This rewrites git history. Coordinate with your team first!
|
||||
|
||||
```bash
|
||||
# 1. Create backup
|
||||
python security/git_cleanup.py --backup
|
||||
|
||||
# 2. Scan for secrets in history
|
||||
python security/git_cleanup.py --scan
|
||||
|
||||
# 3. Run cleanup (after team coordination)
|
||||
python security/git_cleanup.py --cleanup
|
||||
```
|
||||
|
||||
### 4. Regenerate Compromised Credentials
|
||||
🔑 **MANDATORY**: All exposed credentials must be regenerated:
|
||||
- Oracle password: `ROMFASTSOFT`
|
||||
- User passwords: `Parola81`, `eli`
|
||||
- SSH key: `roa_oracle_server`
|
||||
|
||||
## 📋 Security Checklist
|
||||
|
||||
### ✅ Immediate Actions (DONE):
|
||||
- [x] Enhanced root .gitignore with security patterns
|
||||
- [x] Implemented secrets scanner tool
|
||||
- [x] Created git history cleanup tools
|
||||
- [x] Installed git hooks for prevention
|
||||
- [x] Documented security procedures
|
||||
|
||||
### 🔧 Required Actions (TODO):
|
||||
- [ ] **CRITICAL**: Regenerate Oracle password (`ROMFASTSOFT`)
|
||||
- [ ] **CRITICAL**: Regenerate user passwords (`Parola81`, `eli`)
|
||||
- [ ] **CRITICAL**: Regenerate SSH key (`roa_oracle_server`)
|
||||
- [ ] Run git history cleanup (`git_cleanup.py --cleanup`)
|
||||
- [ ] Force push cleaned history to all remotes
|
||||
- [ ] Notify team to re-clone repository
|
||||
- [ ] Update production environment with new credentials
|
||||
|
||||
### 🔒 Ongoing Security:
|
||||
- [ ] Regular security scans in CI/CD pipeline
|
||||
- [ ] Quarterly security audits
|
||||
- [ ] Team training on secrets management
|
||||
- [ ] Implement proper secrets management system
|
||||
|
||||
## 🏗️ CI/CD Integration
|
||||
|
||||
### GitHub Actions Example:
|
||||
```yaml
|
||||
name: Security Scan
|
||||
on: [push, pull_request]
|
||||
jobs:
|
||||
security:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
- name: Security Scan
|
||||
run: python security/secrets_scanner.py
|
||||
```
|
||||
|
||||
### Pre-commit Hook Integration:
|
||||
```yaml
|
||||
# .pre-commit-config.yaml
|
||||
repos:
|
||||
- repo: local
|
||||
hooks:
|
||||
- id: secrets-scan
|
||||
name: Secrets Scanner
|
||||
entry: python security/secrets_scanner.py
|
||||
language: system
|
||||
pass_filenames: false
|
||||
```
|
||||
|
||||
## 🆘 Emergency Response
|
||||
|
||||
### If Secrets Are Accidentally Committed:
|
||||
|
||||
1. **IMMEDIATE**:
|
||||
```bash
|
||||
# Run emergency scan
|
||||
python security/secrets_scanner.py --scan-git-history
|
||||
```
|
||||
|
||||
2. **URGENT**:
|
||||
```bash
|
||||
# Regenerate exposed credentials immediately
|
||||
# Update production systems
|
||||
```
|
||||
|
||||
3. **CLEANUP**:
|
||||
```bash
|
||||
# Clean git history
|
||||
python security/git_cleanup.py --cleanup --force
|
||||
```
|
||||
|
||||
## 📞 Support and Reporting
|
||||
|
||||
### Security Issues:
|
||||
- Report immediately to security team
|
||||
- Use encrypted communication for sensitive details
|
||||
- Follow incident response procedures
|
||||
|
||||
### Tool Issues:
|
||||
- Check logs in security/ directory
|
||||
- Review tool documentation
|
||||
- Test in safe environment first
|
||||
|
||||
## 📚 Best Practices
|
||||
|
||||
### 1. Secrets Management:
|
||||
- Use environment variables for all secrets
|
||||
- Implement proper secrets management (Vault, AWS Secrets Manager)
|
||||
- Never hardcode credentials in source code
|
||||
- Use `.env.example` for configuration templates
|
||||
|
||||
### 2. Git Practices:
|
||||
- Always run security scan before commits
|
||||
- Use meaningful commit messages
|
||||
- Review changes before staging
|
||||
- Keep git history clean and professional
|
||||
|
||||
### 3. Development Workflow:
|
||||
- Use separate credentials for development/testing
|
||||
- Regularly rotate credentials
|
||||
- Monitor for credential exposure
|
||||
- Train team on security practices
|
||||
|
||||
## 🔧 Troubleshooting
|
||||
|
||||
### Common Issues:
|
||||
|
||||
1. **Git hooks failing**:
|
||||
```bash
|
||||
# Reinstall hooks
|
||||
./security/install_hooks.sh
|
||||
```
|
||||
|
||||
2. **Scanner false positives**:
|
||||
- Review patterns in `secrets_scanner.py`
|
||||
- Add exceptions for legitimate uses
|
||||
- Update pattern matching rules
|
||||
|
||||
3. **History cleanup failures**:
|
||||
- Ensure clean working directory
|
||||
- Create backup before attempting
|
||||
- Check git permissions and status
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ CRITICAL REMINDER
|
||||
|
||||
**The credentials found in this repository (`ROMFASTSOFT`, `Parola81`) are potentially compromised and MUST be regenerated immediately. Git history cleanup should be performed BEFORE any other development work to prevent propagation to other repository clones.**
|
||||
|
||||
---
|
||||
|
||||
*Security implementation completed: 2025-08-03*
|
||||
*Tools version: 1.0*
|
||||
*Next security review: 2025-09-03*
|
||||
271
security/SECURITY_PROCEDURES.md
Normal file
271
security/SECURITY_PROCEDURES.md
Normal file
@@ -0,0 +1,271 @@
|
||||
# 🔒 ROA2WEB Security Procedures
|
||||
|
||||
## 📋 Security Incident Response Plan
|
||||
|
||||
### 🚨 CRITICAL: Credentials Compromise Response
|
||||
|
||||
**IMMEDIATE ACTIONS** (within 1 hour):
|
||||
|
||||
1. **Assess Scope of Compromise**:
|
||||
```bash
|
||||
# Run emergency security scan
|
||||
python security/secrets_scanner.py --scan-git-history --save-report emergency_scan.json
|
||||
```
|
||||
|
||||
2. **Isolate Systems**:
|
||||
- Change Oracle database password immediately
|
||||
- Rotate SSH keys for server access
|
||||
- Update application authentication credentials
|
||||
- Notify infrastructure team
|
||||
|
||||
3. **Document Incident**:
|
||||
- Record time of discovery
|
||||
- List all potentially compromised credentials
|
||||
- Identify affected systems and users
|
||||
- Track remediation actions
|
||||
|
||||
### 🔧 REMEDIATION STEPS
|
||||
|
||||
#### Step 1: Credential Regeneration
|
||||
```bash
|
||||
# Oracle Database
|
||||
# 1. Connect to Oracle as admin
|
||||
# 2. Change CONTAFIN_ORACLE password
|
||||
ALTER USER CONTAFIN_ORACLE IDENTIFIED BY "NEW_SECURE_PASSWORD";
|
||||
|
||||
# SSH Keys
|
||||
# 1. Generate new SSH key pair
|
||||
ssh-keygen -t rsa -b 4096 -C "roa2web-$(date +%Y%m%d)" -f roa_oracle_server_new
|
||||
|
||||
# 2. Update server authorized_keys
|
||||
# 3. Test connectivity with new key
|
||||
# 4. Remove old key from server
|
||||
```
|
||||
|
||||
#### Step 2: Git History Cleanup
|
||||
```bash
|
||||
# COORDINATE WITH TEAM FIRST!
|
||||
# 1. Backup repository
|
||||
python security/git_cleanup.py --backup
|
||||
|
||||
# 2. Clean git history
|
||||
python security/git_cleanup.py --cleanup
|
||||
|
||||
# 3. Force push to all remotes
|
||||
git push --force-with-lease --all origin
|
||||
git push --force-with-lease --tags origin
|
||||
|
||||
# 4. Notify team to re-clone
|
||||
```
|
||||
|
||||
#### Step 3: System Updates
|
||||
```bash
|
||||
# Update all environment files
|
||||
# 1. roa2web/.env
|
||||
# 2. roa2web/reports-app/backend/.env
|
||||
# 3. Production environment variables
|
||||
|
||||
# Restart all services
|
||||
# 1. Backend FastAPI application
|
||||
# 2. Frontend Vue.js application
|
||||
# 3. Database connections
|
||||
# 4. SSH tunnel services
|
||||
```
|
||||
|
||||
## 🛡️ Preventive Security Measures
|
||||
|
||||
### Daily Security Checklist
|
||||
|
||||
- [ ] Run security scanner on active branches
|
||||
- [ ] Review new commits for potential secrets
|
||||
- [ ] Monitor system access logs
|
||||
- [ ] Check environment file changes
|
||||
- [ ] Verify git hooks are active
|
||||
|
||||
### Weekly Security Tasks
|
||||
|
||||
- [ ] Review security scan reports
|
||||
- [ ] Update security patterns if needed
|
||||
- [ ] Audit user access permissions
|
||||
- [ ] Check for new security vulnerabilities
|
||||
- [ ] Review backup and recovery procedures
|
||||
|
||||
### Monthly Security Review
|
||||
|
||||
- [ ] Comprehensive repository security audit
|
||||
- [ ] Team security training refresh
|
||||
- [ ] Update security documentation
|
||||
- [ ] Review and test incident response plan
|
||||
- [ ] Credential rotation assessment
|
||||
|
||||
## 🔍 Security Monitoring
|
||||
|
||||
### Automated Monitoring
|
||||
```bash
|
||||
# Set up cron job for daily scans
|
||||
# Add to crontab: crontab -e
|
||||
0 9 * * * cd /path/to/roa-flask && python security/secrets_scanner.py --save-report daily_scan_$(date +\%Y\%m\%d).json
|
||||
|
||||
# Weekly comprehensive scan
|
||||
0 9 * * 1 cd /path/to/roa-flask && python security/secrets_scanner.py --scan-git-history --save-report weekly_scan_$(date +\%Y\%m\%d).json
|
||||
```
|
||||
|
||||
### Alert Triggers
|
||||
- New secrets detected in commits
|
||||
- Suspicious file patterns added
|
||||
- Failed security scans
|
||||
- Unauthorized access attempts
|
||||
- Environment file modifications
|
||||
|
||||
## 📊 Security Metrics and KPIs
|
||||
|
||||
### Track These Metrics:
|
||||
- Number of security violations per month
|
||||
- Time to detect security issues
|
||||
- Time to remediate security issues
|
||||
- Git hook effectiveness rate
|
||||
- Team security training completion
|
||||
|
||||
### Monthly Security Report Template:
|
||||
```
|
||||
ROA2WEB Security Report - [Month/Year]
|
||||
|
||||
📈 Metrics:
|
||||
- Security scans performed: X
|
||||
- Violations detected: X
|
||||
- Violations remediated: X
|
||||
- Average detection time: X hours
|
||||
- Average remediation time: X hours
|
||||
|
||||
🔍 Key Findings:
|
||||
- [List significant security events]
|
||||
- [Pattern analysis]
|
||||
- [Trend identification]
|
||||
|
||||
🎯 Action Items:
|
||||
- [Specific security improvements needed]
|
||||
- [Training requirements]
|
||||
- [Process improvements]
|
||||
|
||||
📋 Recommendations:
|
||||
- [Strategic security initiatives]
|
||||
- [Tool improvements]
|
||||
- [Policy updates]
|
||||
```
|
||||
|
||||
## 🎓 Team Security Training
|
||||
|
||||
### Required Training Topics:
|
||||
|
||||
1. **Secrets Management**:
|
||||
- What constitutes a secret
|
||||
- Proper handling of credentials
|
||||
- Environment variable usage
|
||||
- Secrets management systems
|
||||
|
||||
2. **Git Security**:
|
||||
- Pre-commit security checks
|
||||
- Proper commit message practices
|
||||
- History rewriting consequences
|
||||
- Credential exposure prevention
|
||||
|
||||
3. **Incident Response**:
|
||||
- Recognizing security incidents
|
||||
- Immediate response procedures
|
||||
- Escalation protocols
|
||||
- Post-incident analysis
|
||||
|
||||
### Training Schedule:
|
||||
- **New team members**: Security orientation (first week)
|
||||
- **All team members**: Quarterly security refresh
|
||||
- **Security incidents**: Immediate post-incident training
|
||||
- **Tool updates**: Training when new security tools introduced
|
||||
|
||||
## 🔧 Tool Maintenance
|
||||
|
||||
### Monthly Tool Updates:
|
||||
```bash
|
||||
# Update security patterns
|
||||
# 1. Review new threat intelligence
|
||||
# 2. Update pattern definitions in secrets_scanner.py
|
||||
# 3. Test pattern effectiveness
|
||||
# 4. Deploy updated patterns
|
||||
|
||||
# Verify tool functionality
|
||||
python security/secrets_scanner.py --verbose
|
||||
./security/install_hooks.sh
|
||||
```
|
||||
|
||||
### Tool Health Checks:
|
||||
- Verify git hooks are functioning
|
||||
- Test scanner pattern effectiveness
|
||||
- Check cleanup tool safety measures
|
||||
- Validate backup procedures
|
||||
|
||||
## 📞 Emergency Contacts
|
||||
|
||||
### Security Incident Response Team:
|
||||
- **Primary**: [Security Lead Name] - [Contact Info]
|
||||
- **Secondary**: [DevOps Lead Name] - [Contact Info]
|
||||
- **Escalation**: [CTO/Technical Director] - [Contact Info]
|
||||
|
||||
### External Resources:
|
||||
- **Oracle Support**: [Oracle Support Details]
|
||||
- **Infrastructure Provider**: [Cloud Provider Support]
|
||||
- **Security Consultant**: [External Security Expert]
|
||||
|
||||
## 📋 Compliance and Auditing
|
||||
|
||||
### Regular Audit Requirements:
|
||||
- **Monthly**: Internal security review
|
||||
- **Quarterly**: Comprehensive security audit
|
||||
- **Annually**: External security assessment
|
||||
- **Ad-hoc**: Post-incident security review
|
||||
|
||||
### Audit Checklist:
|
||||
- [ ] All secrets properly managed
|
||||
- [ ] Git history clean of credentials
|
||||
- [ ] Security tools functioning correctly
|
||||
- [ ] Team training up to date
|
||||
- [ ] Incident response plan current
|
||||
- [ ] Backup and recovery tested
|
||||
- [ ] Access controls properly configured
|
||||
- [ ] Documentation updated
|
||||
|
||||
### Compliance Standards:
|
||||
- Follow OWASP security guidelines
|
||||
- Implement ISO 27001 practices where applicable
|
||||
- Ensure GDPR compliance for user data
|
||||
- Meet industry-specific security requirements
|
||||
|
||||
## 🚀 Future Security Improvements
|
||||
|
||||
### Short-term (1-3 months):
|
||||
- [ ] Implement automated secrets management system
|
||||
- [ ] Add security scanning to CI/CD pipeline
|
||||
- [ ] Enhance monitoring and alerting
|
||||
- [ ] Improve team security training program
|
||||
|
||||
### Medium-term (3-6 months):
|
||||
- [ ] Deploy centralized secrets management (Vault/AWS Secrets Manager)
|
||||
- [ ] Implement security scanning in IDE
|
||||
- [ ] Add security metrics dashboard
|
||||
- [ ] Establish security champion program
|
||||
|
||||
### Long-term (6-12 months):
|
||||
- [ ] Full security automation pipeline
|
||||
- [ ] Advanced threat detection
|
||||
- [ ] Security compliance automation
|
||||
- [ ] Comprehensive security culture program
|
||||
|
||||
---
|
||||
|
||||
## ⚠️ CRITICAL REMINDER
|
||||
|
||||
**This document must be reviewed and updated after any security incident. All team members must be familiar with these procedures and know how to execute them under pressure.**
|
||||
|
||||
---
|
||||
|
||||
*Document Version: 1.0*
|
||||
*Last Updated: 2025-08-03*
|
||||
*Next Review: 2025-09-03*
|
||||
368
security/git_cleanup.py
Normal file
368
security/git_cleanup.py
Normal file
@@ -0,0 +1,368 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
🧹 ROA2WEB Git History Cleanup Tool
|
||||
Safely removes secrets from git history using BFG Repo-Cleaner and git filter-branch.
|
||||
|
||||
⚠️ WARNING: This tool rewrites git history. Make sure to:
|
||||
1. Create a complete backup of your repository
|
||||
2. Coordinate with all team members
|
||||
3. Force-push to all remotes after cleanup
|
||||
4. Regenerate all compromised credentials
|
||||
|
||||
Usage:
|
||||
python security/git_cleanup.py --backup --scan --cleanup [--force]
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
import argparse
|
||||
import shutil
|
||||
import json
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import List, Dict
|
||||
|
||||
class GitHistoryCleanup:
|
||||
"""Git history cleanup and secrets removal tool"""
|
||||
|
||||
def __init__(self, repo_path: str = "."):
|
||||
self.repo_path = Path(repo_path).resolve()
|
||||
self.backup_path = None
|
||||
self.cleanup_log = []
|
||||
|
||||
# Files and patterns to remove from history
|
||||
self.FILES_TO_REMOVE = [
|
||||
"app/.env",
|
||||
"roa2web/reports-app/backend/.env",
|
||||
"roa2web/.env",
|
||||
"roa2web/.env.development",
|
||||
"roa2web/.env.production",
|
||||
"roa2web/ssh-tunnel/roa_oracle_server"
|
||||
]
|
||||
|
||||
# Text patterns to replace in history
|
||||
self.SECRETS_TO_REPLACE = {
|
||||
"ACTUAL_ORACLE_PASS": "***REMOVED***",
|
||||
"ACTUAL_USER_PASS": "***REMOVED***",
|
||||
"DB_PASSWORD=ACTUAL_ORACLE_PASS": "DB_PASSWORD=***REMOVED***",
|
||||
'"marius": "ACTUAL_USER_PASS"': '"marius": "***REMOVED***"',
|
||||
'"eli": "eli"': '"eli": "***REMOVED***"'
|
||||
}
|
||||
|
||||
def log_action(self, action: str, details: str = "") -> None:
|
||||
"""Log cleanup actions"""
|
||||
timestamp = datetime.now().isoformat()
|
||||
log_entry = {
|
||||
"timestamp": timestamp,
|
||||
"action": action,
|
||||
"details": details
|
||||
}
|
||||
self.cleanup_log.append(log_entry)
|
||||
print(f"📝 {timestamp}: {action}")
|
||||
if details:
|
||||
print(f" Details: {details}")
|
||||
|
||||
def check_prerequisites(self) -> bool:
|
||||
"""Check if git and required tools are available"""
|
||||
try:
|
||||
# Check git
|
||||
subprocess.run(['git', '--version'], check=True, capture_output=True)
|
||||
|
||||
# Check if we're in a git repo
|
||||
subprocess.run(['git', 'status'], cwd=self.repo_path, check=True, capture_output=True)
|
||||
|
||||
self.log_action("Prerequisites check passed")
|
||||
return True
|
||||
|
||||
except (subprocess.CalledProcessError, FileNotFoundError):
|
||||
print("❌ Error: Git not available or not in a git repository")
|
||||
return False
|
||||
|
||||
def create_backup(self) -> bool:
|
||||
"""Create complete repository backup"""
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
backup_name = f"roa2web_backup_{timestamp}"
|
||||
self.backup_path = self.repo_path.parent / backup_name
|
||||
|
||||
try:
|
||||
print(f"💾 Creating backup at: {self.backup_path}")
|
||||
|
||||
# Use git clone to create a complete backup with all history
|
||||
subprocess.run([
|
||||
'git', 'clone', '--mirror',
|
||||
str(self.repo_path),
|
||||
str(self.backup_path)
|
||||
], check=True)
|
||||
|
||||
self.log_action("Backup created", str(self.backup_path))
|
||||
print(f"✅ Backup created successfully: {self.backup_path}")
|
||||
return True
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"❌ Backup failed: {e}")
|
||||
return False
|
||||
|
||||
def scan_for_secrets(self) -> Dict:
|
||||
"""Scan repository for secrets that need cleanup"""
|
||||
print("🔍 Scanning for secrets in git history...")
|
||||
|
||||
secrets_found = {
|
||||
"files_with_secrets": [],
|
||||
"commits_with_secrets": [],
|
||||
"patterns_found": {}
|
||||
}
|
||||
|
||||
try:
|
||||
# Check if files exist in git history
|
||||
for file_path in self.FILES_TO_REMOVE:
|
||||
result = subprocess.run([
|
||||
'git', 'log', '--oneline', '--', file_path
|
||||
], cwd=self.repo_path, capture_output=True, text=True)
|
||||
|
||||
if result.stdout.strip():
|
||||
secrets_found["files_with_secrets"].append(file_path)
|
||||
print(f" 📄 Found in history: {file_path}")
|
||||
|
||||
# Check for secret patterns in git log
|
||||
for secret_pattern in self.SECRETS_TO_REPLACE.keys():
|
||||
result = subprocess.run([
|
||||
'git', 'log', '-S', secret_pattern, '--oneline'
|
||||
], cwd=self.repo_path, capture_output=True, text=True)
|
||||
|
||||
if result.stdout.strip():
|
||||
commits = result.stdout.strip().split('\n')
|
||||
secrets_found["patterns_found"][secret_pattern] = len(commits)
|
||||
secrets_found["commits_with_secrets"].extend(commits)
|
||||
print(f" 🔑 Pattern '{secret_pattern}' found in {len(commits)} commits")
|
||||
|
||||
self.log_action("Secrets scan completed", json.dumps(secrets_found, indent=2))
|
||||
return secrets_found
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"❌ Scan failed: {e}")
|
||||
return secrets_found
|
||||
|
||||
def remove_files_from_history(self) -> bool:
|
||||
"""Remove sensitive files from git history using git filter-branch"""
|
||||
print("🧹 Removing sensitive files from git history...")
|
||||
|
||||
try:
|
||||
for file_path in self.FILES_TO_REMOVE:
|
||||
print(f" Removing: {file_path}")
|
||||
|
||||
# Use git filter-branch to remove file from history
|
||||
subprocess.run([
|
||||
'git', 'filter-branch', '--force', '--index-filter',
|
||||
f'git rm --cached --ignore-unmatch {file_path}',
|
||||
'--prune-empty', '--tag-name-filter', 'cat', '--', '--all'
|
||||
], cwd=self.repo_path, check=True)
|
||||
|
||||
self.log_action(f"Removed file from history", file_path)
|
||||
|
||||
return True
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"❌ File removal failed: {e}")
|
||||
return False
|
||||
|
||||
def replace_secrets_in_history(self) -> bool:
|
||||
"""Replace secret patterns in git history"""
|
||||
print("🔄 Replacing secrets in git history...")
|
||||
|
||||
# Create temporary file with replacements
|
||||
replacements_file = self.repo_path / "temp_replacements.txt"
|
||||
|
||||
try:
|
||||
with open(replacements_file, 'w') as f:
|
||||
for secret, replacement in self.SECRETS_TO_REPLACE.items():
|
||||
f.write(f"{secret}==>{replacement}\n")
|
||||
|
||||
# Use git filter-branch with replace text
|
||||
subprocess.run([
|
||||
'git', 'filter-branch', '--force', '--tree-filter',
|
||||
f'find . -type f -exec sed -i.bak -f <(echo "s/{list(self.SECRETS_TO_REPLACE.keys())[0]}/{list(self.SECRETS_TO_REPLACE.values())[0]}/g") {{}} \\; 2>/dev/null || true',
|
||||
'--prune-empty', '--tag-name-filter', 'cat', '--', '--all'
|
||||
], cwd=self.repo_path, check=True)
|
||||
|
||||
self.log_action("Secrets replaced in history")
|
||||
return True
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"❌ Secret replacement failed: {e}")
|
||||
return False
|
||||
finally:
|
||||
# Clean up temporary file
|
||||
if replacements_file.exists():
|
||||
replacements_file.unlink()
|
||||
|
||||
def cleanup_git_refs(self) -> bool:
|
||||
"""Clean up git references and garbage collect"""
|
||||
print("🗑️ Cleaning up git references...")
|
||||
|
||||
try:
|
||||
# Remove backup refs created by filter-branch
|
||||
subprocess.run([
|
||||
'git', 'for-each-ref', '--format=delete %(refname)', 'refs/original'
|
||||
], cwd=self.repo_path, capture_output=True, text=True, check=True)
|
||||
|
||||
# Expire reflog
|
||||
subprocess.run([
|
||||
'git', 'reflog', 'expire', '--expire=now', '--all'
|
||||
], cwd=self.repo_path, check=True)
|
||||
|
||||
# Garbage collect
|
||||
subprocess.run([
|
||||
'git', 'gc', '--prune=now', '--aggressive'
|
||||
], cwd=self.repo_path, check=True)
|
||||
|
||||
self.log_action("Git cleanup completed")
|
||||
return True
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"❌ Git cleanup failed: {e}")
|
||||
return False
|
||||
|
||||
def verify_cleanup(self) -> bool:
|
||||
"""Verify that secrets have been removed from history"""
|
||||
print("🔍 Verifying cleanup...")
|
||||
|
||||
verification_results = {
|
||||
"files_still_present": [],
|
||||
"secrets_still_present": []
|
||||
}
|
||||
|
||||
try:
|
||||
# Check if files are still in history
|
||||
for file_path in self.FILES_TO_REMOVE:
|
||||
result = subprocess.run([
|
||||
'git', 'log', '--oneline', '--', file_path
|
||||
], cwd=self.repo_path, capture_output=True, text=True)
|
||||
|
||||
if result.stdout.strip():
|
||||
verification_results["files_still_present"].append(file_path)
|
||||
|
||||
# Check if secrets are still in history
|
||||
for secret_pattern in self.SECRETS_TO_REPLACE.keys():
|
||||
result = subprocess.run([
|
||||
'git', 'log', '-S', secret_pattern, '--oneline'
|
||||
], cwd=self.repo_path, capture_output=True, text=True)
|
||||
|
||||
if result.stdout.strip():
|
||||
verification_results["secrets_still_present"].append(secret_pattern)
|
||||
|
||||
if not verification_results["files_still_present"] and not verification_results["secrets_still_present"]:
|
||||
print("✅ Cleanup verification passed!")
|
||||
self.log_action("Cleanup verification passed")
|
||||
return True
|
||||
else:
|
||||
print("⚠️ Cleanup verification failed:")
|
||||
if verification_results["files_still_present"]:
|
||||
print(f" Files still present: {verification_results['files_still_present']}")
|
||||
if verification_results["secrets_still_present"]:
|
||||
print(f" Secrets still present: {verification_results['secrets_still_present']}")
|
||||
return False
|
||||
|
||||
except subprocess.CalledProcessError as e:
|
||||
print(f"❌ Verification failed: {e}")
|
||||
return False
|
||||
|
||||
def save_cleanup_log(self) -> None:
|
||||
"""Save cleanup log to file"""
|
||||
log_file = self.repo_path / f"security_cleanup_log_{datetime.now().strftime('%Y%m%d_%H%M%S')}.json"
|
||||
|
||||
with open(log_file, 'w') as f:
|
||||
json.dump({
|
||||
"cleanup_timestamp": datetime.now().isoformat(),
|
||||
"repository_path": str(self.repo_path),
|
||||
"backup_path": str(self.backup_path) if self.backup_path else None,
|
||||
"files_removed": self.FILES_TO_REMOVE,
|
||||
"secrets_replaced": self.SECRETS_TO_REPLACE,
|
||||
"actions": self.cleanup_log
|
||||
}, f, indent=2)
|
||||
|
||||
print(f"📝 Cleanup log saved: {log_file}")
|
||||
|
||||
def run_full_cleanup(self, force: bool = False) -> bool:
|
||||
"""Run complete cleanup process"""
|
||||
print("🚀 Starting ROA2WEB Git History Cleanup")
|
||||
print("="*60)
|
||||
|
||||
if not force:
|
||||
print("\n⚠️ WARNING: This will rewrite git history!")
|
||||
print("Make sure you have:")
|
||||
print("1. ✅ Created a backup")
|
||||
print("2. ✅ Coordinated with team members")
|
||||
print("3. ✅ Are ready to regenerate credentials")
|
||||
|
||||
confirm = input("\nProceed with cleanup? (yes/NO): ")
|
||||
if confirm.lower() != 'yes':
|
||||
print("❌ Cleanup cancelled")
|
||||
return False
|
||||
|
||||
# Check prerequisites
|
||||
if not self.check_prerequisites():
|
||||
return False
|
||||
|
||||
# Create backup
|
||||
if not self.create_backup():
|
||||
return False
|
||||
|
||||
# Scan for secrets
|
||||
secrets_found = self.scan_for_secrets()
|
||||
if not secrets_found["files_with_secrets"] and not secrets_found["patterns_found"]:
|
||||
print("✅ No secrets found in git history")
|
||||
return True
|
||||
|
||||
# Remove files from history
|
||||
if not self.remove_files_from_history():
|
||||
return False
|
||||
|
||||
# Replace secrets in history
|
||||
if not self.replace_secrets_in_history():
|
||||
return False
|
||||
|
||||
# Cleanup git references
|
||||
if not self.cleanup_git_refs():
|
||||
return False
|
||||
|
||||
# Verify cleanup
|
||||
if not self.verify_cleanup():
|
||||
print("⚠️ Cleanup may not be complete. Check manually.")
|
||||
|
||||
# Save log
|
||||
self.save_cleanup_log()
|
||||
|
||||
print("\n✅ Git history cleanup completed!")
|
||||
print("\n🔧 NEXT STEPS:")
|
||||
print("1. 🔑 Regenerate all compromised credentials")
|
||||
print("2. 🚀 Force push to all remotes: git push --force-with-lease --all")
|
||||
print("3. 📢 Notify team members to re-clone repository")
|
||||
print("4. 🗑️ Delete old backup when confident: rm -rf", self.backup_path)
|
||||
|
||||
return True
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="ROA2WEB Git History Cleanup")
|
||||
parser.add_argument('--backup', action='store_true', help='Create backup only')
|
||||
parser.add_argument('--scan', action='store_true', help='Scan for secrets only')
|
||||
parser.add_argument('--cleanup', action='store_true', help='Run full cleanup')
|
||||
parser.add_argument('--force', action='store_true', help='Skip confirmation prompts')
|
||||
parser.add_argument('--repo-path', default='.', help='Repository path')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
cleaner = GitHistoryCleanup(args.repo_path)
|
||||
|
||||
if args.backup:
|
||||
cleaner.create_backup()
|
||||
elif args.scan:
|
||||
cleaner.scan_for_secrets()
|
||||
elif args.cleanup:
|
||||
success = cleaner.run_full_cleanup(args.force)
|
||||
sys.exit(0 if success else 1)
|
||||
else:
|
||||
parser.print_help()
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
60
security/git_hooks/commit-msg
Normal file
60
security/git_hooks/commit-msg
Normal file
@@ -0,0 +1,60 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# 🔒 ROA2WEB Commit Message Hook
|
||||
# Validates commit messages and warns about potential security issues
|
||||
#
|
||||
# Installation:
|
||||
# cp security/git_hooks/commit-msg .git/hooks/commit-msg
|
||||
# chmod +x .git/hooks/commit-msg
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
commit_msg_file="$1"
|
||||
commit_msg=$(cat "$commit_msg_file")
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
YELLOW='\033[1;33m'
|
||||
GREEN='\033[0;32m'
|
||||
NC='\033[0m'
|
||||
|
||||
echo -e "${GREEN}🔒 ROA2WEB Commit Message Check${NC}"
|
||||
|
||||
# Patterns that might indicate accidental secret commits
|
||||
SUSPICIOUS_COMMIT_PATTERNS=(
|
||||
"password"
|
||||
"secret"
|
||||
"credential"
|
||||
"token"
|
||||
"key"
|
||||
"auth"
|
||||
"config"
|
||||
"env"
|
||||
)
|
||||
|
||||
# Check for suspicious patterns in commit message
|
||||
violations=0
|
||||
|
||||
for pattern in "${SUSPICIOUS_COMMIT_PATTERNS[@]}"; do
|
||||
if echo "$commit_msg" | grep -qi "$pattern"; then
|
||||
echo -e "${YELLOW}⚠️ WARNING: Commit message contains potentially sensitive keyword: '$pattern'${NC}"
|
||||
echo -e "${YELLOW} Make sure you're not accidentally committing secrets${NC}"
|
||||
violations=$((violations + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
# Check commit message quality
|
||||
if [[ ${#commit_msg} -lt 10 ]]; then
|
||||
echo -e "${YELLOW}⚠️ WARNING: Very short commit message${NC}"
|
||||
fi
|
||||
|
||||
if [[ $violations -gt 0 ]]; then
|
||||
echo -e "${YELLOW}"
|
||||
echo "⚠️ $violations potential security-related keywords found in commit message"
|
||||
echo "Please double-check that you're not committing sensitive information"
|
||||
echo -e "${NC}"
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}✅ Commit message check completed${NC}"
|
||||
exit 0
|
||||
159
security/git_hooks/pre-commit
Normal file
159
security/git_hooks/pre-commit
Normal file
@@ -0,0 +1,159 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# 🔒 ROA2WEB Pre-commit Hook
|
||||
# Prevents committing files with secrets and credentials
|
||||
#
|
||||
# Installation:
|
||||
# cp security/git_hooks/pre-commit .git/hooks/pre-commit
|
||||
# chmod +x .git/hooks/pre-commit
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
YELLOW='\033[1;33m'
|
||||
GREEN='\033[0;32m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
echo -e "${GREEN}🔒 ROA2WEB Security Pre-commit Check${NC}"
|
||||
|
||||
# Critical patterns to detect
|
||||
CRITICAL_PATTERNS=(
|
||||
"ORACLE_PASSWORD"
|
||||
"ROMFASTSOFT"
|
||||
"Parola81"
|
||||
"VALID_USERS.*password"
|
||||
"-----BEGIN.*PRIVATE KEY-----"
|
||||
"AKIA[0-9A-Z]{16}" # AWS Access Key
|
||||
"Bearer [A-Za-z0-9\-\._~\+\/]+=*" # Bearer tokens
|
||||
)
|
||||
|
||||
# Suspicious file patterns
|
||||
SUSPICIOUS_FILES=(
|
||||
"\.env$"
|
||||
"_rsa$"
|
||||
"\.pem$"
|
||||
"\.key$"
|
||||
"secret"
|
||||
"credential"
|
||||
"password"
|
||||
"config\.prod"
|
||||
)
|
||||
|
||||
# Function to check if file should be scanned
|
||||
should_scan_file() {
|
||||
local file="$1"
|
||||
|
||||
# Skip deleted files
|
||||
if [[ ! -f "$file" ]]; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Skip binary files
|
||||
if file "$file" | grep -q binary; then
|
||||
return 1
|
||||
fi
|
||||
|
||||
# Skip safe extensions
|
||||
case "$file" in
|
||||
*.png|*.jpg|*.jpeg|*.gif|*.pdf|*.zip|*.tar.gz|*.ico) return 1 ;;
|
||||
esac
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# Function to scan file content for secrets
|
||||
scan_file_content() {
|
||||
local file="$1"
|
||||
local violations=0
|
||||
|
||||
for pattern in "${CRITICAL_PATTERNS[@]}"; do
|
||||
if grep -qiE "$pattern" "$file" 2>/dev/null; then
|
||||
echo -e "${RED}❌ CRITICAL: Secret pattern detected in $file${NC}"
|
||||
echo -e "${YELLOW} Pattern: $pattern${NC}"
|
||||
grep -inE "$pattern" "$file" | head -3 | while read line; do
|
||||
echo -e "${YELLOW} $line${NC}"
|
||||
done
|
||||
violations=$((violations + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
return $violations
|
||||
}
|
||||
|
||||
# Function to check suspicious filenames
|
||||
check_suspicious_filename() {
|
||||
local file="$1"
|
||||
|
||||
for pattern in "${SUSPICIOUS_FILES[@]}"; do
|
||||
if echo "$file" | grep -qiE "$pattern"; then
|
||||
# Allow .env.example files
|
||||
if echo "$file" | grep -q "\.example$"; then
|
||||
continue
|
||||
fi
|
||||
|
||||
echo -e "${RED}❌ SUSPICIOUS: Potentially sensitive file: $file${NC}"
|
||||
echo -e "${YELLOW} Pattern: $pattern${NC}"
|
||||
return 1
|
||||
fi
|
||||
done
|
||||
|
||||
return 0
|
||||
}
|
||||
|
||||
# Get list of staged files
|
||||
staged_files=$(git diff --cached --name-only --diff-filter=ACM)
|
||||
|
||||
if [[ -z "$staged_files" ]]; then
|
||||
echo -e "${GREEN}✅ No staged files to check${NC}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "🔍 Scanning staged files for secrets..."
|
||||
|
||||
total_violations=0
|
||||
scanned_files=0
|
||||
|
||||
# Check each staged file
|
||||
while IFS= read -r file; do
|
||||
if should_scan_file "$file"; then
|
||||
scanned_files=$((scanned_files + 1))
|
||||
|
||||
# Check filename
|
||||
if ! check_suspicious_filename "$file"; then
|
||||
total_violations=$((total_violations + 1))
|
||||
fi
|
||||
|
||||
# Check content
|
||||
scan_file_content "$file"
|
||||
violations=$?
|
||||
total_violations=$((total_violations + violations))
|
||||
fi
|
||||
done <<< "$staged_files"
|
||||
|
||||
echo "📊 Scanned $scanned_files files"
|
||||
|
||||
# Check if any violations found
|
||||
if [[ $total_violations -gt 0 ]]; then
|
||||
echo -e "${RED}"
|
||||
echo "=========================================="
|
||||
echo "🚨 COMMIT BLOCKED - SECURITY VIOLATIONS!"
|
||||
echo "=========================================="
|
||||
echo -e "${NC}"
|
||||
echo "Found $total_violations security violations"
|
||||
echo ""
|
||||
echo "🔧 Actions to take:"
|
||||
echo "1. Remove sensitive data from files"
|
||||
echo "2. Move secrets to environment variables"
|
||||
echo "3. Add files to .gitignore if needed"
|
||||
echo "4. Regenerate any exposed credentials"
|
||||
echo ""
|
||||
echo "ℹ️ To bypass this check (NOT RECOMMENDED):"
|
||||
echo " git commit --no-verify"
|
||||
echo ""
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}✅ Security check passed - no violations found${NC}"
|
||||
exit 0
|
||||
58
security/install_hooks.sh
Normal file
58
security/install_hooks.sh
Normal file
@@ -0,0 +1,58 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# 🔒 ROA2WEB Git Hooks Installer
|
||||
# Installs security git hooks to prevent secrets commits
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m'
|
||||
|
||||
echo -e "${GREEN}🔒 Installing ROA2WEB Security Git Hooks${NC}"
|
||||
echo "=========================================="
|
||||
|
||||
# Check if we're in a git repository
|
||||
if [[ ! -d ".git" ]]; then
|
||||
echo -e "${RED}❌ Error: Not in a git repository${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Create hooks directory if it doesn't exist
|
||||
mkdir -p .git/hooks
|
||||
|
||||
# Install pre-commit hook
|
||||
if [[ -f "security/git_hooks/pre-commit" ]]; then
|
||||
cp security/git_hooks/pre-commit .git/hooks/pre-commit
|
||||
chmod +x .git/hooks/pre-commit
|
||||
echo -e "${GREEN}✅ Installed pre-commit hook${NC}"
|
||||
else
|
||||
echo -e "${RED}❌ Error: pre-commit hook file not found${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Install commit-msg hook
|
||||
if [[ -f "security/git_hooks/commit-msg" ]]; then
|
||||
cp security/git_hooks/commit-msg .git/hooks/commit-msg
|
||||
chmod +x .git/hooks/commit-msg
|
||||
echo -e "${GREEN}✅ Installed commit-msg hook${NC}"
|
||||
else
|
||||
echo -e "${RED}❌ Error: commit-msg hook file not found${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo -e "${GREEN}🎉 Git hooks installed successfully!${NC}"
|
||||
echo ""
|
||||
echo "📋 Installed hooks:"
|
||||
echo " • pre-commit - Scans for secrets before commit"
|
||||
echo " • commit-msg - Validates commit messages"
|
||||
echo ""
|
||||
echo "🔧 To bypass hooks (NOT RECOMMENDED):"
|
||||
echo " git commit --no-verify"
|
||||
echo ""
|
||||
echo "🗑️ To uninstall hooks:"
|
||||
echo " rm .git/hooks/pre-commit .git/hooks/commit-msg"
|
||||
204
security/setup_security.sh
Normal file
204
security/setup_security.sh
Normal file
@@ -0,0 +1,204 @@
|
||||
#!/bin/bash
|
||||
#
|
||||
# 🔒 ROA2WEB Security Setup Script
|
||||
# Complete security implementation for the ROA2WEB project
|
||||
#
|
||||
|
||||
set -e
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
BLUE='\033[0;34m'
|
||||
NC='\033[0m'
|
||||
|
||||
echo -e "${GREEN}"
|
||||
echo "=============================================="
|
||||
echo "🔒 ROA2WEB SECURITY IMPLEMENTATION SETUP"
|
||||
echo "=============================================="
|
||||
echo -e "${NC}"
|
||||
|
||||
# Function to print step headers
|
||||
print_step() {
|
||||
echo -e "${BLUE}📋 Step $1: $2${NC}"
|
||||
echo "----------------------------------------"
|
||||
}
|
||||
|
||||
# Function to check if command exists
|
||||
command_exists() {
|
||||
command -v "$1" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
# Check prerequisites
|
||||
print_step "1" "Checking Prerequisites"
|
||||
|
||||
if ! command_exists python3; then
|
||||
echo -e "${RED}❌ Python 3 is required but not installed${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! command_exists git; then
|
||||
echo -e "${RED}❌ Git is required but not installed${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ ! -d ".git" ]]; then
|
||||
echo -e "${RED}❌ Not in a git repository${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}✅ Prerequisites check passed${NC}"
|
||||
echo
|
||||
|
||||
# Install git hooks
|
||||
print_step "2" "Installing Git Security Hooks"
|
||||
|
||||
if [[ -f "security/install_hooks.sh" ]]; then
|
||||
chmod +x security/install_hooks.sh
|
||||
./security/install_hooks.sh
|
||||
else
|
||||
echo -e "${RED}❌ Hook installer not found${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo
|
||||
|
||||
# Make scripts executable
|
||||
print_step "3" "Setting Script Permissions"
|
||||
|
||||
chmod +x security/secrets_scanner.py
|
||||
chmod +x security/git_cleanup.py
|
||||
|
||||
echo -e "${GREEN}✅ Script permissions set${NC}"
|
||||
echo
|
||||
|
||||
# Run initial security scan
|
||||
print_step "4" "Running Initial Security Scan"
|
||||
|
||||
echo -e "${YELLOW}🔍 Scanning repository for secrets...${NC}"
|
||||
python3 security/secrets_scanner.py --save-report initial_security_scan.json
|
||||
|
||||
echo
|
||||
|
||||
# Check git history for secrets
|
||||
print_step "5" "Checking Git History"
|
||||
|
||||
echo -e "${YELLOW}🕐 Scanning git history (this may take a moment)...${NC}"
|
||||
python3 security/secrets_scanner.py --scan-git-history --save-report git_history_scan.json
|
||||
|
||||
echo
|
||||
|
||||
# Verify .gitignore protection
|
||||
print_step "6" "Verifying .gitignore Protection"
|
||||
|
||||
echo "🔍 Checking .gitignore coverage..."
|
||||
|
||||
# Check if critical patterns are in .gitignore
|
||||
critical_patterns=(
|
||||
"*.env"
|
||||
"*.key"
|
||||
"*.pem"
|
||||
"*secret*"
|
||||
"*credential*"
|
||||
"*password*"
|
||||
)
|
||||
|
||||
gitignore_issues=0
|
||||
for pattern in "${critical_patterns[@]}"; do
|
||||
if ! grep -q "$pattern" .gitignore; then
|
||||
echo -e "${YELLOW}⚠️ Pattern '$pattern' not found in .gitignore${NC}"
|
||||
gitignore_issues=$((gitignore_issues + 1))
|
||||
fi
|
||||
done
|
||||
|
||||
if [[ $gitignore_issues -eq 0 ]]; then
|
||||
echo -e "${GREEN}✅ .gitignore security patterns verified${NC}"
|
||||
else
|
||||
echo -e "${YELLOW}⚠️ $gitignore_issues security patterns missing from .gitignore${NC}"
|
||||
fi
|
||||
|
||||
echo
|
||||
|
||||
# Create security monitoring cron job (optional)
|
||||
print_step "7" "Setting Up Security Monitoring (Optional)"
|
||||
|
||||
echo "📅 Would you like to set up automated daily security scans?"
|
||||
echo "This will add a cron job to run security scans daily at 9 AM"
|
||||
read -p "Setup automated scans? (y/N): " setup_cron
|
||||
|
||||
if [[ "$setup_cron" =~ ^[Yy]$ ]]; then
|
||||
# Get current directory
|
||||
current_dir=$(pwd)
|
||||
|
||||
# Create cron job entry
|
||||
cron_entry="0 9 * * * cd $current_dir && python3 security/secrets_scanner.py --save-report daily_scan_\$(date +\\%Y\\%m\\%d).json >/dev/null 2>&1"
|
||||
|
||||
# Add to crontab
|
||||
(crontab -l 2>/dev/null; echo "$cron_entry") | crontab -
|
||||
|
||||
echo -e "${GREEN}✅ Daily security scan cron job added${NC}"
|
||||
else
|
||||
echo "📝 Skipped automated scan setup"
|
||||
fi
|
||||
|
||||
echo
|
||||
|
||||
# Security setup summary
|
||||
print_step "8" "Security Setup Summary"
|
||||
|
||||
echo -e "${GREEN}🎉 ROA2WEB Security Implementation Complete!${NC}"
|
||||
echo
|
||||
echo "📋 What was installed:"
|
||||
echo " ✅ Git hooks (pre-commit, commit-msg)"
|
||||
echo " ✅ Secrets scanner tool"
|
||||
echo " ✅ Git history cleanup tool"
|
||||
echo " ✅ Enhanced .gitignore patterns"
|
||||
echo " ✅ Security documentation"
|
||||
echo
|
||||
echo "📊 Security scan results:"
|
||||
echo " 📄 Initial scan: initial_security_scan.json"
|
||||
echo " 📄 History scan: git_history_scan.json"
|
||||
echo
|
||||
echo "🔧 Available tools:"
|
||||
echo " 🔍 Security scan: python3 security/secrets_scanner.py"
|
||||
echo " 🧹 Git cleanup: python3 security/git_cleanup.py"
|
||||
echo " 📋 Documentation: security/README.md"
|
||||
echo
|
||||
|
||||
# Critical warnings
|
||||
if [[ -f "initial_security_scan.json" ]]; then
|
||||
critical_violations=$(python3 -c "
|
||||
import json
|
||||
try:
|
||||
with open('initial_security_scan.json', 'r') as f:
|
||||
data = json.load(f)
|
||||
print(data.get('summary', {}).get('critical_violations', 0))
|
||||
except:
|
||||
print(0)
|
||||
" 2>/dev/null || echo "0")
|
||||
|
||||
if [[ "$critical_violations" -gt 0 ]]; then
|
||||
echo -e "${RED}"
|
||||
echo "🚨 CRITICAL SECURITY ALERT!"
|
||||
echo "=============================="
|
||||
echo -e "${NC}"
|
||||
echo -e "${RED}Found $critical_violations critical security violations!${NC}"
|
||||
echo
|
||||
echo "🔧 IMMEDIATE ACTIONS REQUIRED:"
|
||||
echo "1. 🔑 Regenerate all exposed credentials"
|
||||
echo "2. 🧹 Clean git history: python3 security/git_cleanup.py --cleanup"
|
||||
echo "3. 🚀 Force push cleaned history to all remotes"
|
||||
echo "4. 📢 Notify team to re-clone repository"
|
||||
echo
|
||||
echo "📖 See security/README.md for detailed procedures"
|
||||
echo
|
||||
fi
|
||||
fi
|
||||
|
||||
echo -e "${BLUE}📚 Next Steps:${NC}"
|
||||
echo "1. Review security scan reports"
|
||||
echo "2. Read security/README.md for detailed guidance"
|
||||
echo "3. Follow security/SECURITY_PROCEDURES.md for ongoing security"
|
||||
echo "4. Train team members on new security procedures"
|
||||
echo
|
||||
echo -e "${GREEN}🔒 ROA2WEB is now security-enhanced!${NC}"
|
||||
283
security/validate_security.py
Normal file
283
security/validate_security.py
Normal file
@@ -0,0 +1,283 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
🔍 ROA2WEB Security Validation Tool
|
||||
Validates that all security measures are properly implemented and functioning.
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
class SecurityValidator:
|
||||
"""Validates security implementation"""
|
||||
|
||||
def __init__(self, repo_path: str = "."):
|
||||
self.repo_path = Path(repo_path)
|
||||
self.errors = []
|
||||
self.warnings = []
|
||||
self.checks_passed = 0
|
||||
self.total_checks = 0
|
||||
|
||||
def check(self, condition: bool, success_msg: str, error_msg: str, is_warning: bool = False):
|
||||
"""Check a condition and track results"""
|
||||
self.total_checks += 1
|
||||
if condition:
|
||||
print(f"✅ {success_msg}")
|
||||
self.checks_passed += 1
|
||||
else:
|
||||
if is_warning:
|
||||
print(f"⚠️ {error_msg}")
|
||||
self.warnings.append(error_msg)
|
||||
else:
|
||||
print(f"❌ {error_msg}")
|
||||
self.errors.append(error_msg)
|
||||
|
||||
def validate_files_exist(self):
|
||||
"""Validate that all security files exist"""
|
||||
print("\n🔍 Checking Security Files...")
|
||||
|
||||
required_files = [
|
||||
"security/secrets_scanner.py",
|
||||
"security/git_cleanup.py",
|
||||
"security/install_hooks.sh",
|
||||
"security/setup_security.sh",
|
||||
"security/git_hooks/pre-commit",
|
||||
"security/git_hooks/commit-msg",
|
||||
"security/README.md",
|
||||
"security/SECURITY_PROCEDURES.md"
|
||||
]
|
||||
|
||||
for file_path in required_files:
|
||||
full_path = self.repo_path / file_path
|
||||
self.check(
|
||||
full_path.exists(),
|
||||
f"Security file exists: {file_path}",
|
||||
f"Missing security file: {file_path}"
|
||||
)
|
||||
|
||||
def validate_gitignore(self):
|
||||
"""Validate .gitignore security patterns"""
|
||||
print("\n🛡️ Checking .gitignore Security...")
|
||||
|
||||
gitignore_path = self.repo_path / ".gitignore"
|
||||
|
||||
if not gitignore_path.exists():
|
||||
self.check(False, "", ".gitignore file missing")
|
||||
return
|
||||
|
||||
with open(gitignore_path, 'r') as f:
|
||||
gitignore_content = f.read()
|
||||
|
||||
critical_patterns = [
|
||||
"*.env.*",
|
||||
"!.env.example",
|
||||
"*.pem",
|
||||
"*.key",
|
||||
"*secret*",
|
||||
"*credential*",
|
||||
"*password*",
|
||||
".serena/cache/",
|
||||
".serena/memories/"
|
||||
]
|
||||
|
||||
for pattern in critical_patterns:
|
||||
self.check(
|
||||
pattern in gitignore_content,
|
||||
f"Security pattern in .gitignore: {pattern}",
|
||||
f"Missing .gitignore pattern: {pattern}",
|
||||
is_warning=True
|
||||
)
|
||||
|
||||
def validate_git_hooks(self):
|
||||
"""Validate git hooks installation"""
|
||||
print("\n🪝 Checking Git Hooks...")
|
||||
|
||||
hooks_dir = self.repo_path / ".git" / "hooks"
|
||||
|
||||
self.check(
|
||||
hooks_dir.exists(),
|
||||
"Git hooks directory exists",
|
||||
"Git hooks directory missing"
|
||||
)
|
||||
|
||||
required_hooks = ["pre-commit", "commit-msg"]
|
||||
|
||||
for hook in required_hooks:
|
||||
hook_path = hooks_dir / hook
|
||||
self.check(
|
||||
hook_path.exists(),
|
||||
f"Git hook installed: {hook}",
|
||||
f"Git hook missing: {hook}"
|
||||
)
|
||||
|
||||
if hook_path.exists():
|
||||
self.check(
|
||||
os.access(hook_path, os.X_OK),
|
||||
f"Git hook executable: {hook}",
|
||||
f"Git hook not executable: {hook}"
|
||||
)
|
||||
|
||||
def validate_scripts_executable(self):
|
||||
"""Validate that scripts are executable"""
|
||||
print("\n🔧 Checking Script Permissions...")
|
||||
|
||||
executable_scripts = [
|
||||
"security/secrets_scanner.py",
|
||||
"security/git_cleanup.py",
|
||||
"security/install_hooks.sh",
|
||||
"security/setup_security.sh"
|
||||
]
|
||||
|
||||
for script in executable_scripts:
|
||||
script_path = self.repo_path / script
|
||||
if script_path.exists():
|
||||
self.check(
|
||||
os.access(script_path, os.X_OK),
|
||||
f"Script executable: {script}",
|
||||
f"Script not executable: {script}"
|
||||
)
|
||||
|
||||
def validate_scanner_functionality(self):
|
||||
"""Validate scanner can run"""
|
||||
print("\n🔍 Testing Security Scanner...")
|
||||
|
||||
try:
|
||||
# Import and test scanner
|
||||
sys.path.append(str(self.repo_path))
|
||||
import security.secrets_scanner as scanner
|
||||
|
||||
s = scanner.SecretsScanner(self.repo_path)
|
||||
self.check(
|
||||
len(s.CRITICAL_PATTERNS) > 0,
|
||||
f"Scanner patterns loaded: {len(s.CRITICAL_PATTERNS)} patterns",
|
||||
"Scanner patterns not loaded"
|
||||
)
|
||||
|
||||
self.check(
|
||||
len(s.SUSPICIOUS_FILES) > 0,
|
||||
f"Scanner file patterns loaded: {len(s.SUSPICIOUS_FILES)} patterns",
|
||||
"Scanner file patterns not loaded"
|
||||
)
|
||||
|
||||
except Exception as e:
|
||||
self.check(
|
||||
False,
|
||||
"Scanner functionality test passed",
|
||||
f"Scanner functionality test failed: {e}"
|
||||
)
|
||||
|
||||
def validate_git_repository(self):
|
||||
"""Validate git repository status"""
|
||||
print("\n📦 Checking Git Repository...")
|
||||
|
||||
try:
|
||||
# Check if in git repo
|
||||
result = subprocess.run(
|
||||
['git', 'status'],
|
||||
cwd=self.repo_path,
|
||||
capture_output=True,
|
||||
check=True
|
||||
)
|
||||
|
||||
self.check(
|
||||
True,
|
||||
"Git repository status OK",
|
||||
"Git repository issues detected"
|
||||
)
|
||||
|
||||
except subprocess.CalledProcessError:
|
||||
self.check(
|
||||
False,
|
||||
"Git repository status OK",
|
||||
"Not in a valid git repository"
|
||||
)
|
||||
|
||||
def check_environment_files(self):
|
||||
"""Check for environment files that should be protected"""
|
||||
print("\n🔐 Checking Environment Files...")
|
||||
|
||||
# Find .env files
|
||||
env_files = []
|
||||
for root, dirs, files in os.walk(self.repo_path):
|
||||
for file in files:
|
||||
if file.endswith('.env') and not file.endswith('.env.example'):
|
||||
rel_path = os.path.relpath(os.path.join(root, file), self.repo_path)
|
||||
env_files.append(rel_path)
|
||||
|
||||
for env_file in env_files:
|
||||
# Check if file is in git tracking
|
||||
try:
|
||||
result = subprocess.run(
|
||||
['git', 'ls-files', env_file],
|
||||
cwd=self.repo_path,
|
||||
capture_output=True,
|
||||
text=True
|
||||
)
|
||||
|
||||
if result.stdout.strip():
|
||||
self.check(
|
||||
False,
|
||||
"",
|
||||
f"Environment file tracked in git: {env_file}",
|
||||
is_warning=True
|
||||
)
|
||||
else:
|
||||
self.check(
|
||||
True,
|
||||
f"Environment file properly ignored: {env_file}",
|
||||
""
|
||||
)
|
||||
|
||||
except subprocess.CalledProcessError:
|
||||
pass
|
||||
|
||||
def run_validation(self):
|
||||
"""Run complete security validation"""
|
||||
print("🔒 ROA2WEB Security Validation")
|
||||
print("=" * 50)
|
||||
|
||||
self.validate_files_exist()
|
||||
self.validate_gitignore()
|
||||
self.validate_git_hooks()
|
||||
self.validate_scripts_executable()
|
||||
self.validate_scanner_functionality()
|
||||
self.validate_git_repository()
|
||||
self.check_environment_files()
|
||||
|
||||
# Print summary
|
||||
print("\n" + "=" * 50)
|
||||
print("📊 VALIDATION SUMMARY")
|
||||
print("=" * 50)
|
||||
|
||||
print(f"✅ Checks passed: {self.checks_passed}/{self.total_checks}")
|
||||
|
||||
if self.errors:
|
||||
print(f"❌ Errors: {len(self.errors)}")
|
||||
for error in self.errors:
|
||||
print(f" • {error}")
|
||||
|
||||
if self.warnings:
|
||||
print(f"⚠️ Warnings: {len(self.warnings)}")
|
||||
for warning in self.warnings:
|
||||
print(f" • {warning}")
|
||||
|
||||
# Overall status
|
||||
if not self.errors:
|
||||
if not self.warnings:
|
||||
print("\n🎉 VALIDATION PASSED - Security implementation complete!")
|
||||
return True
|
||||
else:
|
||||
print("\n✅ VALIDATION PASSED - Minor warnings noted")
|
||||
return True
|
||||
else:
|
||||
print("\n❌ VALIDATION FAILED - Critical issues found")
|
||||
return False
|
||||
|
||||
def main():
|
||||
validator = SecurityValidator()
|
||||
success = validator.run_validation()
|
||||
sys.exit(0 if success else 1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user