desk-moloni/PHP_MIGRATION_ROLLBACK_STRATEGY.md

# 🔙 PHP MIGRATION ROLLBACK STRATEGY
## desk-moloni: Emergency Recovery and Rollback Plan

**Document Owner:** System Development Agent  
**Critical Priority:** IMMEDIATE ACCESS during deployment  
**Last Updated:** 2025-09-12  
**Emergency Contacts:** See escalation matrix  

---

## 🚨 EMERGENCY ROLLBACK OVERVIEW

### When to Execute Rollback
**IMMEDIATE ROLLBACK TRIGGERS:**
- 🔴 **Critical System Failure** - Core functionality broken
- 🔴 **Security Breach** - New vulnerabilities detected
- 🔴 **Data Corruption** - Database integrity compromised
- 🔴 **Performance Degradation** - >50% performance loss
- 🔴 **API Integration Failure** - External services not accessible
- 🔴 **User-Blocking Issues** - System unusable for end users

### Rollback Decision Matrix

| Issue Severity | Response Time | Approval Required | Rollback Type |
|---------------|---------------|-------------------|---------------|
| **CRITICAL** | <15 minutes | CTO Only | Full Rollback |
| **HIGH** | <30 minutes | Technical Director | Partial Rollback |
| **MEDIUM** | <60 minutes | Development Lead | Code Rollback Only |
| **LOW** | <4 hours | Team Consensus | Fix Forward |

---

## ⚡ IMMEDIATE ROLLBACK (< 15 minutes)

### Emergency Rollback Command Sequence
```bash
#!/bin/bash
# EMERGENCY_ROLLBACK.sh - Execute immediately on critical failure

echo "🚨 EMERGENCY ROLLBACK INITIATED $(date)"
echo "User: $(whoami) | Host: $(hostname)"

# 1. IMMEDIATE GIT ROLLBACK (30 seconds)
echo "Step 1: Git rollback..."
git checkout main
git reset --hard HEAD~1  # Or specific commit hash
git push origin main --force

# 2. DATABASE ROLLBACK (2 minutes)
echo "Step 2: Database rollback..."
mysql -u root -p desk_moloni < /backups/pre_migration_backup_$(date +%Y%m%d).sql

# 3. SERVICE RESTART (1 minute)
echo "Step 3: Service restart..."
systemctl restart apache2
systemctl restart php8.0-fpm  # Rollback to PHP 8.0
systemctl restart mysql

# 4. IMMEDIATE VERIFICATION (30 seconds)
echo "Step 4: Quick verification..."
curl -f http://localhost/desk_moloni/health_check.php || echo "❌ HEALTH CHECK FAILED"

echo "✅ EMERGENCY ROLLBACK COMPLETED $(date)"
echo "🔍 CHECK LOGS: tail -f /var/log/apache2/error.log"
```

### Critical File Rollback Locations
```bash
# Keep these paths updated with actual backup locations
BACKUP_ROOT="/backups/php_migration_$(date +%Y%m%d)"
CODE_BACKUP="$BACKUP_ROOT/codebase_backup.tar.gz"
DB_BACKUP="$BACKUP_ROOT/database_backup.sql"  
CONFIG_BACKUP="$BACKUP_ROOT/config_backup.tar.gz"
```

---

## 🗂️ ROLLBACK PREPARATION CHECKLIST

### Pre-Migration Backup Creation (Day 17 - Before Production)

#### 1. Complete Codebase Backup
```bash
#!/bin/bash
# create_codebase_backup.sh

BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/php_migration_$BACKUP_DATE"
mkdir -p $BACKUP_DIR

echo "Creating complete codebase backup..."

# Full project backup
tar -czf "$BACKUP_DIR/codebase_backup.tar.gz" \
    --exclude='.git' \
    --exclude='vendor' \
    --exclude='node_modules' \
    --exclude='tests/.phpunit.cache' \
    /path/to/desk-moloni/

# Configuration files
tar -czf "$BACKUP_DIR/config_backup.tar.gz" \
    /etc/apache2/sites-available/desk-moloni.conf \
    /etc/php/*/apache2/php.ini \
    /etc/mysql/mysql.conf.d/mysqld.cnf

# Document backup details
echo "Backup created: $BACKUP_DATE" > "$BACKUP_DIR/backup_info.txt"
echo "PHP Version: $(php -v | head -n 1)" >> "$BACKUP_DIR/backup_info.txt"
echo "Commit Hash: $(git rev-parse HEAD)" >> "$BACKUP_DIR/backup_info.txt"

echo "✅ Codebase backup completed: $BACKUP_DIR"
```

#### 2. Database Backup with Verification
```bash
#!/bin/bash  
# create_database_backup.sh

BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/php_migration_$BACKUP_DATE"
DB_NAME="desk_moloni"

echo "Creating database backup with verification..."

# Full database backup
mysqldump -u root -p \
    --single-transaction \
    --lock-tables=false \
    --routines \
    --triggers \
    --events \
    $DB_NAME > "$BACKUP_DIR/database_backup.sql"

# Verify backup integrity
echo "Verifying backup integrity..."
mysql -u root -p -e "CREATE DATABASE test_restore_$$"
mysql -u root -p test_restore_$$ < "$BACKUP_DIR/database_backup.sql"

if [ $? -eq 0 ]; then
    echo "✅ Database backup verified successfully"
    mysql -u root -p -e "DROP DATABASE test_restore_$$"
else
    echo "❌ Database backup verification FAILED"
    exit 1
fi

# Create backup metadata
echo "Database: $DB_NAME" > "$BACKUP_DIR/database_info.txt"
echo "Backup Date: $BACKUP_DATE" >> "$BACKUP_DIR/database_info.txt"
echo "Table Count: $(mysql -u root -p $DB_NAME -e 'SHOW TABLES' | wc -l)" >> "$BACKUP_DIR/database_info.txt"
echo "Data Size: $(du -sh $BACKUP_DIR/database_backup.sql)" >> "$BACKUP_DIR/database_info.txt"

echo "✅ Database backup completed and verified"
```

#### 3. System Configuration Backup
```bash
#!/bin/bash
# create_system_backup.sh

BACKUP_DATE=$(date +%Y%m%d_%H%M%S)
BACKUP_DIR="/backups/php_migration_$BACKUP_DATE"

echo "Creating system configuration backup..."

# Apache configuration
mkdir -p "$BACKUP_DIR/apache"
cp -r /etc/apache2/sites-available/ "$BACKUP_DIR/apache/"
cp -r /etc/apache2/sites-enabled/ "$BACKUP_DIR/apache/"

# PHP configuration (for all versions)
mkdir -p "$BACKUP_DIR/php"
cp -r /etc/php/ "$BACKUP_DIR/php/"

# MySQL configuration
mkdir -p "$BACKUP_DIR/mysql"
cp /etc/mysql/mysql.conf.d/mysqld.cnf "$BACKUP_DIR/mysql/"

# Cron jobs
crontab -l > "$BACKUP_DIR/crontab_backup.txt" 2>/dev/null || echo "No crontab found"

# System info snapshot
echo "System backup created: $BACKUP_DATE" > "$BACKUP_DIR/system_info.txt"
echo "PHP Version: $(php -v | head -n 1)" >> "$BACKUP_DIR/system_info.txt"
echo "Apache Version: $(apache2 -v | head -n 1)" >> "$BACKUP_DIR/system_info.txt"
echo "MySQL Version: $(mysql --version)" >> "$BACKUP_DIR/system_info.txt"
echo "OS Info: $(lsb_release -a)" >> "$BACKUP_DIR/system_info.txt"

echo "✅ System configuration backup completed"
```

---

## 🔄 ROLLBACK EXECUTION PROCEDURES

### Level 1: Code-Only Rollback (5-10 minutes)
**Use When:** Code issues, no database changes affected

```bash
#!/bin/bash
# code_rollback.sh

echo "🔄 Executing Code-Only Rollback..."

# 1. Git rollback to last stable commit
git checkout main
git log --oneline -5  # Show recent commits
read -p "Enter commit hash to rollback to: " COMMIT_HASH

git reset --hard $COMMIT_HASH
git clean -fd  # Remove untracked files

# 2. Restore vendor dependencies
composer install --no-dev --optimize-autoloader

# 3. Clear caches
rm -rf tests/.phpunit.cache
php -r "opcache_reset();" 2>/dev/null || true

# 4. Restart services
systemctl reload apache2

echo "✅ Code rollback completed"
```

### Level 2: Partial System Rollback (15-20 minutes)  
**Use When:** Configuration changes need reverting

```bash
#!/bin/bash
# partial_rollback.sh

BACKUP_DIR="/backups/php_migration_$(date +%Y%m%d)"

echo "🔄 Executing Partial System Rollback..."

# 1. Code rollback (from Level 1)
./code_rollback.sh

# 2. Restore PHP configuration
systemctl stop apache2
tar -xzf "$BACKUP_DIR/config_backup.tar.gz" -C /

# 3. Switch PHP version back (if needed)
a2dismod php8.4
a2enmod php8.0
systemctl start apache2

# 4. Verify services
systemctl status apache2
systemctl status mysql

echo "✅ Partial rollback completed"
```

### Level 3: Full System Rollback (30-45 minutes)
**Use When:** Database changes need reverting

```bash
#!/bin/bash
# full_rollback.sh

BACKUP_DIR="/backups/php_migration_$(date +%Y%m%d)"

echo "🔄 Executing Full System Rollback..."

# 1. Stop all services
systemctl stop apache2
systemctl stop mysql

# 2. Database rollback
systemctl start mysql
mysql -u root -p -e "DROP DATABASE IF EXISTS desk_moloni_rollback_temp"
mysql -u root -p -e "CREATE DATABASE desk_moloni_rollback_temp"
mysql -u root -p desk_moloni_rollback_temp < "$BACKUP_DIR/database_backup.sql"

# Verify database restore
if mysql -u root -p desk_moloni_rollback_temp -e "SELECT COUNT(*) FROM tbldeskmoloni_config" >/dev/null 2>&1; then
    mysql -u root -p -e "DROP DATABASE desk_moloni"
    mysql -u root -p -e "RENAME DATABASE desk_moloni_rollback_temp TO desk_moloni" 2>/dev/null || {
        mysql -u root -p -e "CREATE DATABASE desk_moloni"
        mysqldump -u root -p desk_moloni_rollback_temp | mysql -u root -p desk_moloni
        mysql -u root -p -e "DROP DATABASE desk_moloni_rollback_temp"
    }
    echo "✅ Database rollback successful"
else
    echo "❌ Database rollback verification failed"
    exit 1
fi

# 3. Code and configuration rollback
./partial_rollback.sh

# 4. Full service restart
systemctl start mysql
systemctl start apache2

echo "✅ Full system rollback completed"
```

---

## 📊 ROLLBACK VERIFICATION PROCEDURES

### Immediate Health Checks (< 2 minutes)
```bash
#!/bin/bash
# rollback_verification.sh

echo "🔍 Running rollback verification checks..."

# 1. Basic connectivity
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost/desk_moloni/)
if [ "$HTTP_STATUS" = "200" ]; then
    echo "✅ HTTP connectivity working"
else
    echo "❌ HTTP connectivity failed: $HTTP_STATUS"
fi

# 2. Database connectivity  
mysql -u root -p desk_moloni -e "SELECT 1" >/dev/null 2>&1
if [ $? -eq 0 ]; then
    echo "✅ Database connectivity working"
else
    echo "❌ Database connectivity failed"
fi

# 3. PHP functionality
php -r "echo 'PHP working: ' . PHP_VERSION . PHP_EOL;"

# 4. Critical tables exist
TABLES=(
    "tbldeskmoloni_config"
    "tbldeskmoloni_sync_log"
    "tbldeskmoloni_mapping"
    "tbldeskmoloni_sync_queue"
)

for table in "${TABLES[@]}"; do
    if mysql -u root -p desk_moloni -e "SELECT 1 FROM $table LIMIT 1" >/dev/null 2>&1; then
        echo "✅ Table $table exists and accessible"
    else
        echo "❌ Table $table missing or inaccessible"
    fi
done

echo "🔍 Verification completed"
```

### Extended Functionality Testing (10-15 minutes)
```bash
#!/bin/bash
# extended_verification.sh

echo "🔍 Running extended functionality verification..."

# 1. Admin dashboard access
curl -s "http://localhost/desk_moloni/admin" | grep -q "Dashboard" && echo "✅ Admin dashboard accessible" || echo "❌ Admin dashboard failed"

# 2. API endpoints
curl -s "http://localhost/desk_moloni/api/health" | grep -q "ok" && echo "✅ API endpoints responding" || echo "❌ API endpoints failed"

# 3. Database operations
php -r "
    \$ci = &get_instance();
    \$ci->load->model('desk_moloni/desk_moloni_config_model');
    \$config = \$ci->desk_moloni_config_model->get('api_base_url');
    echo \$config ? '✅ Database operations working' : '❌ Database operations failed';
    echo PHP_EOL;
"

# 4. Queue processing
php modules/desk_moloni/tests/contract/test_queue_processing.php --verify-only

echo "🔍 Extended verification completed"
```

---

## 📞 COMMUNICATION PLAN DURING ROLLBACK

### Immediate Notifications (< 5 minutes)
```bash
# Stakeholder notification script
echo "🚨 ROLLBACK INITIATED: $(date)" | mail -s "CRITICAL: PHP Migration Rollback" stakeholders@company.com
```

### Communication Template
```
SUBJECT: [CRITICAL] PHP Migration Rollback in Progress - desk-moloni

TIMELINE: $(date)
STATUS: Rollback in progress
ESTIMATED RESTORATION: [15/30/45] minutes
CAUSE: [Brief description]

ACTIONS TAKEN:
- System backed up
- Rollback initiated
- Services being restored

NEXT UPDATE: In [15] minutes

Technical Team
```

### Post-Rollback Communication
```
SUBJECT: [RESOLVED] PHP Migration Rollback Completed - desk-moloni  

STATUS: System restored to previous stable state
ROLLBACK COMPLETED: $(date)
SERVICES: All operational
IMPACT: [Duration] of service disruption

ROOT CAUSE: [Detailed explanation]
LESSONS LEARNED: [Key findings]
NEXT STEPS: [Future plan]

Technical Team
```

---

## 🔍 POST-ROLLBACK ANALYSIS

### Immediate Investigation Checklist
- [ ] **Identify Root Cause** - What triggered the rollback?
- [ ] **Log Analysis** - Review all error logs during migration
- [ ] **Performance Metrics** - Compare before/during/after metrics
- [ ] **Data Integrity Check** - Verify no data corruption occurred
- [ ] **Security Assessment** - Ensure no security vulnerabilities introduced
- [ ] **Stakeholder Impact** - Document business impact and user effect

### Rollback Report Template
```markdown
# PHP Migration Rollback Report

## Incident Summary
- **Date/Time:** [DateTime]
- **Duration:** [Duration]
- **Rollback Type:** [Code/Partial/Full]
- **Root Cause:** [Cause]

## Timeline
- Migration Start: [Time]
- Issue Detected: [Time]  
- Rollback Initiated: [Time]
- System Restored: [Time]

## Impact Assessment
- **Users Affected:** [Number]
- **Services Impacted:** [List]
- **Data Loss:** [None/Description]
- **Business Impact:** [Description]

## Technical Analysis
- **Failed Components:** [List]
- **Error Messages:** [Key errors]
- **Performance Impact:** [Metrics]

## Actions Taken
- [List all actions during rollback]

## Lessons Learned
- [Key findings]
- [Improvement opportunities]

## Recommendations
- [Future prevention measures]
- [Process improvements]

## Approval
Technical Director: ________________
Date: ________________
```

---

## 🛡️ ROLLBACK PREVENTION STRATEGIES

### Pre-Deployment Prevention
1. **Comprehensive Testing** - Ensure all tests pass
2. **Staging Validation** - Complete staging environment testing
3. **Gradual Rollout** - Consider blue-green deployment
4. **Monitoring Setup** - Comprehensive monitoring before deployment
5. **Team Readiness** - Ensure all team members are available

### Early Detection Systems
```bash
# monitoring_setup.sh - Deploy before migration
echo "Setting up migration monitoring..."

# 1. Performance monitoring
echo "* * * * * curl -s http://localhost/desk_moloni/health_check.php || echo 'ALERT: Health check failed' | mail -s 'Migration Alert' admin@company.com" | crontab

# 2. Error log monitoring  
tail -f /var/log/apache2/error.log | grep -E "(FATAL|ERROR)" --line-buffered | while read line; do
    echo "MIGRATION ERROR: $line" | mail -s "Migration Error Alert" admin@company.com
done &

# 3. Database monitoring
mysql -u root -p desk_moloni -e "SELECT COUNT(*) FROM tbldeskmoloni_config" || echo "DB Alert" | mail admin@company.com
```

---

## 🔧 ROLLBACK TESTING PLAN

### Pre-Migration Rollback Testing (Day 16)
```bash
#!/bin/bash
# test_rollback_procedures.sh

echo "🧪 Testing rollback procedures..."

# 1. Create test backup
./create_codebase_backup.sh
./create_database_backup.sh

# 2. Make small test change
echo "<!-- Test change $(date) -->" >> index.html

# 3. Test code rollback
./code_rollback.sh

# 4. Verify rollback worked
if ! grep -q "Test change" index.html; then
    echo "✅ Code rollback test passed"
else
    echo "❌ Code rollback test failed"
fi

# 5. Test database rollback with temporary database
mysql -u root -p -e "CREATE DATABASE rollback_test"
mysql -u root -p rollback_test < /backups/php_migration_*/database_backup.sql
mysql -u root -p -e "DROP DATABASE rollback_test"

echo "✅ Rollback procedures tested successfully"
```

---

## 📋 ROLLBACK DECISION FLOWCHART

```
Migration Issue Detected
           |
           ▼
    Is it Critical?
      /         \
   YES           NO
    |             |
    ▼             ▼
Immediate      Attempt Fix
Rollback        (30 min)
    |             |
    ▼             ▼
Execute        Fix Success?
Level 3          /      \
Rollback      YES      NO
    |          |        |
    ▼          ▼        ▼
Notify     Continue   Rollback
Team       Monitoring  Level 1/2
    |          |        |
    ▼          ▼        ▼
Post-Rollback Success  Verify
Analysis   Report    Success
```

---

## 📚 ROLLBACK KNOWLEDGE BASE

### Common Issues and Solutions

#### Issue: "Database connection failed after rollback"
**Solution:**
```bash
# Check MySQL service
systemctl status mysql
systemctl restart mysql

# Verify database exists
mysql -u root -p -e "SHOW DATABASES;" | grep desk_moloni

# Test connection with application user
mysql -u desk_moloni_user -p desk_moloni -e "SELECT 1"
```

#### Issue: "PHP version mismatch after rollback"
**Solution:**
```bash
# Check current PHP version
php -v

# Switch PHP version
a2dismod php8.4
a2enmod php8.0
systemctl restart apache2

# Verify correct version is active
php -v
```

#### Issue: "Composer dependencies conflict"
**Solution:**
```bash
# Clear composer cache
composer clear-cache

# Remove vendor directory
rm -rf vendor/

# Restore from backup
tar -xzf /backups/*/codebase_backup.tar.gz

# Or reinstall fresh
composer install --no-dev --optimize-autoloader
```

### Emergency Contact Information
```
PRIMARY CONTACTS:
- Development Lead: [Phone] / [Email]
- DevOps Engineer: [Phone] / [Email]  
- Database Administrator: [Phone] / [Email]

ESCALATION CONTACTS:
- Technical Director: [Phone] / [Email]
- CTO: [Phone] / [Email]
- Infrastructure Team: [Phone] / [Email]

VENDOR SUPPORT:
- Hosting Provider: [Support Number]
- Database Support: [Support Number]
```

---

## ✅ FINAL ROLLBACK CHECKLIST

### Pre-Migration Preparation
- [ ] Rollback scripts tested and verified
- [ ] Complete backups created and validated
- [ ] Emergency contact list updated
- [ ] Team trained on rollback procedures
- [ ] Monitoring systems configured
- [ ] Communication templates prepared

### During Migration Monitoring
- [ ] Rollback scripts accessible and ready
- [ ] Team members on standby
- [ ] Monitoring dashboards active
- [ ] Communication channels open
- [ ] Decision makers available

### Post-Rollback Actions  
- [ ] System functionality verified
- [ ] Performance metrics restored
- [ ] All services operational
- [ ] Users notified of restoration
- [ ] Incident documentation completed
- [ ] Lessons learned captured

---

**REMEMBER:** The best rollback is the one you never need to use. Thorough testing and preparation are your best defense against needing emergency rollbacks.

**Document Status:** READY FOR USE  
**Last Tested:** [Update after testing]  
**Next Review:** [Update quarterly]  

*Keep this document easily accessible during migration deployment.*