🎬 AI-powered video transcription platform combining audio transcription, AI lip reading, and speaker diarization for accessible, accurate video captions.

• 🎤 Audio Transcription: AWS Transcribe with speaker diarization
• 👄 AI Lip Reading: LipCoordNet visual speech recognition via SageMaker
• 👤 Face Detection: AWS Rekognition face tracking
• 🎯 Speaker Fusion: Intelligent alignment of audio + visual + face data
• 🌐 Web Interface: Modern drag-&-drop upload + caption viewer
• ⚡ GPU-Accelerated: Fast inference on SageMaker Serverless
• 🏗️ Serverless Pipeline: AWS Lambda + Step Functions orchestration

• AWS Account with CLI configured
• Git, Terraform, Python 3.8+
• SSH client (for EC2 access)

git clone https://github.com/gordowuu/GORGGLES2.git
cd GORGGLES2

# Package Lambda layers
cd scripts
.\package_lambda_layers.ps1 -Region us-east-1

# Apply Terraform
cd ..\infra\terraform
terraform init
terraform apply

# Build model artifact
python scripts/build_lipcoordnet_artifact.py

# Upload to S3
aws s3 cp artifacts/model-lipcoordnet.tar.gz s3://gorggle-dev-uploads/sagemaker-models/

# Deploy SageMaker endpoint
python scripts/deploy_lipcoordnet.py `
  --endpoint-name gorggle-lipcoordnet-dev `
  --role-arn arn:aws:iam::YOUR_ACCOUNT:role/service-role/AmazonSageMaker-ExecutionRole `
  --instance-type ml.g5.xlarge

# Test with video from S3
python scripts/test_lipcoordnet_endpoint.py `
  --video-bucket gorggle-dev-uploads `
  --video-key test-video.mov

# Open web interface
start web\index.html

┌─────────────┐
│   User      │
│  Upload MP4 │
└──────┬──────┘
       │
       ▼
┌─────────────────────────────────────────────────────┐
│              S3 Upload Bucket                       │
│           s3://gorggle-dev-uploads                  │
└────────────────────┬────────────────────────────────┘
                     │ S3 Event Trigger
                     ▼
┌─────────────────────────────────────────────────────┐
│         Step Functions State Machine                │
│          (Parallel Processing)                      │
├─────────────────────────────────────────────────────┤
│                                                     │
│  ┌──────────────────┐  ┌─────────────────────┐    │
│  │ Extract Media    │  │ AWS Transcribe      │    │
│  │ (FFmpeg/OpenCV)  │  │ (Audio + Speakers)  │    │
│  └────────┬─────────┘  └──────────┬──────────┘    │
│           │                        │               │
│           │  ┌─────────────────────┴──┐            │
│           │  │  AWS Rekognition       │            │
│           │  │  (Face Detection)      │            │
│           │  └─────────────┬──────────┘            │
│           │                │                        │
│           ▼                │                        │
│  ┌──────────────────────┐ │                        │
│  │  LipCoordNet         │ │                        │
│  │  SageMaker Serverless│ │                        │
│  │  (GPU Inference)     │ │                        │
│  └────────┬─────────────┘ │                        │
│           │                │                        │
│           └────────────────┴───────────┐            │
│                                        ▼            │
│                              ┌──────────────────┐   │
│                              │  Fuse Results    │   │
│                              │  (Align & Merge) │   │
│                              └────────┬─────────┘   │
└───────────────────────────────────────┼─────────────┘
                                        │
                     ┌──────────────────┴──────────────────┐
                     ▼                                     ▼
          ┌────────────────────┐              ┌─────────────────┐
          │  S3 Processed      │              │   DynamoDB      │
          │  Bucket (JSON)     │              │   (Job Index)   │
          └─────────┬──────────┘              └─────────────────┘
                    │
                    │
          ┌─────────▼──────────────────────────────────┐
          │       API Gateway + Lambda                 │
          │   GET /results/{jobId}                     │
          └─────────┬──────────────────────────────────┘
                    │
                    ▼
          ┌──────────────────────┐
          │   Web Interface      │
          │   (Caption Viewer)   │
          └──────────────────────┘

LipCoordNet is a state-of-the-art visual speech recognition model:

✅ Trained on GRID corpus for high-accuracy lip reading
✅ Lightweight and fast inference (optimized for production)
✅ Works with 128×64 mouth ROI crops
✅ Pre-trained weights available on HuggingFace
✅ Integrates seamlessly with AWS SageMaker

Deployment: Serverless SageMaker inference endpoints with auto-scaling

Model: SilentSpeak/LipCoordNet on HuggingFace

GORGGLES2/
├── 📄 README.md                          # You are here
├── 📄 ARCHITECTURE.md                    # System design details
├── 📄 DEPLOYMENT_CHECKLIST.md            # Pre-deployment validation
├── 📄 TODO.md                            # Roadmap
│
├── 📂 sagemaker/                         # SageMaker model deployment
│   ├── inference_lipcoordnet.py          # Custom inference handler
│   ├── requirements_lipcoordnet.txt      # Model dependencies
│   └── container/                        # Docker container (optional)
│
├── 📂 infra/terraform/                   # Infrastructure as Code
│   ├── main.tf                           # Core resources
│   ├── security_groups.tf                # VPC networking
│   ├── lambda_layers.tf                  # Lambda layers config
│   └── variables.tf                      # Configuration variables
│
├── 📂 lambdas/                           # Lambda functions
│   ├── extract_media/                    # FFmpeg extraction
│   ├── invoke_lipreading/                # SageMaker caller
│   ├── fuse_results/                     # Result merger
│   ├── get_results/                      # API handler
│   ├── s3_trigger/                       # Pipeline starter
│   ├── start_transcribe/                 # AWS Transcribe
│   └── start_rekognition/                # Face detection
│
├── 📂 scripts/                           # Deployment automation
│   ├── build_lipcoordnet_artifact.py     # Build model.tar.gz
│   ├── deploy_lipcoordnet.py             # Deploy to SageMaker
│   ├── test_lipcoordnet_endpoint.py      # Test endpoint
│   └── package_lambda_layers.ps1         # Layer packager
│
└── 📂 web/                               # Frontend
    ├── index.html                        # Upload + viewer UI
    └── README.md                         # Frontend guide

Monthly cost for moderate usage (~50 videos/month, 5 min avg):

Cost optimization tips:

• Use SageMaker Serverless (pay per inference, no idle costs)
• Set S3 lifecycle policies (delete old files after 30 days)
• Use Step Functions Express workflows for cheaper executions
• Batch multiple videos to reduce cold starts

• ✅ IAM roles with least-privilege policies
• ✅ VPC security groups for network isolation
• ✅ Lambda VPC integration for EC2 access
• ✅ S3 encryption at rest (SSE-S3)
• ✅ API Gateway with CORS configuration
• ✅ SSH key-based EC2 access only

Best Practices:

• Never commit AWS credentials to Git
• Use AWS Secrets Manager for sensitive data
• Enable CloudTrail for audit logging
• Rotate SSH keys regularly
• Monitor with CloudWatch alarms

# Test with S3 video
python scripts/test_lipcoordnet_endpoint.py `
  --endpoint-name gorggle-lipcoordnet-dev `
  --video-bucket gorggle-dev-uploads `
  --video-key test-video.mov

# Upload test video
aws s3 cp test-video.mp4 s3://gorggle-dev-uploads/uploads/test-001.mp4

# Monitor Step Functions
aws stepfunctions list-executions `
  --state-machine-arn arn:aws:states:us-east-1:ACCOUNT:stateMachine:gorggle-dev-pipeline

# Fetch results
curl https://your-api-id.execute-api.us-east-1.amazonaws.com/results/test-001

Contributions welcome! Areas for improvement:

• Fine-grained word-level timestamps
• Multi-language support
• Real-time streaming processing
• Mobile app integration
• SRT/VTT export
• Speaker name assignment
• Batch processing API

MIT License - see LICENSE for details.

• LipCoordNet: SilentSpeak
• Transformers: HuggingFace
• dlib: Davis King
• AWS: For Transcribe, Rekognition, SageMaker, and cloud infrastructure

• Issues: GitHub Issues
• Documentation: See project documentation files
• Model: LipCoordNet on HuggingFace

Made with ❤️ for accessible AI-powered video transcription

🎬 Start processing videos now! Follow the Quick Start guide above.

Serverless Inference (Recommended):

• Pay only for inference time
• Auto-scales from 0 to thousands of concurrent requests
• 30-60s cold start latency
• Ideal for sporadic workloads

Real-time Endpoint:

• Always-on, no cold starts
• Higher cost (~$1.41/hour for ml.g5.xlarge)
• Use for high-throughput production workloads

The invoke_lipreading Lambda requires the SageMaker endpoint name as an environment variable:

SAGEMAKER_ENDPOINT=gorggle-lipcoordnet-dev

Update this in lambdas/invoke_lipreading/handler.py or set via Terraform.

LipCoordNet requires specific preprocessing:

• Frame rate: 25 FPS
• Mouth ROI: 128×64 pixels
• Face detection: Uses dlib 68-point landmarks
• Crop region: Mouth landmarks (points 48-67)

The SageMaker inference handler automatically performs these steps.

1. Use Terraform for reproducible infrastructure
2. Tag resources with project and environment labels
3. Enable CloudWatch logging for all Lambda functions
4. Set S3 lifecycle rules to auto-delete old videos
5. Use IAM roles with least-privilege policies
6. Monitor costs with AWS Cost Explorer and budgets
7. Test with small videos first (~30 seconds)
8. Enable X-Ray tracing for debugging Step Functions

Problem: Endpoint deployment fails
Solution: Check CloudWatch logs at /aws/sagemaker/Endpoints/gorggle-lipcoordnet-dev

Problem: Cold start timeout
Solution: Increase Lambda timeout to 300s or use async invocation

Problem: Out of memory errors
Solution: Increase SageMaker instance memory (use ml.g5.2xlarge)

Problem: FFmpeg not found in extract_media
Solution: Add FFmpeg Lambda layer ARN to Terraform configuration

Problem: Module import errors
Solution: Package dependencies with pip install -r requirements.txt -t .

Problem: VPC timeout errors
Solution: Ensure Lambda has VPC access and security groups allow outbound traffic

Problem: No face detected
Solution: Ensure person faces camera frontally, adequate lighting

Problem: Poor lip reading accuracy
Solution: LipCoordNet works best with clear frontal face views and minimal motion blur

Problem: Transcribe fails
Solution: Ensure video has audio track and is in supported format (MP4, MOV)

This project previously used AV-HuBERT on EC2. Current version uses LipCoordNet on SageMaker Serverless for better cost-efficiency and scalability.

Key changes:

• ✅ Replaced EC2 GPU instance with SageMaker Serverless
• ✅ Switched from AV-HuBERT to LipCoordNet (HuggingFace)
• ✅ Eliminated infrastructure management overhead
• ✅ Reduced costs by 85% ($150/mo vs $1,014/mo)
• ✅ Faster deployment (2-3 min vs 7-15 min)

Old EC2/AV-HuBERT code is available in git history if needed.