Skip to content

paiml/data-pipelines-deno-typescript-course

Repository files navigation

🚀 Data Pipelines with Deno: EU to USA Conversion Course

Deno TypeScript Coverage SATD

Build production-grade data pipelines with Deno, TypeScript, and AI-assisted programming. Focus on real-world EU to USA data conversion scenarios with zero technical debt.

🎯 Course Overview

This comprehensive 4-week course teaches you to build high-performance data pipelines using Deno's modern runtime. You'll master EU to USA data conversions while learning enterprise-grade project management with PMAT and AI-first development practices.

What You'll Build

A complete, production-ready pipeline that converts:

  • 💱 Currency: EUR → USD with real-time rates
  • 📏 Units: Metric → Imperial (meters → feet, Celsius → Fahrenheit)
  • 📅 Dates: DD/MM/YYYY → MM/DD/YYYY
  • 🔢 Numbers: 1.234,56 → 1,234.56
  • 🏠 Addresses: EU format → USA format
  • 📞 Phone: +44 → +1 formats
  • 💰 Tax: VAT → Sales Tax
  • 🔒 Privacy: GDPR → CCPA compliance

Performance Targets

  • Latency: <100ms P99
  • 🚄 Throughput: >10,000 records/second
  • 📊 Reliability: 99.99% uptime
  • 🧪 Quality: 90% test coverage, 0 technical debt

🚦 Quick Start

Prerequisites

# Required versions
deno --version  # >=2.0.0
git --version   # >=2.40.0

# Optional but recommended
node --version  # >=20.0.0 (for tooling)
docker --version # >=24.0.0

Installation

# Clone the repository
git clone https://github.com/yourusername/data-pipelines-deno-typescript-course.git
cd data-pipelines-deno-typescript-course

# Initialize Deno configuration
deno task init

# Run initial tests
deno task test

# Start development
deno task dev

📚 Course Structure

Week 1: Foundations 🏗️

  • Deno runtime mastery
  • TypeScript strict mode
  • AI programming patterns
  • Basic converters (currency, units, dates)
  • Project: Multi-converter CLI tool

Week 2: Core Pipeline 🔧

  • Stream processing with Web Streams API
  • Backpressure handling
  • Error boundaries
  • Pipeline composition
  • Project: Streaming data pipeline

Week 3: Advanced Processing ⚙️

  • Complex conversions (address, tax, privacy)
  • Performance optimization
  • Caching strategies
  • Parallel processing
  • Project: High-performance pipeline

Week 4: Production Systems 🚀

  • Deployment strategies
  • Monitoring and alerting
  • CI/CD pipeline
  • Performance validation
  • Final Project: Production EU→USA pipeline

🛠️ Development

Project Structure

src/
├── converters/           # Conversion modules
│   ├── currency/        # EUR → USD
│   ├── units/          # Metric → Imperial
│   └── formats/        # Dates, numbers, etc.
├── pipeline/           # Stream processing
│   ├── ingestion/      # Input validation
│   ├── transformation/ # Core conversions
│   └── output/        # Delivery layer
└── shared/            # Common utilities
    ├── types/         # TypeScript types
    ├── utils/         # Helper functions
    └── constants/     # Configuration

Common Commands

# Development
deno task dev          # Start with watch mode
deno task test         # Run tests with coverage
deno task bench        # Run performance benchmarks

# Quality
deno task lint         # Lint code
deno task fmt          # Format code
deno task check        # Type check
deno task quality      # Run all quality checks

# Build
deno task build        # Compile for production
deno task deploy       # Deploy to Deno Deploy

🛠️ Deno Tools Demo

Try out all Deno tooling commands with our guaranteed-working demo files:

# Basic commands that work perfectly
deno check demo.ts         # ✅ Type check (no errors!)
deno fmt demo.ts           # ✅ Format code  
deno lint demo.ts          # ✅ Lint code
deno run demo.ts           # ✅ Run the demo
deno test demo.test.ts     # ✅ Run unit tests
deno bench demo.bench.ts   # ✅ Performance benchmarks
deno doc demo.ts           # ✅ Generate documentation
deno compile demo.ts       # ✅ Create executable

# Test all commands at once
./test-demo-commands.sh    # Automated verification

📚 See DenoTools.md for complete documentation with examples.

⚡ Deno Task Runner Demo

Explore Deno's powerful task runner with our comprehensive examples:

# Demo tasks (guaranteed to work)
deno task demo:basic       # ✅ Run the demo app
deno task demo:test        # ✅ Run demo tests  
deno task demo:bench       # ✅ Run demo benchmarks
deno task demo:all         # ✅ Run all demo tasks

# Development workflow
deno task dev              # Start with file watching
deno task serve            # Start HTTP server
deno task quality          # Run all quality checks
deno task build            # Build for production

# Advanced task pipelines
deno task pipeline:quality # Format → Lint → Check → Test
deno task pipeline:build   # Quality checks → Build
deno task pipeline:deploy  # Build → Deploy to staging

# Utility tasks
deno task clean            # Clean build artifacts
deno task health           # Run health checks
deno task info             # Show project information

🔗 See DemoTasks.md for complete task runner documentation with 50+ examples.

🎨 Ruchy Language Integration Demo

Explore advanced cross-language integration with our Ruchy programming language demo:

# Ruchy integration demo
deno task ruchy:demo           # ✅ Complete Deno × Ruchy integration
deno task ruchy:demo:shell     # ✅ Interactive Ruchy shell via Deno

# Direct Ruchy commands (requires ruchy installation)
deno task ruchy:check          # Check if Ruchy is installed
deno task ruchy:eval           # Evaluate Ruchy expressions
deno task ruchy:repl           # Start Ruchy REPL
deno task ruchy:run            # Run Ruchy scripts
deno task ruchy:fmt            # Format Ruchy code
deno task ruchy:test           # Run Ruchy tests
deno task ruchy:ast            # Generate AST from Ruchy code

# Install Ruchy (requires Rust)
cargo install ruchy           # Install from crates.io

Demo Features:

  • 🔄 Bi-directional Integration: Deno TypeScript invokes Ruchy language
  • Performance Comparison: Fibonacci benchmarks between languages
  • 🧪 Code Examples: Currency conversion, pattern matching, functional programming
  • 🌳 AST Generation: Parse and analyze Ruchy code structure
  • 🎯 Error Handling: Comprehensive error reporting and validation
  • 🔍 Syntax Checking: Validate Ruchy code before execution

📋 See ruchy-demo.ts for complete implementation and examples.

📊 Quality Standards

Zero SATD Policy

  • ❌ No TODO, FIXME, HACK, or XXX comments
  • ✅ Complete implementations or create tickets
  • 🚨 Build fails on any SATD detection

Complexity Limits

  • Cyclomatic complexity: ≤10
  • Cognitive complexity: ≤15
  • Function length: ≤50 lines
  • File length: ≤300 lines

Coverage Requirements

  • Week 1: 70% minimum
  • Week 2: 80% minimum
  • Week 3: 85% minimum
  • Week 4: 90% minimum

🤖 AI Programming

This course emphasizes AI-assisted development:

  1. Specification-first: Write detailed specs, AI generates implementation
  2. Test-driven with AI: Define tests, AI creates code to pass them
  3. AI code review: Automated reviews for style and conventions
  4. Prompt patterns: Learn effective prompting for 10x productivity

Recommended Tools

📈 Progress Tracking

Current Sprint: Week 1 - Foundation Setup

  • Configure Deno development environment
  • Setup AI programming tools
  • Initialize PMAT configuration
  • Create project structure
  • Setup quality gates
  • Implement basic converters

Track detailed progress in ROADMAP.md

🧪 Testing

# Run all tests
deno test

# Run specific test file
deno test src/converters/currency/eur-to-usd.test.ts

# Run with coverage
deno test --coverage=coverage

# Generate coverage report
deno coverage coverage --lcov > coverage.lcov

# Run benchmarks
deno bench

🚀 Deployment

Deno Deploy (Recommended)

# Install deployctl
deno install --allow-all --no-check -r -f https://deno.land/x/deploy/deployctl.ts

# Deploy to production
deployctl deploy --project=eu-usa-pipeline src/main.ts

Alternative Platforms

  • AWS Lambda with Deno Layer
  • Cloudflare Workers
  • Google Cloud Run
  • Docker containers

📖 Documentation

🤝 Contributing

  1. Follow the Code of Conduct
  2. Check the ROADMAP.md for current tasks
  3. Create tickets for features >3 hours
  4. Maintain 90% test coverage
  5. Zero SATD policy applies

📝 License

This course is proprietary educational content. See LICENSE for details.

🙏 Acknowledgments

  • Deno team for the amazing runtime
  • TypeScript team for the type system
  • Claude and GitHub Copilot for AI assistance
  • Toyota Production System for quality principles

Ready to build production-grade pipelines? Start with Week 1: Foundations

About

"Build with AI: Data Pipelines and Stream Processing with Deno

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors