gcp_kms_emulator

GCP KMS Emulator

IAM-enforced KMS emulator — Test encryption AND permissions locally, fail like production would.

A production-grade KMS implementation with optional pre-flight IAM enforcement. Unlike standard emulators that allow everything, this can deny unauthorized cryptographic operations using production-style IAM authorization policies.

Dual protocol support: Native gRPC + REST/HTTP. Real encryption: AES-256-GCM (not mocked). No GCP credentials required.

Why This Emulator Is Different

Most KMS emulators skip authorization. This one can enforce production-style IAM authorization policies using the IAM Emulator as a control plane.

Pre-flight enforcement catches permission bugs in development/CI, not production.

The Hermetic Seal

Before Blackwell, "GCP Hermetic Testing" was essentially impossible.

Google's official emulators have a critical flaw: they ignore authorization. Your tests pass locally because the emulator allows everything, then fail in production when IAM denies the request.

The two bad options:

1. Fake Auth - Emulator ignores permissions (fast, but catches zero IAM bugs)
2. Staging Leak - Call real GCP IAM API (hermetic seal broken, tests become flaky)

Blackwell closes the hermetic seal:

With IAM enforcement enabled, your tests:

• Fail for the same authorization reasons production would (PermissionDenied errors)
• Run completely offline (no network, no GCP credentials)
• Execute deterministically (0ms IAM propagation delay vs 1-60s in real GCP)

This is true hermetic testing - all dependencies sealed inside the boundary, no external leaks.

Enforcement Modes

Scope note: IAM enforcement in this emulator is scoped for CI authorization testing. It validates authorization intent and high-risk KMS operations (encrypt, decrypt, destroy, policy changes), not full GCP IAM semantic parity.

• Off (default) - No IAM checks, fast iteration
• Permissive - Enforce when IAM available, allow on connectivity errors (fail-open)
• Strict - Always enforce, deny on connectivity errors (fail-closed, CI-ready)

The Security Paradox:

"A test that cannot fail due to a permission error is a test that has not fully validated the code's production readiness."

Use strict mode in CI to catch IAM bugs before deployment, not during Friday night incidents.

Usage Modes

Standalone - Run independently for KMS-only testing:

server-dual
# Single service, no IAM enforcement (mode=off)

With IAM Enforcement - Run standalone with IAM checks:

# Start IAM emulator first
cd ../gcp-iam-emulator && ./bin/server --config policy.yaml

# Start KMS with enforcement
IAM_MODE=strict IAM_EMULATOR_HOST=localhost:8080 server-dual
# Now requires valid permissions for encrypt/decrypt operations

Orchestrated Ecosystem - Use with GCP IAM Control Plane for multi-service testing:

gcp-emulator start
# KMS + Secret Manager + IAM emulator
# Single policy file, unified authorization

Choose standalone for simple workflows, IAM-enforced for production-like testing.

Features

Core Functionality

• Dual Protocol Support - Native gRPC + REST/HTTP APIs (choose what fits your workflow)
• SDK Compatible - Drop-in replacement for official cloud.google.com/go/kms (gRPC)
• curl Friendly - Full REST API with JSON, test from any language or terminal
• Real Encryption - AES-256-GCM for symmetric encryption (not mocked)
• Key Versioning - Rotation, primary version switching, state transitions

IAM Enforcement (Optional)

• Pre-Flight Authorization - Checks permissions before cryptographic operations
• Deterministic Policy Evaluation - Uses IAM Emulator control plane for decisions
• Three Modes - Off (default), Permissive (fail-open), Strict (fail-closed)
• Production Semantics - Same permission names as real GCP (cloudkms.cryptoKeys.encrypt)
• Fail Like Production - Catch permission bugs in CI, not production

Operations

• No GCP Credentials - Works entirely offline without authentication
• Fast & Lightweight - In-memory storage, starts in milliseconds
• Docker Support - Pre-built containers (gRPC-only, REST-only, or dual)
• Thread-Safe - Concurrent access with proper synchronization

Supported Operations

Key Management

• CreateKeyRing - Create new keyrings
• GetKeyRing - Retrieve keyring metadata
• ListKeyRings - List all keyrings
• CreateCryptoKey - Create encryption/decryption keys
• GetCryptoKey - Retrieve key metadata
• ListCryptoKeys - List all keys in a keyring
• UpdateCryptoKey - Update key metadata (labels)

Key Versioning

• CreateCryptoKeyVersion - Create new key versions for rotation
• GetCryptoKeyVersion - Get specific version details
• ListCryptoKeyVersions - List all versions of a key
• UpdateCryptoKeyPrimaryVersion - Switch to a different key version
• UpdateCryptoKeyVersion - Update version state (enable/disable)
• DestroyCryptoKeyVersion - Schedule version for destruction

Encryption

• Encrypt - Encrypt data with a crypto key (AES-256-GCM)
• Decrypt - Decrypt data with a crypto key (works with any enabled version)

Version State Transitions

PENDING_GENERATION → ENABLED → DISABLED → DESTROY_SCHEDULED → DESTROYED
                        ↑          ↓
                        └──────────┘

Not Yet Implemented

• Key lifecycle (RestoreCryptoKeyVersion)
• Asymmetric operations (AsymmetricSign, AsymmetricDecrypt, GetPublicKey)
• MAC operations (MacSign, MacVerify)
• Import/Export (ImportCryptoKeyVersion, CreateImportJob, etc.)
• Raw operations (RawEncrypt, RawDecrypt, Decapsulate)
• Random generation (GenerateRandomBytes)

Current coverage: 14 of ~26 methods (54%) - complete key management + lifecycle

Quick Start

Choose Your Protocol

Three server variants available:

Install

# gRPC only (recommended for SDK users)
go install github.com/blackwell-systems/gcp-kms-emulator/cmd/server@latest

# REST API only
go install github.com/blackwell-systems/gcp-kms-emulator/cmd/server-rest@latest

# Both protocols
go install github.com/blackwell-systems/gcp-kms-emulator/cmd/server-dual@latest

Run Server

gRPC server:

# Start on default port 9090
server

# Custom port
server --port 8080

REST server:

# Start on default ports (gRPC: 9090, HTTP: 8080)
server-rest

# Custom ports
server-rest --grpc-port 9090 --http-port 8080

Dual protocol server:

# Start both protocols (gRPC: 9090, HTTP: 8080)
server-dual

# Custom ports
server-dual --grpc-port 9090 --http-port 8080

Use with GCP SDK

package main

import (
    "context"
    "fmt"

    kms "cloud.google.com/go/kms/apiv1"
    "google.golang.org/api/option"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials/insecure"
)

func main() {
    ctx := context.Background()

    // Connect to emulator instead of real GCP
    conn, _ := grpc.NewClient(
        "localhost:9090",
        grpc.WithTransportCredentials(insecure.NewCredentials()),
    )

    client, _ := kms.NewKeyManagementClient(ctx, option.WithGRPCConn(conn))
    defer client.Close()

    // Use client normally - API is identical to real GCP
    // ...
}

Use with REST API

Start REST server:

server-rest
# HTTP gateway listening at :8080

Create a keyring:

curl -X POST "http://localhost:8080/v1/projects/my-project/locations/global/keyRings?keyRingId=my-keyring"

Create a crypto key:

curl -X POST "http://localhost:8080/v1/projects/my-project/locations/global/keyRings/my-keyring/cryptoKeys?cryptoKeyId=my-key" \
  -H "Content-Type: application/json" \
  -d '{"purpose":"ENCRYPT_DECRYPT"}'

Encrypt data:

curl -X POST "http://localhost:8080/v1/projects/my-project/locations/global/keyRings/my-keyring/cryptoKeys/my-key:encrypt" \
  -H "Content-Type: application/json" \
  -d '{"plaintext":"'$(echo -n "my-secret-data" | base64)'"}'

Decrypt data:

curl -X POST "http://localhost:8080/v1/projects/my-project/locations/global/keyRings/my-keyring/cryptoKeys/my-key:decrypt" \
  -H "Content-Type: application/json" \
  -d '{"ciphertext":"<base64-ciphertext>"}'

REST API matches GCP's official REST endpoints - same paths, same JSON format, same behavior.

IAM Integration

The KMS emulator supports optional permission checks using the GCP IAM Emulator.

Configuration

Environment Variables:

• IAM_MODE - Controls permission enforcement (default: off)
  • off - No permission checks (legacy behavior)
  • permissive - Check permissions, fail-open on connectivity errors
  • strict - Check permissions, fail-closed on connectivity errors (for CI)
• IAM_EMULATOR_HOST - IAM emulator address (default: localhost:8080)

Usage

Without IAM (default):

# No permission checks - all operations succeed
server

With IAM (permissive mode):

# Start IAM emulator first
iam-emulator

# Start KMS with IAM checks (fail-open)
IAM_MODE=permissive IAM_EMULATOR_HOST=localhost:8080 server

With IAM (strict mode for CI):

# All operations require valid permissions
IAM_MODE=strict IAM_EMULATOR_HOST=localhost:8080 server

Principal Injection

Specify the calling principal for permission checks:

gRPC:

ctx := metadata.AppendToOutgoingContext(ctx, "x-emulator-principal", "user:admin@example.com")
resp, err := client.CreateKeyRing(ctx, req)

REST:

curl -H "X-Emulator-Principal: user:admin@example.com" \
  -X POST "http://localhost:8080/v1/projects/my-project/locations/global/keyRings?keyRingId=my-keyring"

Permissions

KMS operations map to GCP IAM permissions:

Mode Differences

Use off for local dev, permissive for integration tests, strict for CI.

Why IAM Enforcement Uses Curated Permissions (On Purpose)

IAM enforcement in this emulator is deliberately scoped for authorization testing, not comprehensive permission modeling. The underlying IAM emulator checks a small set of built-in roles (primitives + Secret Manager + KMS) plus unlimited custom role definitions to catch the bugs that actually break production: missing permissions, wrong role assignments, and unauthorized destructive operations (destroy, setIamPolicy, disable). This curated-first approach catches 95% of real-world authorization bugs while maintaining hermetic execution (no GCP credentials required), deterministic behavior (0ms propagation delay vs 1-60s in real GCP), and zero maintenance burden from tracking GCP's evolving role catalog. If you need to test additional permissions, define them explicitly in the IAM emulator's policy.yaml as custom roles — this explicit approach is simpler, more reliable, and avoids the catalog staleness problem that plagues comprehensive IAM emulation. We optimize for authorization failures that matter, not theoretical IAM completeness.

Docker

Build Docker Images

# Build all variants
make docker

# Or build individually
docker build --build-arg VARIANT=grpc -t kms-emulator:grpc .  # gRPC only (default)
docker build --build-arg VARIANT=rest -t kms-emulator:rest .  # REST only
docker build --build-arg VARIANT=dual -t kms-emulator:dual .  # Both protocols

Run Docker Containers

gRPC only:

docker run -p 9090:9090 gcp-kms-emulator:grpc

REST only:

docker run -p 8080:8080 gcp-kms-emulator:rest

Dual protocol:

docker run -p 9090:9090 -p 8080:8080 gcp-kms-emulator:dual

In CI/CD

GitHub Actions:

services:
  gcp-kms:
    image: gcp-kms-emulator:dual
    ports:
      - 9090:9090
      - 8080:8080

Docker Compose:

services:
  gcp-kms:
    image: gcp-kms-emulator:dual
    ports:
      - "9090:9090"  # gRPC
      - "8080:8080"  # REST

Use Cases

• Local Development - Test KMS encryption without cloud access
• CI/CD Pipelines - Fast integration tests without GCP credentials
• Unit Testing - Deterministic encryption behavior
• Security Testing - Validate encryption workflows
• Cost Reduction - Avoid GCP API charges during development

Maintained By

Maintained by Dayna Blackwell — founder of Blackwell Systems, building reference infrastructure for cloud-native development.

GitHub · LinkedIn · Blog

Related Projects

• GCP IAM Control Plane - CLI to orchestrate the Local IAM Control Plane (this emulator + IAM + others)
• GCP IAM Emulator - Policy engine (the brain) for IAM enforcement
• GCP Secret Manager Emulator - IAM-enforced Secret Manager data plane
• gcp-emulator-auth - Enforcement proxy library (the guard)

Who's Using This?

If you're using this KMS emulator — in CI, locally, or in a test harness — I'd love to hear how you're using it.

• What crypto bugs did you catch? (unauthorized encrypt/decrypt, key version issues, IAM policy problems)
• Are you using real cryptographic operations? (testing with actual key material, or mocking crypto)
• How are you managing keys? (versioning, rotation, destruction workflows)
• What's still friction? (missing algorithms, IAM integration complexity, key import limitations)

Open an issue, start a discussion, or reach out directly:

📬 dayna@blackwell-systems.com

This helps shape the roadmap and ensures the project stays aligned with real-world needs.

License

Apache 2.0 - See LICENSE for details.