A Comprehensive Guide to Setting Resource Limits for Docker Containers with Ulimits

Docker has rapidly become the de-facto standard for packaging and deploying applications in modern software delivery pipelines. Based on Linux containers, Docker provides reproducible environments to run applications in isolation with portability across environments.

However, running containers efficiently in production requires careful resource management to ensure availability, performance and stability. This in-depth guide covers how Docker leverages ulimits to restrict resource consumption and best practices around configuring container limits for production workloads.

Understanding Process Resource Limits

In Linux, the ulimit command allows setting per-user and per-process limits on utilization of various system resources like CPU, memory, files etc.

Internally, ulimit configures the RLIMIT values within the Linux kernel that governs the ceiling of resources available to any process.

Some common ulimit resource restrictions are:

Resource	Description
nofile	Max open files
nproc	Max processes
memlock	Max locked memory
cpu	Max CPU time
fsize	Max file size

These limits are set via two values:

• Soft limit – the maximum limit a process can set for a resource using ulimit. Processes can attempt to increase this up to the hard limit.
• Hard limit – the absolute maximum resource limit no process can exceed. Only configurable by root.

For example:

# Set open files soft limit to 10000, hard limit to 20000
ulimit -Sn 10000 

ulimit -Hn 20000  

Setting appropriate ulimit values prevents processes from overusing critical system resources.

How Docker Applies Ulimits

Docker containers share the host machine‘s resources. Without restrictions, containers can utilize as much resources as available and starve others.

This is where ulimit comes into play. Docker applies ulimits to set effective resource ceilings per container.

By default, Docker containers inherit the same default ulimit values as the Docker daemon. Typically this allows relatively liberal resource usage.

We can override the defaults and supply custom ulimit values per container to restrict resource utilization.

Additionally, modifying daemon defaults allows setting global container limits cluster-wide. This provides system-level safeguards for capacity planning.

Let us look at various ways to configure ulimits for Docker environments.

Tuning Recommendations for Production

Determining appropriate ulimit values is key for production grade performance.

As a rule of thumb for memory limits, contention can happen when total container memory exceeds 75% of system RAM capacity.

The table below provides recommended baseline ulimit values for production workloads –

Resource	Value	Production Recommendation
nofile	1024	64000+
nproc	1024	Evaluate per workload
memlock	32 MB	128-256 MB
cpu (soft)	Unlimited	~2 CPU cores
fsize	Unlimited	10-100GB

• The open files limit tend to run small for containers, tuning to > 64000 is recommended based on best practices. Too low can lead to errors under load.
• Memory locks prevent memory from being paged out, 20-25% of container memory limit is optimal.
• Per core CPU quotas should be set based on usage to prevent starvation.
• File sizes between 10-100GB are reasonable limits depending on storage layer.

Additionally, processes like databases and message queues often require tweaking these limits specific to their usage.

Now let us look at how to configure these ulimits in Docker environments.

Setting Ulimits for Docker Daemon

To set system-wide defaults for containers across a Docker host, we need to configure ulimit settings on the Docker daemon.

The daemon runs as a system service process dockerd responsible for managing container lifecycles and exposing commands like docker run.

Most Docker installation like desktop apps and Kubernetes don‘t directly invoke dockerd. Instead they configure daemon parameters via configuration files.

Configure via Daemon Config File

The /etc/docker/daemon.json file allows setting daemon level configurations like container defaults.

To configure ulimits here, we need to specify the JSON structure:

{
  "default-ulimits": {
    "nofile": {
      "Name": "nofile",
      "Hard": 65535,
      "Soft": 65535
    },
    "nproc": {
      "Name": "nproc",
      "Hard": 4096,
      "Soft": 1024
    }
  }
}

This sets open files limit to 65535 and max processes to 4096 globally for all containers.

For this to apply, we have to restart the Docker service for changes to be picked up.

Configure with dockerd

For hosts running dockerd directly, we can set --default-ulimit flags on startup:

dockerd \
  --default-ulimit nofile=65535:65535 \
  --default-ulimit nproc=4096:1024

This starts the daemon with overriden ulimits from defaults.

In most production environments, configuring via daemon.json is recommended since that avoids directly modifying service startup scripts.

Pros and Cons of Daemon Ulimits

Setting global daemon ulimits for containers has a few pros and cons:

Pros:

• Simple cluster-wide safeguards for capacity planning.
• Prevents runaway resource usage from malicious or defective containers.

Cons:

• Cannot set nuanced limits per workload.
• Changes require daemon restart affecting all containers.

As such generally a moderate baseline should be set on the Docker host complemented by custom tuning via Docker run overrides on a case basis.

Setting Ulimits with Docker Run

For granularity in resource allocation, it is common to configure ulimits uniquely per container.

Docker allows overriding the daemon defaults when directly invoking docker run via the --ulimit flag.

docker run \ 
  --ulimit nofile=131072:131072 \
  --ulimit nproc=8192:16384 
  --rm webapp 

This allows starting containers from any image with the specified ulimit values applied.

We can further parameterize docker run to script setting variable limits:

container.sh

#!/bin/bash

NOFILE=$1
NPROC=$2

docker run \
  --ulimit nofile=${NOFILE}:${NOFILE} 
  --ulimit nproc=${NPROC}:$(expr $NPROC \* 2)
  --rm webapp

And invoke above as –

container.sh 131072 16384

This provides a reusable way to launch containers with parameterized resource constraints.

Setting appropriate limits close to application needs allows using host capacity optimally without statistical multiplexing overhead.

An Example Capacity Planning Calculation

Let us look at a quick example to size production ulimits for a containerized web application server like Nginx.

• We expect ~1000 concurrent requests with spikes upto 5000 requests during events
• Each concurrent request holds at-least one socket open
• With keep-alive enabled – around 1.2 open socket per request
• Memory usage around 300MB on average

Using this workload profile, we can derive ulimit values:

• Open files
  • Average case – 1000 concurrent requests x 1.2 sockets per request = 1200
  • Spike case – 5000 concurrent requests x 1.2 sockets per request = 6000
• Set nofile soft limit to 6000 and hard limit to 10000 for buffer
• Memory lock 10-25% of 300MB = 30MB to 75MB
• Set nproc, cpu etc. limits through benchmarking

Similarly production usage expectations can be translated into container ulimit configurations.

Managing Ulimits with Docker Compose

For local development and testing containers defined via docker-compose manifests, we can set ulimit configuration as:

services:
  webapp: 
    image: webapp:local
    ulimits:
      memlock:
        soft: 100000
        hard: 200000
      nofile:
        soft: 1000000 
        hard: 1100000   

This allows revisions during iterative devlopment without affecting an entire Docker daemon.

Running standard benchmarks while tweaking ulimits helps finalize appropriate production values.

Monitoring Resource Usage for Optimization

Once containers are running with restrictions, actually utilization needs to be monitored to prevent unexpected app degradation.

The docker stats CLI provides live resource usage statistics for running containers:

CONTAINER ID   NAME    CPU %   MEM USAGE / LIMIT   MEM %   NET I/O   BLOCK I/O    PIDS
c265e246c706   webapp  2.23%   209MiB / 1.5GiB   13.79%  788B / 648B   10.42MB / 64.4MB  55

This shows current memory, CPU usage against the limits along side network and disk activity.

Ideally average usage should reach 40-50% of defined limits to balance application performance and overprovisioning.

For containers throttled unexpectedly, iterative benchmarking while adjusting ulimits helps determine the optimal thresholds.

Performance Impact of Limits

Resource limits provide isolation and prevent noisy neighbor issues amongst containers. However incorrectly sized limits can severely impact application performance.

As an example, database transaction throughput changes with open files limit:

Open Files Limit	Transactions Per Second
10,000	Incomplete requests
100,000	170 TPS
1,000,000	650 TPS

We see 2-3x throughput fluctuation based on the file descriptor limit which allows more client connections.

Similarly, low memory limits can cause aggressive kernel paging of container processes. High application usage typically triggers the [[OOM killer]] that halts containers to avoid system instability.

So while limits successfully prevent resource hijacking, adequate buffer should be provided driven by data from application profiling and monitoring.

Real World Examples of Limit-Driven Failure

Let‘s analyze a few ways running with overly restrictive default ulimits can become production disasters –

1. Kubernetes CronJob – A data warehouse ETL ingestion cron crashed consistently due to low open files limit inherited from cluster defaults disallowing bulk import with multiple output tables.

2. Kafka Cluster – Rack-aware replicas kept getting stuck during rebalance without any offset movement. Root cause was fixed JVM heap limits disallowing the large in-memory offsets topic when scaling brokers. Reverting to default dynamic sizing fixed stability.

3. Redis Cluster – Persistent high redis master CPU usage was attributed to an ancient default 1GB ram limit preventing cache performance coupled with lazy eviction configuration. Raising this eliminated stall spikes.

In all cases, blindly inheriting daemon defaults without use case consideration led to near-unusable systems often requiring deep investigation into root causes.

Setting declarative resource limits aligned to production traffic expectations optimizes environment utilization and costs. But this requires thorough data-driven analysis into application needs.

Kubernetes and Cluster Resource Management

For container orchestration platforms like Kubernetes, achieving high resource utilization across infrastructure clusters is critical from cost and reliability perspective.

Kubernetes provides first class support for cluster level and container level resource management through its ResourceQuota, LimitRange and Pod requests/limits semantics.

For example, resource quotas allow restricting resource consumption across a namespace –

apiVersion: v1
kind: ResourceQuota  
metadata:
  name: compute-quota
  namespace: production
spec:
  hard:
    requests.cpu: "24"
    requests.memory: 100Gi
    limits.cpu: "48"
    limits.memory: 200Gi

This ensures production workloads stay within capacity boundaries.

Additionally container resource requests can be specified declaratively –

apiVersion: v1
kind: Pod
metadata:
  name: webapp-pod
  labels:

spec:

  containers:
  - name: webapp
    image: webapp:v1
    resources:
      requests:
        memory: "1Gi"
        cpu: "500m" 
      limits:
        memory: "2Gi"
        cpu: "1"

This allows Kubernetes to binpack containers to nodes efficiently.

Internally, Kubernetes interacts with container runtimes (like Docker) to actually enforce the specified constraints using mechanisms like control groups, namespaces and ulimits.

A typical production cluster runs at 60-70% capacity with moderate overprovisioning thanks to native integration of resource analytics and controls.

Fine grained resource allocation tuned to application delivery needs ensures high cluster utilization without performance degradation.

Conclusion

From small developer laptops to massive production clusters, resource allocation is critical for container deployments. ulimit provides a simple yet powerful construct for controlling process resource consumption in Linux. Docker transparently maps ulimit to set effective ceilings per container.

While daemon defaults help safeguard hosts, nuanced tuning taking workload concurrency and data throughput into account is vital to prevent unexpected app issues. Kubernetes enhances this further with native cluster monitoring and quota management.

The key insight is application performance tightly couples to its environment limits. Purpose built resource allocation aligned to production delivery needs optimizes infrastructure without overprovisioning while maintaining velocity.

I hope this comprehensive guide helped you learn all about setting resource constraints for containers using ulimits including real world best practices! Let me know if you have any other questions.