Overview

Relevant source files

• CHANGELOG.md
• LICENSE
• README.md
• shard.yml
• src/crest.cr

This page provides a technical overview of the Crest HTTP client library for Crystal, including its architecture, core components, and capabilities.

Purpose and Scope

Crest is an HTTP and REST client library for Crystal that wraps the standard library's HTTP::Client to provide higher-level abstractions for common HTTP operations. The library is inspired by Ruby's RestClient gem.

The main module is defined in src/crest.cr1-72 and distributed as the crest shard. The current version is 1.6.2 as specified in shard.yml2 and CHANGELOG.md5

Primary capabilities:

• Automatic redirect following (301-308 status codes)
• HTTP Basic and Digest authentication via external shards
• Pluggable parameter encoding strategies (flat, nested, enumerated)
• Multipart file uploads with MIME type detection
• Response streaming for large payloads
• Request/response logging with sensitive data filtering
• HTTP/HTTPS proxy support with authentication
• TLS/SSL configuration via OpenSSL contexts

Sources: src/crest.cr1-72 shard.yml1-29 README.md17-31 CHANGELOG.md5

Architectural Overview

Crest uses a three-tier architecture where multiple public APIs funnel through a common execution layer that delegates to Crystal's standard library HTTP::Client.

Component Architecture

Component Architecture with Code Entities

This diagram maps the public API classes (Crest module in src/crest.cr Crest::Request in src/crest/request.cr Crest::Resource in src/crest/resource.cr) to the execution layer method Crest::Request#execute, which coordinates parameter encoding via Crest::ParamsEncoder#encode, form generation via Crest::DataForm and Crest::UrlencodedForm, and HTTP execution via HTTP::Client#exec from Crystal's standard library.

Sources: src/crest.cr36-69 Diagram 1 from architectural context

Core Component Responsibilities

Component	File Location	Primary Responsibility	Key Methods/Classes
Crest module	src/crest.cr36-69	Static convenience methods generated via metaprogramming for HTTP verbs	Crest.get, Crest.post, Crest.put, Crest.delete, Crest.patch, Crest.options, Crest.head (defined in src/crest.cr44-68)
Crest::Request	src/crest/request.cr	Request builder and executor with full configuration options	Request.new(:method, url, form, **options), Request#execute, Request.get/post/put/delete/patch/options/head
Crest::Resource	src/crest/resource.cr	RESTful resource abstraction maintaining base URL and default configuration	Resource.new(url, **defaults), Resource#[], Resource#get/post/put/delete/patch/options/head
Crest::Response	src/crest/response.cr	Wrapper around HTTP::Client::Response with convenience accessors	Response#body, Response#body_io, Response#status, Response#status_code, Response#headers, Response#cookies, Response#history
Crest::Redirector	src/crest/redirector.cr	Handles automatic redirect following with history tracking	Redirector.new(request, response), Redirector#follow
Crest::ParamsEncoder	src/crest/params_encoder.cr	Abstract base class for parameter encoding strategies	ParamsEncoder#encode, FlatParamsEncoder, NestedParamsEncoder, EnumeratedFlatParamsEncoder, ZeroEnumeratedFlatParamsEncoder
Crest::DataForm	src/crest/forms/data_form.cr	Handles multipart/form-data encoding for file uploads	DataForm#generate_body_string
Crest::UrlencodedForm	src/crest/forms/urlencoded_form.cr	Handles application/x-www-form-urlencoded encoding	UrlencodedForm#generate
Crest::Logger	src/crest/logger.cr	Abstract logging interface with regex-based filtering	Logger#request, Logger#response, Logger#filter, CommonLogger
Crest::Curlify	src/crest/curlify.cr	Converts Crest::Request to equivalent cURL commands	Curlify.new(request), Curlify#to_curl
Crest::ParamsDecoder	src/crest/params_decoder.cr	Decodes URL query strings into parameter hashes	ParamsDecoder.decode(query_string)

Sources: src/crest.cr36-69 README.md44-77 Diagrams 1 and 4 from architectural context

Request Execution Flow

Requests flow through a multi-stage pipeline: parameter encoding, form preparation, authentication setup, HTTP execution, response wrapping, redirect handling, and error processing.

Execution Pipeline Sequence

Request Execution Sequence with Method Calls

This diagram shows the method call chain from Crest.get through Request.new, Request#execute, ParamsEncoder#encode, HTTP::Client#exec, Response.new, and Redirector#follow, illustrating how parameters, authentication, logging, and redirect handling integrate into the execution flow.

Sources: Diagram 2 from architectural context, src/crest.cr44-68

Execution Stages

The execution pipeline consists of the following stages:

1. Request Construction: Request.new(:method, url, form, **options) validates parameters and stores configuration
2. Parameter Encoding: Query parameters are encoded via ParamsEncoder#encode (default: FlatParamsEncoder)
3. Form Body Generation: Form data is converted to body string via DataForm#generate_body_string (multipart) or UrlencodedForm#generate (urlencoded)
4. Authentication Setup: Authorization header is set for HTTP Basic or Digest authentication
5. Proxy Configuration: HTTP::Proxy from the http_proxy shard configures proxy settings if specified
6. TLS Configuration: OpenSSL::SSL::Context::Client is applied for HTTPS requests
7. HTTP Execution: HTTP::Client#exec performs the actual network request
8. Response Wrapping: Response.new(http_client_response, request) wraps the stdlib response
9. Redirect Handling: Redirector#follow recursively calls Request#execute for 3xx status codes
10. Error Processing: If handle_errors: true, non-success status codes raise exceptions from the Crest::RequestFailed hierarchy

Sources: Diagram 2 from architectural context, README.md81-111

Execution Stages

1. Configuration Loading: The request gathers parameters including URL, method, headers, authentication credentials, proxy settings, and timeouts.

2. Parameter Encoding: Query parameters are encoded using a ParamsEncoder strategy. The default is Crest::FlatParamsEncoder which produces key[] notation for arrays.

3. Authentication: If credentials are provided via :user and :password parameters, the library applies HTTP Basic or Digest authentication by setting appropriate headers.

4. HTTP Execution: The request is delegated to Crystal's HTTP::Client from the standard library, which handles the actual network communication.

5. Response Wrapping: The raw HTTP::Client::Response is wrapped in a Crest::Response object that provides convenience methods.

6. Redirect Following: If the status code indicates a redirect (301-308) and :max_redirects is not zero, the Redirector component automatically follows the redirect chain.

7. Error Handling: If :handle_errors is true (default) and the response has an error status code, an exception is raised from the Crest::RequestFailed hierarchy.

Sources: Diagram 2 from context, README.md81-111

Entry Points and API Surface

Crest provides three entry points that all converge on Crest::Request#execute:

1. Crest Module Static Methods

The Crest module in src/crest.cr36-69 defines seven HTTP verb methods generated via metaprogramming in src/crest.cr44-68:

Each method instantiates Crest::Request and calls request.execute. Block variants yield Crest::Response for streaming.

Supported keyword arguments (partial list from README.md88-111):

• :headers - Hash containing request headers
• :cookies - Hash containing request cookies
• :params - Hash or String for query parameters
• :params_encoder - Parameter encoder class (default: Crest::FlatParamsEncoder)
• :auth - Authentication method (:basic or :digest)
• :user and :password - Authentication credentials
• :json - Boolean to send JSON payload with appropriate headers
• :max_redirects - Maximum redirect count (default: 10)
• :logging - Enable logging (default: false)
• :handle_errors - Raise exceptions for error status codes (default: true)

Sources: src/crest.cr36-69 README.md44-77 README.md81-111

2. Crest::Request Class Methods

The Crest::Request class in src/crest/request.cr provides both class methods and instance methods:

Class methods (static constructors):

• Request.get(url, form, **args)
• Request.post(url, form, **args)
• Request.put(url, form, **args) (etc. for all HTTP verbs)
• Request.execute(:method, url, form, **args)

Instance methods:

• Request.new(:method, url, form, **args) - Constructor
• request.execute - Execute the request and return Crest::Response
• request.to_curl - Generate equivalent cURL command
• request.close - Close the underlying HTTP connection
• request.closed? - Check if connection is closed

The class also accepts a block during initialization to modify the request before execution.

Sources: README.md79-161 README.md114-122

3. Crest::Resource Class

The Crest::Resource class in src/crest/resource.cr maintains default configuration that applies to all requests:

Constructor:

Subresource allocation:

• resource["/subpath"] - Returns new Resource with concatenated URL and inherited defaults

HTTP verb methods:

• resource.get(suburl, form, **args)
• resource.post(suburl, form, **args) (etc.)

Final parameters for each request are merged from: (1) defaults from Resource.new, (2) per-request arguments.

Sources: README.md163-248 README.md179-228

Key Features and Capabilities

Authentication and Security

Authentication Methods:

Method	Configuration	Implementation
HTTP Basic	user: "username", password: "secret"	Base64-encoded Authorization: Basic header set by Crest::Request
HTTP Digest	auth: "digest", user: "username", password: "secret"	HTTP::Client::DigestAuth from http-client-digest_auth shard (version >= 0.6.0 per shard.yml13-15)

TLS/SSL Support:

• Custom TLS contexts via tls: OpenSSL::SSL::Context::Client parameter
• Example: Crest.get("https://expired.badssl.com", tls: OpenSSL::SSL::Context::Client.insecure)
• Only applied to HTTPS requests

Sources: README.md478-511 shard.yml13-15 CHANGELOG.md99

Parameter Encoding Strategies

Parameter encoding uses a strategy pattern with the abstract Crest::ParamsEncoder class in src/crest/params_encoder.cr Four concrete encoders are provided:

Encoder Class	Array Encoding	Example Output	Added
FlatParamsEncoder (default)	key[] notation	tags[]=ruby&tags[]=crystal	Initial release
NestedParamsEncoder	Repeated keys	tags=ruby&tags=crystal	v1.1.0 (CHANGELOG.md133)
EnumeratedFlatParamsEncoder	1-based indexing	tags[1]=ruby&tags[2]=crystal	v1.2.0 (CHANGELOG.md112)
ZeroEnumeratedFlatParamsEncoder	0-based indexing	tags[0]=ruby&tags[1]=crystal	v1.2.0 (CHANGELOG.md112)

Usage:

Encoder Methods:

• ParamsEncoder#encode(params : Hash) : String - Converts hash to query string
• ParamsEncoder.flatten_params(object, parent_key) - Recursively flattens nested structures

Sources: README.md319-376 CHANGELOG.md112-152 Diagram 4 from architectural context

Form Data Processing

Form encoding is automatically selected based on content type:

Encoding	Content-Type Header	Trigger Condition	Handler Class
JSON	application/json	json: true option	Serialized via to_json
Multipart	multipart/form-data	Form includes File, IO, or Bytes	Crest::DataForm in src/crest/forms/data_form.cr
URL-encoded	application/x-www-form-urlencoded	Default for hash/string form data	Crest::UrlencodedForm in src/crest/forms/urlencoded_form.cr

Multipart Features:

• Automatic MIME type detection for files
• Support for File.open, IO::Memory, and Bytes objects
• Default MIME type: application/octet-stream (added in v1.3.0 per CHANGELOG.md87)
• Direct file upload support via form: File.open("file.png") (added in v1.3.0 per CHANGELOG.md88-93)

JSON Payload:

• Set json: true to serialize form data to JSON
• Automatically sets Content-Type: application/json header
• Example: Crest.post(url, {"key" => "value"}, json: true)

Sources: README.md405-441 CHANGELOG.md87-93 Diagram 2 from architectural context

Redirect Handling

The Crest::Redirector class in src/crest/redirector.cr automatically follows redirects:

Redirect Behavior:

• Status codes handled: 301, 302, 303, 307, 308
• Default maximum: 10 redirects (configurable via max_redirects option)
• Set max_redirects: 0 to disable redirect following
• Redirects transform POST/PUT/PATCH to GET for 301/302/303
• Request headers and authentication preserved across redirects

Response Properties:

• Response#history : Array(Response) - Array of intermediate responses
• Original request preserved in each response via Response#request

Example:

Sources: README.md577-586 CHANGELOG.md333 Diagram 6 from architectural context

Streaming Responses

Block syntax enables streaming for large responses without loading entire body into memory:

Streaming API:

Key Methods:

• Response#body_io : IO - IO handle for streaming response body
• Response#filename : String? - Extract filename from Content-Disposition header
• Response#content_length : Int64 - Response size from Content-Length header

Implementation:

• All verb methods (Crest.get, Request.get, Resource.get) support block syntax
• Block yields Crest::Response with body_io available for streaming
• Response body not buffered when using block syntax

Sources: README.md378-398 CHANGELOG.md329-333

Logging and Debugging

Logging System:

The Crest::Logger abstract class in src/crest/logger.cr provides logging with two required methods:

• Logger#request(request : Crest::Request) - Log outgoing request
• Logger#response(response : Crest::Response) - Log incoming response

Default Logger:

• Crest::CommonLogger - Colorized console output with timestamp, method, and URL
• Example output: crest | 2018-07-04 14:49:49 | GET | http://httpbin.org/get

Sensitive Data Filtering:

Custom Loggers: Extend Crest::Logger and override request and response methods, then pass via logger: parameter.

cURL Command Generation:

The Crest::Curlify class in src/crest/curlify.cr converts requests to cURL commands:

Available via:

• Request#to_curl instance method
• Response#to_curl delegated method
• Crest::Curlify.new(request).to_curl direct usage

Sources: README.md537-575 README.md620-651 CHANGELOG.md359

Proxy Configuration

HTTP/HTTPS proxy support via the http_proxy shard (version >= 0.14.0 per shard.yml16-18):

Proxy Parameters:

• p_addr : String - Proxy hostname or IP address
• p_port : Int32 - Proxy port number
• p_user : String? - Proxy authentication username (optional)
• p_pass : String? - Proxy authentication password (optional)

Example:

Behavior:

• Proxy configuration is per-request (no global proxy setting)
• Proxy settings persist across redirects (fixed in v0.9.6 per CHANGELOG.md417)
• Both HTTP and HTTPS requests can be proxied

Sources: README.md513-535 shard.yml16-18 CHANGELOG.md417

Exception Hierarchy

Crest defines status-code-specific exceptions in src/crest/exceptions.cr All inherit from Crest::RequestFailed.

Exception Classes

Exception Class	HTTP Status Codes	Base Class
Crest::RequestFailed	All non-success codes	Base exception
Crest::BadRequest	400	RequestFailed
Crest::Unauthorized	401	RequestFailed
Crest::Forbidden	403	RequestFailed
Crest::NotFound	404	RequestFailed
Crest::MethodNotAllowed	405	RequestFailed
Crest::RequestTimeout	408	RequestFailed
Crest::Conflict	409	RequestFailed
Crest::Gone	410	RequestFailed
Crest::UnprocessableEntity	422	RequestFailed
Crest::InternalServerError	500	RequestFailed
Crest::NotImplemented	501	RequestFailed
Crest::BadGateway	502	RequestFailed
Crest::ServiceUnavailable	503	RequestFailed
Crest::Found	302	RequestFailed
Crest::MovedPermanently	301	RequestFailed
Crest::RedirectLimitReached	-	RequestFailed

Exception Behavior

Default Behavior (handle_errors: true):

• 2xx status codes: Return Crest::Response
• 3xx status codes: Automatically follow redirects (if max_redirects > 0)
• 4xx/5xx status codes: Raise corresponding exception

Access Response from Exception:

Disable Exception Raising (handle_errors: false):

When Added: Exception hierarchy introduced in v0.9.10 (CHANGELOG.md395-396)

Sources: README.md266-317 CHANGELOG.md395-396

Dependencies and Integration

Crest integrates with several external libraries and Crystal standard library components:

External Dependencies

From shard.yml12-18:

• http-client-digest_auth (>= 0.6.0): Digest authentication support
• http_proxy (>= 0.14.0): HTTP/HTTPS proxy support

Standard Library Integration

The library builds on Crystal's standard library:

• HTTP::Client: Core HTTP protocol implementation
• HTTP::Cookie: Cookie handling
• OpenSSL::SSL::Context::Client: TLS/SSL support
• URI::Punycode: International Domain Name support (mentioned in CHANGELOG.md240)
• MIME: MIME type detection for file uploads

Sources: shard.yml12-18 src/crest.cr1-8 CHANGELOG.md240

Version and Compatibility

Current version: 1.6.1 (released 2026-02-05) as documented in CHANGELOG.md5 and shard.yml2

Crystal compatibility: Requires Crystal >= 1.0.0 as specified in shard.yml10

The library maintains a detailed changelog at CHANGELOG.md1-507 documenting breaking changes, new features, and bug fixes across 40+ releases since the initial 0.9.2 release in November 2017.

Sources: CHANGELOG.md1-507 shard.yml2-10