Cogini

Building and Testing Elixir Containers with GitHub Actions

2024-01-08T00:00:00+08:00

Here are the slides for the presentation Building and Testing Elixir Containers with GitHub Actions I gave to the Denver Elixir user's group.

It covers testing and other practical concerns when implementing microservices in Elixir.

Here is the example code:

phoenix_container_example: CI/CD system based on containerized build and test running in GitHub Actions, deploying to AWS ECS using Terraform
absinthe_federation_example: federated GraphQL with Apollo Router

Breaking up the monolith: building, testing, and deploying microservices

2024-01-02T00:00:00+08:00

I recently worked with a large e-commerce company to "break up the monolith".

They have three main applications, a web front end in Elixir/Phoenix, a large Ruby on Rails application used for internal processing, and a large Absinthe GraphQL API that glues the pieces together and integrates with third party APIs.

The primary motivation for the project was to improve reliability by decoupling the back end application from the public website. With big applications, it's easy for a change in one component to break another unrelated component. Just upgrading a library can be dangerous, but is necessary to deal with security vulnerabilities.

Another goal was to make independent components that could be developed more quickly. Finally, they wanted to improve security by compartmentalizing access and making them easier to audit.

We ended up with a plan to extract relatively-large components into services such as "users", which handled user registration, login, and settings. We needed to be able to incrementally update the system the system while keeping compatibility and ability to roll back. We used federation to break up the GraphQL backend into separate services while keeping the same public API.

Some things that are "nice to have" in monolithic systems become critical with microservices, e.g.:

Effective automated testing, including comprehensive tests which ensure that services can be updated without breaking clients
Fast, easy, and reliable deployment
Effective processes for development and QA that work with multiple components
Observability to identify and debug problems which go across components

Some specific things we implemented:

Testing against the deployed OS image, allowing OS updates to be tested against the code, particularly important when security issues are identified in base images
Static code analysis tools and security scanners, improving quality
Test results integrated into the pull request UI, providing actionable feedback to developers
Supporting development and QA across multiple services in local or review environments
Improved build performance through better caching and parallel execution, reducing cost through better efficiency and improving developer experience by reducing time spent waiting when deploying

Some challenges we faced included:

Difficulty splitting up complex GraphQL schemas and untangling application dependencies
Application configuration, particularly secret management
Difficulty assigning code ownership

Show me the code!

The rest of this post gives details and motivation for the architecture at a high level. But first, here are some running examples that show how to do it:

phoenix_container_example shows shows a CI/CD system based on containerized build and tests running in GitHub Actions, deploying to AWS using Terraform.
absinthe_federation_example shows how to test federated GraphQL applications based on Apollo Router.

See below for a description of how the code works.

Testing is critical

Testing is the most important part of microservices, particularly in a situation where the monolith has become big enough that it's causing problems.

We need to have a hierarchy of tests, going unit tests that are quick to run but fake to integration tests that test the real running system:

Unit tests with synthetic data embedded in the code or external files
Unit tests with data from a database
Unit tests for an external service with mocked APIs, i.e., not actually communicating with the service
Unit tests for an external service (e.g., Salesforce or Algolia) communicating with the service in a test environment
External tests with data from a database or external service in a test environment
External tests that combine data from multiple services, i.e., GraphQL federation.
Health checks

The core process is based on running automated tests in dev and CI, with optional manual testing by QA. We can confidently deploy code if we have good automated code coverage. We also need observability support to identify regressions in production and debug them. Additional deeper checks may be valuable after the code is released, e.g., load tests or security checks.

Unit tests are written in Elixir or Ruby their unit test frameworks. They start with synthetic data embedded in code or external files. Subsequent tests cover higher-level APIs that pull data from a database or communicate with an external service. We start by creating a mock version of the external service which returns test data. Some services provide a test sandbox or test mode for the production service, allowing us to perform integration testing against the real service. We may also be able to run services in containers for testing.

Here, external/integration tests use Postman/Newman. Postman is an interactive UI for creating and managing API tests, used by testers in development or QA environments. It includes a headless runner (Newman) for CI.

There is a trade-off between synthetic tests and integration tests. Synthetic tests run quickly and reliably but may end up not being accurate, as they are effectively only testing fake data and mocks. Integration tests exercise the full end-to-end functionality, e.g., talking to actual external services. They are more accurate but run slower and may be flaky when external services are unavailable.

Mocks and fake data

Mocks are functions that have the same interface as a library or service but return predefined data instead of calling the real service. They avoid dependencies on external services when running tests and improve test speed. Mock data may not reflect the actual behavior of the system, however. We may end up testing our mocks, not reality.

As we break up services, we need a solid test suite that validates the external API for each service, providing a contract for consumers. Whatever changes we make internally should not break other parts of the system. We can collect queries running on the current system and use them as a regression test to ensure that we are not breaking clients, even if they are somehow invalid. For example, after upgrading Absinthe GraphQL, it may start rejecting invalid queries that it accepted before.

While useful, the mocking process can result in code and configuration complexity. Where possible, we should avoid mocking in favor of using real data.

Using Tesla for the HTTP client library simplifies the mocking process. It includes a test “adapter” that removes the need to change the server URL at runtime, something that causes configuration problems.

Seed data is also needed to provide a stable base for external API tests. It provides a base of data that tests can expect to be there, e.g., users and products. Other data may be created during the tests, e.g., creating an order for a product.

A larger but more comprehensive database snapshot is also useful for dev and review environments. It is an anonymized subset of the production database, with enough realistic data to support testing. Larger databases can also be used for, e.g., overnight load testing in staging.

One performance trick is to snapshot the schema and test data as SQL, then build it into Postgres test container. This is much faster than running lots of little database migrations.

Static code analysis tools

Static analysis tools perform quality, consistency, and security checks on code. Manually written tests tend to exercise the normal execution code paths, while consistency tools find problems with less common code paths. Quality tools identify error-prone programming patterns, failure to handle error cases, security issues, and the like.

Examples for Elixir include:

Credo (code quality)
Dialyzer (type consistency checking)
mix audit (security check for Elixir packages with known issues)
Test coverage (checking what percentage of the code the tests exercise)
Sobelow (web security for Phoenix, e.g., cross-site scripting, SQL injection)
Trivy (vulnerability scanner for code and operating system). Other similar tools include Grype, Gitleaks, Snyk, GitHub Advanced Security, and SonarQube
Styler is a plugin for the Elixir formatter that fixes problems identified by Credo instead of complaining.

The results of static code analysis need to be surfaced to developers so they can take action. Best is to integrate results into the developer’s editor, showing errors inline. Next best is running in CI.

Using a tool like Hound, we can integrate test results into the PR process as comments. This avoids the "wall of text" that nobody reads unless it actually breaks the build. Developers can first fix these issues, allowing reviewers to focus on high-level concerns.

Adding quality checks to an existing project can be challenging, as it results in hundreds of issues. It can be tough getting them through a traditional PR approval process. One solution is to snapshot the results and only break on new problems.

Testing in containers

Many CI systems don't test the actual running system. They run tests in a generic Linux container, effectively just checking out the code and running mix test the same as on a developer's machine.

Instead, we should run tests in an environment that matches the target system as closely as possible, allowing us to identify problems from library incompatibilities or misconfiguration. This lets us safely upgrade the base image, e.g., in response to security vulnerabilities, and test that the code works with it.

Security vulnerabilities represent a fundamentally different workflow from development. In the normal process, we start with a PR, run it through a test and review process, and then release it when it is ready. In the security process, we start with the production code/released container, then run security scans which identify newly-discovered vulnerabilities. Those trigger a process of upgrading libraries/OS releases and testing code against them. The review process for security issues needs to be able to run quickly and safely, mostly based on automated tests, as we are vulnerable to security problems until the update is in production.

Example CI build and test process on GitHub Actions

The GitHub Actions build process runs tests against containers, including unit tests, code quality checks, external API tests, and security scans.

As part of CI, it builds two containers, test and prod. The test container has the source code and potentially other test tools. The prod image only has the minimal OS and application. The application is built using Erlang releases, which contain only the code needed for the production application, reducing size and attack surface.

The CI process first builds the two containers in parallel, then runs tests against them in parallel.

It uses a containerized version of the database which is initialized using migrations and seed data. Internal tests use a combination of ephemeral test data and persistent seed data in the database. It also includes containerized versions of other back end services such as Redis and Kafka.

External tests make calls using Newman against the service's public GraphQL API, returning results from persistent seed data in the database. These integration tests have full fidelity to the ultimate environment. In order to implement this, we need to write the API tests (or extract them from unit tests) and build seed data to support them.

For a microservices system, we can run the external tests against multiple containers at once. The absinthe_federation_example tests bring up containers for Apollo GraphQL Router, the container for the newly-updated code, as well as release containers for other services that are part of the same user scenario. We then run tests using Newman against the Apollo Router container, which routes requests to one or more service containers. Similarly, we can run tests for a website that calls a back-end API. Or we can run a headless browser testing framework to test the front end.

When we run CI for a service, we test that new code for the service works well in isolation and in combination with other services. Once these tests pass, we create a new production image for the service. We push this image to the GitHub Container Registry (GHCR) as part of testing. In the final step, we push to a production AWS ECR repository. GHCR runs internally to GitHub, improving build performance. As a result, tests always have access to the production container image for each released service as well as the new release of the code for the current service. This allows us to run integration tests against all the services as a group.

All of the above tests can run in parallel, so the total build and test time is fast (about two minutes). Tests can also be parallelized by partitioning the test run if they take a lot of time.

Security scanning

The build process runs security scanners in three phases:

During testing, run unit tests, code quality tools, and security checks on the code.

These run on the test image, which includes a checkout of the app source code. We run tools such as Trivy to check for security problems on our code and dependencies. It reads JavaScript package files to identify dependencies with known vulnerabilities.
After building the prod container, run security checks on it.

This container has only the minimum needed to run the app and has minimized versions of final JavaScript code. It runs security checks again, this time checking the OS image for known vulnerabilities and configuration problems, e.g., world-writable directories.
After release, periodically run security checks to identify newly identified vulnerabilities.

Code-level security problems should break the build. It’s probably also the best location to deal with vulnerable JavaScript libraries. These kinds of issues could be found after release, as well, so it probably makes sense to run most of this in the periodic job as well.

Here, the scan outputs results in SARIF formats. It then uploads the results to GitHub, where they appear in the Security tab. This requires a GitHub Advanced Security license, though it is free for public open source. It is straightforward to use a similar mechanism to run any other security scanning tools. Using command line tools avoids pricing models that charge per developer.

Health checks

External API tests and scenario-based production health checks are quite similar. In production, instead of simply checking for success or failure, we can run a request against the production database and ensure that we get back the data that we expect.

Developer experience

Splitting up services can improve the experience for developers, as they can work on a smaller codebase with faster, more reliable tests and shorter development cycles. They can make changes without worrying that they will break another part of the code that they are not involved with. CI tests and static code analysis make deployments more reliable.

Developers need a quick feedback loop when developing code, so they need quick tests that fail fast. We then run additional tests in CI that take more time or rely on external services. These services may be unreliable and block deployment, as we need to retry tests until they succeed.

Fast builds also reduce the impact of outages. When your tests take 30 minutes to run, the duration of outages with code fixes are all multiples of 30 minutes.

In this application, the Ruby on Rails tests took 4 hours to run on a developer's machine, so everyone relied on CI. With parallel execution, it would take about 25 minutes to run there. Flaky tests were a big problem. They would randomly fail, then succeed on the next run (or the next). We spent effort to instrument the builds to identify flaky tests and prioritize fixing them, improving developer experience and avoiding wasted time.

The downside with microservices is that developers need more dependencies up and running in order to work on a piece of code. They might need a dozen services running to be able to test their changes. Containerized development helps with that. The containerized build process means that standard, pre-built images for each service are available in GitHub, so developers do not need to build images locally.

The biggest complaint about containerized development is performance. Some recent improvements to disk I/O to Docker help. It’s should be possible to run a local database and develop code locally while talking to containerized versions of other services.

Using local Kubernetes for development makes it more consistent with the production environment. Another option is to give each developer their own equivalent environment in the cloud. They can then use one or more services locally while connecting back to the development environment, e.g., with https://www.telepresence.io.

Jobs vs Events

2024-01-01T00:00:00+08:00

Kafka is popular as a backbone messaging service to connect services. You may be used to using background job processing services like Resque in Ruby or Oban in Elixir, and wonder whether you can replace them with Kafka. In practice, both have their benefits.

This post discusses issues that come up when using Kafka and architectural patterns that can help.

Kafka architecture

Kafka is fundamentally different from Resque and other messaging systems like RabbitMQ. These systems have items that are added to queues, processed, and removed. Kafka instead uses a "topic", which acts as a log or ledger. Publishers add records to the end of the log, and they never go away. The Kafka server assigns each record an ever-increasing integer "offset" which indicates its position in the log. For practical reasons, we may purge old records from the system, e.g., to save space, but the offset keeps increasing.

Multiple independent applications can read from the same topic. Each application keeps track of which records it has processed by remembering the offset. One application might start at the beginning of the available records and read each one by one, doing some work or analysis. Another might wait for new records and perform some action.

This architecture makes Kafka great as a permanent record of events, but it doesn't specify what should be done with the records. Most importantly, it doesn't cover error handling.

For example, assume we have an application that looks for new items in the product catalog and puts them into Elasticsearch. It reads a record from Kafka, parses the data, creates a JSON document, and submits it to Elasticsearch for indexing.

Permanent failures

If the app cannot parse a data record, then it needs to log the issue and continue, otherwise, the bad record blocks further processing. The failure is “permanent”, i.e., it's fundamental to the data.

In practice, it may be that changing the code would allow the data to be handled, and we could reprocess the failed records. We also need to be able to debug problems with the system, figuring out where the data came from and how often the issues are occurring. A common strategy is to write records to a "Dead Letter Queue" (DLQ) which we can monitor, review, and reprocess.

The big difference between normal processing and DLQ processing is error handling. When processing normally, we read a record from Kafka, try to parse it, and if it fails (or there is some other permanent error like data validation), we put it in the DLQ.

When processing the DLQ, we start at a particular offset in the DLQ and try to parse records. If they succeed, we do the normal processing. Otherwise we just move on to the next record. As we process the DLQ the same way as regular records, doing it in the same application generally makes sense. We can just logically "poke" the consumer and tell it to process the DLQ. When processing, we may want to keep track of which records we have already processed and avoid processing them again. This depends on the service, however, whether processing is idempotent and whether processing records out of order matters.

One of the most significant use cases is having a bug or version skew in the consumer, causing mass failures in parsing input records. In this case, we can update the code and then reprocess the DLQ, most of which would succeed. This can result in huge numbers of messages on the DLQ. We need to monitor the number of messages and the rate of errors, then ops in case of problems.

Transient failures

Temporary problems are more common in everyday processing. It might be that the target system, e.g., Elasticsearch, is unavailable or overloaded. The application needs to retry if it can't connect or rate-limit processing. A naive sender can overwhelm a target system, "kicking it when it's down".

Kakfa is generally silent about these kinds of error-handling strategies, as each application has different requirements. The application or programming language framework needs to make its own decisions. It needs to handle retry logic, rate limiting, duplicate processing, and coordinate between multiple processes working together.

Job frameworks

Job processing frameworks such as Resque focus almost entirely on the transient processing of messages. They consider each job to be unique. They have sophisticated functions to register the job, schedule it, retry it if it fails, and troubleshoot the system.

Resque is often used to deal with the fact that Ruby doesn't do concurrency particularly well and that Rails processes are heavyweight, taking up a lot of system resources. It is useful for performance to split processing into interactive parts and asynchronous background processes.

For example, in an e-commerce system, the user creates an order, and the application responds immediately with "Thank you for your order". It then triggers multiple background jobs, e.g., sending a confirmation email, running anti-fraud checks, and starting fulfillment. Any one of these processes might fail temporarily and be retried.

Sometimes we do work in the background for practical reasons. A good example is starting a report process which might take a long time to complete. If we could reliably handle it immediately from the same process, we would. It reduces system resources, however, to trigger a background job, returning immediately and then notifying the user when the job is done. It may also provide a better user experience.

Leveraging both

In practice, job handling systems are convenient to use and mature. We can combine Kafka for "events" and Resque/Oban for "jobs". A Kafka consumer simply reads records from a topic and creates job records, which it schedules for processing. We still need to avoid overloading the job system, but otherwise, the error-handling logic is relatively simple.

Access to data

A key question is how the application accesses the data it needs to process.

One option is that the record has everything needed to perform the work. Another is that it only has a reference to the data, and the app reads the data from an external source.

For simple events, we can put everything into the job. For example, we could generate an event whenever someone fails to log into the system. It might be an account ID, email address, timestamp, and metadata like IP address and browser. An anti-fraud system could look at that and identify that we are under attack from bots.

The data could be huge, though, with a complex schema. For example, when we create a new item for sale, we probably don't want to serialize it to JSON and put it into Kafka. The processes reading the record would need to understand the format and extract the parts they care about. Evolving the schema over time is hard, and we risk getting out of sync, resulting in many parse failures (see the DLQ discussion above).

If everything is on the same system, then we just need a database ID, and the processor can read from the same database using, e.g., the same ActiveRecord models.

In a services architecture, we create services that logically own data and other applications can access the service via an API to get the data they need. For example, we might have a Customer service that handles registrations and manages information such as shipping addresses and payment methods. The service would publish an event for a new registration and another whenever it changes. The event includes the customer's unique identifier that API consumers can use to access the data. Similarly, we can have a Catalog service that manages information about items for sale.

We could use gRPC for service-to-service communication, as it gives better performance, but the schema is relatively rigid. Using a GraphQL API may make clients more resilient to schema updates.

Another option is a “change data capture” stream. Whenever the system of record changes, it sends an update with the change. For example, if a customer changes their delivery address in a Customer service, the event might include the new address as a key/value pair. Receiving systems can then handle the update directly.

A similar pattern can be used for systems that require an audit trail that indicates who made a specific change. Kafka is ideal for this, as it provides an immutable record of changes over time, as opposed to retaining only the current value.

Events

A relatively generic event schema could be as follows:

Event type, e.g., CUSTOMER_CREATED
Key/value data related to the event, e.g., customer_id
Timestamp
Source, e.g., CUSTOMER_SERVICE

As an example, the order process could create an ITEM_SOLD event, and potentially multiple consumer processes would be interested in it. So one might read the event, read the details of the item, then write a commissions table. Similarly, publishing a product could trigger indexing Elasticsearch for it.

Event sourcing

Event sourcing is an architecture where a system records a series of updates, and other systems consume those updates to keep their own state of the world.

This is particularly helpful when tracking a series of changes over time. For hundreds of years, this has been the approach used for a financial ledger. For example, a bank account tracks deposits and withdrawals over time, keeping a running balance.

Ledgers are great for financial systems but may be too much detail for inter-service communication.

It is common to use this approach in hospital information systems to synchronize between systems. When a patient registers, it creates a new patient record, and any systems that might need to interact with patients (e.g., the pharmacy) can create a corresponding record in their system. If a change or deletion occurs, the patient management system can send an update message, and the systems can update their records.

Data ownership

Generally speaking, we would prefer to have one system that owns the data and keeps a single point of truth. This system generates events when it changes.

In a services architecture, however, we might have multiple systems that generate events based on their responsibility, and we need to put the pieces together to get the whole picture.

For example, we might have a catalog that owns product items, an order management system that records sales of items, and a logistics system that records shipments of the orders. These systems generate change events that may be incidental to their internal processing, but are useful triggers for other systems.

When a customer buys a product, we might need to update the quantity on hand in an inventory management system. That might trigger the product detail page to show a different availability date.

We might also separate internal systems from the services needed to run the public website, improving public site reliability and scalability.

It can be helpful to think about the business processes as a flow of data, coordinated between multiple systems. For example, we have the internal processes associated with defining products, then we "publish" them to the website. At that point, the public catalog, order handling, and customer management systems become critical. After an order is placed, internal fulfillment and accounting systems take over.

In between, some processes need to manage product data as a whole, e.g., merchandising looks for products in the Catalog which match conditions and create a marketing campaign for them. Machine learning processes may analyze all the products in the database to determine pricing, group similar products, create suggestions, or optimize product descriptions for SEO.

Kubernetes Health Checks for Elixir Apps

2024-01-01T00:00:00+08:00

Health checks are an important part of making your application reliable and manageable in production. They can also help make development with containers faster.

Kubernetes health checks

Kubernetes has well-defined semantics for how health checks should behave, distinguishing between "startup", "liveness", and "readiness".

Liveness is the core health check. It determines whether the app is alive and able to respond to requests. It should be relatively fast, as it is called frequently, but should include checks for dependencies, e.g., whether the app can connect to a database or back-end service. If the liveness check fails for a specified period, Kubernetes kills and replaces the instance.

Startup checks whether the app has finished booting up. It is useful when the app may take significant time to start, e.g., because it loads data from a database into a cache. Separating this from liveness allows us to use different timeouts rather than making the liveness timeout long enough to support startup. Once startup has completed successfully, Kubernetes does not call it again, it uses the liveness check.

Readiness checks whether the app should receive requests. Kubernetes uses it to decide whether to route traffic to the instance. If the readiness probe fails, Kubernetes doesn't kill and restart the container. Instead it marks the pod as "unready" and stops sending traffic to it, e.g., in the ingress. It is useful to be able to temporarily stop serving traffic, e.g., when the instance is overloaded or it has transient problems connecting to a back-end service.

Kubernetes checks themselves rely only on the HTTP response code to determine service health. A code greater than or equal to 200 and less than 400 indicates success, and any other code indicates failure. While Kubernetes treats the health check response as binary, i.e., ok or not, the health check can return additional information about the cause of the error, making troubleshooting easier for developers or ops staff. This might be formatted as a simple string or JSON format, e.g. {"status": "OK"} / {"status": "error", "code": 503, "reason": "timeout connecting to downstream service"}.

In addition, the service should generally write a message to the log or add information to a trace, allowing people to find and debug systems that are having problems.

The kubernetes_health_check project provides a Plug that handles Kubernetes health check requests. It is driven by a module that is custom to the app. Following is an example:

defmodule Example.Health do
  @moduledoc """
  Collect app status for Kubernetes health checks.
  """
  alias Example.Repo

  @app :example
  @repos Application.compile_env(@app, :ecto_repos) || []

  @doc """
  Check if the app has finished booting up.

  This returns app status for the Kubernetes `startupProbe`.
  Kubernetes checks this probe repeatedly until it returns a successful
  response. After that Kubernetes switches to executing the other two probes.
  If the app fails to successfully start before the `failureThreshold` time is
  reached, Kubernetes kills the container and restarts it.

  For example, this check might return OK when the app has started the
  web-server, connected to a DB, connected to external services, and performed
  initial setup tasks such as loading a large cache.
  """
  @spec startup ::
          :ok
          | {:error, {status_code :: non_neg_integer(), reason :: binary()}}
          | {:error, reason :: binary()}
  def startup do
    # Return error if there are available migrations which have not been executed.
    # This supports deployment to AWS ECS using the following strategy:
    # https://engineering.instawork.com/elegant-database-migrations-on-ecs-74f3487da99f
    #
    # By default Elixir migrations lock the database migration table, so they
    # will only run from a single instance.
    migrations =
      @repos
      |> Enum.map(&Ecto.Migrator.migrations/1)
      |> List.flatten()

    if Enum.empty?(migrations) do
      liveness()
    else
      {:error, "Database not migrated"}
    end
  end

  @doc """
  Check if the app is alive and working properly.

  This returns app status for the Kubernetes `livenessProbe`.
  Kubernetes continuously checks if the app is alive and working as expected.
  If it crashes or becomes unresponsive for a specified period of time,
  Kubernetes kills and replaces the container.

  This check should be lightweight, only determining if the server is
  responding to requests and can connect to the DB.
  """
  @spec liveness ::
          :ok
          | {:error, {status_code :: non_neg_integer(), reason :: binary()}}
          | {:error, reason :: binary()}
  def liveness do
    case Ecto.Adapters.SQL.query(Repo, "SELECT 1") do
      {:ok, %{num_rows: 1, rows: [[1]]}} ->
        :ok

      {:error, reason} ->
        {:error, inspect(reason)}
    end
  rescue
    e ->
      {:error, inspect(e)}
  end

  @doc """
  Check if app should be serving public traffic.

  This returns app status for the Kubernetes `readinessProbe`.
  Kubernetes continuously checks if the app should serve traffic. If the
  readiness probe fails, Kubernetes doesn't kill and restart the container,
  instead it marks the pod as "unready" and stops sending traffic to it, e.g.,
  in the ingress.

  This is useful to temporarily stop serving requests. For example, if the app
  gets a timeout connecting to a back end service, it might return an error for
  the readiness probe. After multiple failed attempts, it would switch to
  returning false for the `livenessProbe`, triggering a restart.

  Similarly, the app might return an error if it is overloaded, shedding
  traffic until it has caught up.
  """
  @spec readiness ::
          :ok
          | {:error, {status_code :: non_neg_integer(), reason :: binary()}}
          | {:error, reason :: binary()}
  def readiness do
    liveness()
  end

  @spec basic ::
          :ok
  # | {:error, {status_code :: non_neg_integer(), reason :: binary()}}
  # | {:error, reason :: binary()}
  def basic do
    :ok
  end
end

Dependencies

Services also need health checks for the services they depend on. In development, these might be databases or Kafka running in a container. In production, those might be managed services in AWS.

For services that do not provide an HTTP API, we can define a command that runs within the container.

For example, this probe checks a Postgres database container:

readinessProbe:
  exec:
    command: ["pg_isready"]
  initialDelaySeconds: 10
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 20

livenessProbe:
  exec:
    command: ["psql", "-w", "-U", "postgres", "-d", "my-db", "-c", "SELECT 1"]
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 1

For historical reasons, Kubernetes checks are different from Docker healthcheck definitions.

In docker-compose.yml, health checks look like:

---
version: "3.9"
services:
  deploy:
    image: example-service
    healthcheck:
      test: ["CMD", "curl", "http://127.0.0.1:4001/healthz"]
      start_period: 6s
      interval: 2s
      timeout: 5s
      retries: 20
    depends_on:
      postgres:
        condition: service_healthy

postgres:
  image: postgres:14.1-alpine
    restart: always
    healthcheck:
      test: ["CMD-SHELL", "pg_isready"]
      start_period: 5s
      interval: 2s
      timeout: 5s
      retries: 20

router:
    image: ghcr.io/apollographql/router:v1.2.1
    ports:
      # GraphQL endpoint
      - "4000:4000"
      # Health check
      - "8088:8088"
    environment:
      # https://www.apollographql.com/docs/router/configuration/overview
      APOLLO_ROUTER_LOG: "debug"
      APOLLO_ROUTER_SUPERGRAPH_PATH: /dist/schema/local.graphql
      APOLLO_ROUTER_CONFIG_PATH: /router.yaml
      APOLLO_ROUTER_HOT_RELOAD: "true"
    volumes:
      - "./apollo-router.yml:/router.yaml"
      - "./supergraph.graphql:/dist/schema/local.graphql"
    healthcheck:
      test: ["CMD-SHELL", "curl", "-v", "--fail", "http://127.0.0.1:8088/health"]
      start_period: 5s
      interval: 2s
      timeout: 5s
      retries: 20
    depends_on:
      deploy:
        condition: service_healthy

With containerized tests, we might run tests via docker-compose. The external API tests bring up the app container, containers for other services that it depends on, associated database containers, and the Apollo Router container. They then run tests using Postman/Newman. Then we can run docker-compose up router to bring up all the containers, waiting until they are up and healthy, and then run Newman tests on it.

Robust health checks for each component in the stack help the system to come up quickly and reliably, and they provide messages that help us debug startup failures easily.

Running OS commands

Instead of external HTTP checks, we can instead execute a health check inside the container. That may use curl to call the app on localhost. While this does not exercise the HTTP stack, it may be more reliable.

An example Kubernetes check which runs a command is

livenessProbe:
  exec:
    command:
    - /app/grpc-health-probe
    - -addr=:50051
    - -connect-timeout=5s
    - -rpc-timeout=5s
  failureThreshold: 3
  initialDelaySeconds: 60
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 10
readinessProbe:
  exec:
    command:
    - /app/grpc-health-probe
    - -addr=:50051
    - -connect-timeout=5s
    - -rpc-timeout=5s
  failureThreshold: 3
  initialDelaySeconds: 1
  periodSeconds: 10
  successThreshold: 1
  timeoutSeconds: 10

When an Elixir app is running via releases, we can evaluate code directly to run the health check, e.g.:

healthcheck:
  test: ["CMD", "bin/api", "eval", "API.Health.liveness()"]
  start_period: 2s
  interval: 1s
  timeout: 1s

Higher-level checks

The above health checks are used at the infrastructure level to identify problems. They help Kubernetes automatically resolve problems by restarting containers, scaling resources, etc.

We can also add production checks that indicate end-customer visible problems. For example, if a user can’t log in, then we should alert. Some of these can be metrics, e.g., if we would normally get 100 successful logins a minute, and now we are getting 0, then there is a problem. Or we might get 1% unsuccessful logins in a period, and now we are getting 50%.

We use external API tests as part of containerized testing in CI. We can leverage these tests to make production health checks for standard scenarios across multiple services, e.g., a customer logs in, adds an item to their cart, and then checks out.

See the following articles for more background information:

Presentation on thinking functionally in Elixir 2020

2020-12-15T00:00:00+08:00

Here are the slides for the presentation on thinking functionally in Elixir I gave to the local Elixir user's group.

Choosing a Linux distribution

2020-01-13T00:00:00+08:00

Short answer: Use Ubuntu LTS

Long answer

There are two main families, RedHat and Debian. RedHat traditionally comes from the corporate world, and Debian from the free software community. I have been using Linux since 1993, so I will give a bit of a history lesson to explain the motivation behind the popular distributions.

RedHat was one of the first major commercial Linux distributions, and is the most successful. They started by selling CDs which you could install on as many servers as you like, hoping that larger customers would buy support contracts. That didn't work, so they switched to a per-server licensing model. When that happened, volunteers took the source code from RedHat Enterprise Linux and made their own releases. Because they couldn't use the RedHat trademark, they called it something else. There was always a bit of a delay with releases, but it worked fine.

There are a couple of distributions like this, with more or less value add. CentOS is the most pure. A few years ago, RedHat bought CentOS, so it's now an official part of RedHat, formalizing the model and giving them more resources. RedHat also has a "bleeding edge" free distribution called Fedora. It's popular with enthusiasts to run on the desktop, but is not commonly run on servers.

Debian is one of the first non-commercial Linux distributions, and is the most popular. Traditionally many of the people who actually write open source packages ran Debian, and it was used by expert users who ran e.g. internet service providers. It is generally high quality, and has the most software packages. At one point, most of the core users were running a "rolling update" version of Debian. Since it is run by volunteers, and there was nobody who cared that much, it once went more than two years without a formal release. Ubuntu was started by a .com millionaire who wanted to give back to the open source community. Ubuntu was basically "Debian with regular releases."

RedHat focused on the kernel and making stable, supported releases for enterprise customers to run on their servers. They traditionally hire many of the core Linux kernel developers. Ubuntu focused on building a good Linux desktop experience. They offer predictable releases and commercial support, and work with partners to certify Debian.

With this background, you can see a bit of the strengths and weaknesses of the different distributions. Here is the current situation:

The majority of enterprise servers run CentOS. If an enterprise needs commercial support, then they pay for RedHat. Many commercial software packages like Oracle were certified and supported on RedHat. Most dedicated servers run CentOS by default, and it may be the only distribution a host supports. Oracle created their own version of RedHat, competing with RedHat for support contracts. Amazon Linux is based on RedHat. The specific differences between Amazon Linux and RedHat are unclear, and it's a bit of a moving target. You can't practically run Amazon Linux outside of Amazon.

Ubuntu made deals with various partners like Amazon. It is a first class supported distribution on AWS, and is very popular there. Ubuntu has largely abandoned its desktop ambitions and is focused on the server side. Rather than try to do proprietary development, it focuses on packaging upstream software in a way that works well.

Recommendations

I have personally run both distributions for many years, and my preferences have shifted back and forth over time. We manage hundreds of virtual and dedicated servers, and we are currently about 50% CentOS and 50% Ubuntu.

Until recently, my preference was CentOS 7, for these reasons:

It is designed for servers, and has a long term support model so I don't have to upgrade frequently
It doesn't cost money, but RedHat has a sustainable business model behind it
RedHat employs many of the core kernel developers, making it solid from the bottom up. Ubuntu has traditionally been weaker at kernel support (and I have the scars from it)
I can run it everywhere, on dedicated servers, cloud instances and in local dev environments
Since Amazon Linux is based on RedHat, their software agents work well with it

CentOS has some disadvantages relative to Debian. It has fewer OS packages and relatively old versions. This is the downside of stability. RedHat supports fewer software packages, with fewer options. You may need to find third party repositories to get software or build your own. It's rare, though, as the packages you need for server use are generally there.

Over the long support life of the distro, packages can get pretty old. For most packages, e.g. mail server software, it doesn't matter. For things like the database, you may want newer features. Major projects like PostgreSQL have their own supported packages, which lets us stay up to date. We generally stick to the EPEL repo. You can often download random packages off the net, but it feels a bit questionable sometimes. Why run a "supported" enterprise distro then rely on packages from some random dude?

The long term support for a server is much more important in a bare-metal world, where upgrading can be traumatic. You want to be able to regularly install security packages without worrying that it will take down the server.

In the cloud, however, it's easy for us to set up a new instance with the latest OS, verify that it works, then switch. For that, I prefer Ubuntu.

It's a solid, well supported distro. The community is more friendly for beginners, partly coming from the community nature of Debian, partly from the desktop focus. It has access to all the Debian packages, with a commercial model and regular releases. Packages are generally more up to date than CentOS, and tend to have a more direct line from upstream projects.

We normally use Packer and Ansible to build an AMI specifically for an application. When we need to update the AMI, we can run it through the same CI/CD process that we use for the app, running tests and then deploying. If there is a problem, we can roll it back, same way we would with an application issue.

This works well for applications which are under continuous development. We can run the LTS (Long Term Support) versions of Ubuntu or more frequent releases. They are supported for long enough to keep things stable, but we get access to the latest versions of software.

Ubuntu is well supported in cloud environments. We generally run Minimal Ubuntu, which gives us a small install while still staying compatible with other software.

I prefer to run the community AMIs instead of marketplace AMIs, as it avoids licencing weirdness keeping me from being able to recover a machine when it won't boot.

The future

Long term, the business model of selling Linux is questionable. Cloud providers want to "commoditize the common good", i.e. they sell you machine hours, so they want software to be free. Instead of running e.g. Oracle on RedHat, they would prefer us to use RDS.

RedHat recently sold to IBM. I would not be surprised if Ubuntu sells to Microsoft next. We will be left with a community support model, and Debian is the best working example of that right now. The community can occasionally be dysfunctional, driven by politics, the "United Nations" of free software. It works, though, and I expect it to continue.

The future is containers, where we run only the minimum parts of the operating system necessary to support a specific application. There are specialized distros like Alpine, but I still prefer to run Minimal Ubuntu. It's reasonably small (about 30 MB), but compatible with regular Ubuntu, making development and testing easier.

Deploying an Elixir app to Digital Ocean with mix_deploy

2020-01-13T00:00:00+08:00

This is a gentle introduction to getting your Elixir / Phoenix app up and running on a server at Digital Ocean (affiliate link). It starts from zero, assuming minimal experience with servers.

It builds the app on the server, then uses Erlang releases to run the code under systemd. It uses the mix_deploy library to handle deployment tasks.

Digital Ocean's smallest $5/month plan runs Elixir great. This guide uses their managed databases service so you don't need to manage the database.

We will be using a boilerplate Phoenix project with PostgreSQL database. It assumes you are running macOS on your dev machine and Ubuntu 18.04 on the server.

These instructions are based on this working example application and the principles described in the blog post "Best practices for deploying Elixir apps".

This post includes basic instructions to prepare your existing Elixir/Phoenix application for deployment using mix_deploy. See preparing an existing project for deployment for more details.

If you have any questions, contact me on the Elixir Slack at jakemorrison or open an issue on GitHub.

Overall approach

Create the server
Configure ssh
Configure the build / deploy user
Check out code on the server from git and build a release
Deploy the release

You can first get the template running, then prepare your own project for deployment.

NOTE: This guide works with Ubuntu 18.04, CentOS 7, Ubuntu 16.04, and Debian 9.4. If you are not sure which distro to use, choose Ubuntu 18.04. The approach here works for dedicated servers and cloud instances as well.

The actual work of building and deploying releases is handled by simple shell scripts which you run on the build server or from your dev machine via ssh, e.g.:

ssh -A deploy@web-server
cd build/mix-deploy-example
git pull

# Build release
bin/build

# Extract release to target directory on local machine, creating current symlink
bin/deploy-release

# Run database migrations
bin/deploy-migrate

# Restart the systemd unit
sudo bin/deploy-restart

Create the server

Go to Digital Ocean (affiliate link) and create a Droplet (virtual server).

Choose an image: Choose Ubuntu 18.04
Choose a plan: Standard is fine
Choose a size: The smallest, $5/month Droplet is fine
Choose a datacenter region: Select a data center near you
Add your SSH keys: Select the "New SSH Key" button, and paste the contents of your ~/.ssh/id_rsa.pub file. Create an ssh key, if you don't have one already. On Mac OS, you can copy your SSH key to clipboard by running cat ~/.ssh/id_rsa.pub | pbcopy.
Choose a hostname: The default name is fine, but awkward to remember and type. Use "web-server" or whatever you like

The defaults for everything else are fine. Click the "Create" button.

Configure ssh to talk to your server

Note the IP address of your new droplet in the Digital Ocean UI.

Configure ~/.ssh/config on your local dev machine so you can connect to the server.

# Change the IP address below to the actual IP address of your Droplet
Host web-server
  HostName 123.45.67.89

Create a deploy user on the web server

For security, we use two operating system user accounts, the deploy user to build and deploy the app, and the app user to run the app.

Connect to the web server as root:

ssh root@web-server

Create the deploy user:

useradd -m -s /bin/bash deploy

Configure sudo to allow the deploy user run commands as root without a password:

echo "deploy ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/10-app-deploy

There are more sophisticated ways to manage users. We normally manage user accounts with Ansible.

Configure ssh access to the `deploy` user.

Create the .ssh directory and set permissions:

mkdir -p ~deploy/.ssh
chown deploy:deploy ~deploy/.ssh
chmod 700 ~deploy/.ssh

Allow the ssh key you set for the droplet root user to log into the deploy user account:

cp ~root/.ssh/authorized_keys ~deploy/.ssh
chown deploy:deploy ~deploy/.ssh/authorized_keys
chmod 600 ~deploy/.ssh/authorized_keys

Exit the ssh session and connect again using the deploy user.

ssh -A deploy@web-server

If it doesn't work, check that the ssh key on your dev machine (.ssh/id_rsa.pub) is in the ~/.ssh/authorized_keys file for the deploy user and check the file permissions. Try with -vv or look at /var/log/auth.log on the server.

The -A flag on the ssh command gives the session on the server access to your local ssh keys without copying them to the server. If your local user can access a GitHub repo, then you can do it on the server.

Make sure that the deploy user can run commands with sudo:

sudo -s
exit

Check out the app source

As the deploy user on the build machine, create the build dir:

mkdir -p ~/build

Check out the app source:

cd ~/build
git clone https://github.com/cogini/mix-deploy-example # or your app repo
cd mix-deploy-example

Install build dependencies

In order to build the app, we need to install Erlang, Elixir and Node.js on the build server. Run the following script to install Erlang, Elixir and Node.js from OS packages:

sudo bin/build-install-deps-ubuntu

See the instructions on the Elixir website for more details on installing Elixir and dependencies.

We generally use ASDF to manage build tools. That allows us to precisely specify versions and install multiple versions at once.

Create the database

Most apps use a database. You can install the database on the same Droplet as you run your app. It works fine and is cheaper, but then you have to manage the db. This guide uses Digital Ocean's managed databases service.

In the Digital Ocean UI, select Create → Databases.

Choose a database engine: Select PostgreSQL 11
Choose a cluster configuration: $15/month is fine
Choose a datacenter: Use the same data center as your droplet
Choose a unique database cluster name: The default name is fine

While the database is being created, in the “Getting started” section of the page, click the bullet point that says “Secure this database cluster.” Under “Restrict inbound connections” select your droplet and click “Allow these inbound sources only.” This ensures that only your application server can connect to the database.

Create app databases and users

If you are creating a database per app, you can use the defaultdb database and doadmin user that the setup wizard created for you. However, it is better to create a separate database and database user for each app environment.

Configuration

There are four kinds of things that we may want to configure:

Static data, e.g. file paths. This is the same for all machines.
Settings specific to the environment, e.g. the hostname of the db server.
Secrets such as db passwords, API keys or TLS certificates.
Dynamic attributes such as the IP addresses of the server or other machines in the cluster.

In a simple deployment, you can put all the configuration in config files like config/prod.exs. It will then be compiled into the release package.

When building on a different machine, e.g. a CI/CD system, it's more secure to keep secrets separate from the release and load them at runtime. Similarly, we may want to build a release and run it in a test environment, then deploy it in production.

In these cases, we keep the config outside the release file and load it at runtime. We might read it from environment variables or external config files. Or we might read it from a system like AWS Systems Manager Parameter Store. See "Best practices for deploying Elixir apps" for more details.

Building

On your build machine, build the app by running the build script:

bin/build

In addition to the normal Phoenix build steps, this command sets up the deploy scripts by running the following mix_systemd and mix_deploy commands:

# NOTE: run by bin/build
mix systemd.init
mix systemd.generate

mix deploy.init
mix deploy.generate

The configuration is minimal. We just change the name of the OS user that the app runs under to app in config/prod.exs:

config :mix_systemd,
  # Run db migrations before starting the app
  # exec_start_pre: [
  #   [:deploy_dir, "/bin/deploy-migrate"]
  # ],
  app_user: "app",
  app_group: "app"

config :mix_deploy,
  # Generate runtime scripts from templates
  templates: [
    # Systemd wrappers
    "start",
    "stop",
    "restart",
    "enable",

    # System setup
    "create-users",
    "create-dirs",
    "set-perms",

    # Local deploy
    "init-local",
    "copy-files",
    "release",
    "rollback",

    # DB migrations
    "migrate"
  ],
  app_user: "app",
  app_group: "app"

Prepare runtime configuration

Edit config/prod.exs with the runtime settings:

import Config

config :mix_deploy_example, MixDeployExample.Repo,
  username: "doadmin",
  password: "CHANGEME",
  database: "defaultdb",
  hostname: "db-postgresql-sfo2-xxx-do-user-yyy-0.db.ondigitalocean.com",
  port: 25060,
  ssl: true,
  pool_size: 15

config :mix_deploy_example, MixDeployExampleWeb.Endpoint,
  secret_key_base: "CHANGEME2"

You can generate a unique value for secret_key_base using this command:

mix phx.gen.secret

Build

Build the app and make a release:

bin/build

Initialize local system

Run this once to set up the system for the app, creating users and directories for releases, runtime configuration, etc.:

sudo bin/deploy-init-local

As this script changes group membership, you should log out and in again to reload user privileges.

Deploy the app

Deploy the release to the local machine:

# Extract release to target directory, creating current symlink
bin/deploy-release

# Run database migrations
bin/deploy-migrate

# Restart the systemd unit
sudo bin/deploy-restart

Check that it works

Make a request to the server:

curl -v http://localhost:4000/

You can get a console on the running release:

sudo -i -u app /srv/mix-deploy-example/bin/deploy-remote-console

You can also have a look at the logs:

sudo systemctl status mix-deploy-example
sudo journalctl -r -u mix-deploy-example

You can roll back the release with the following:

bin/deploy-rollback
sudo bin/deploy-restart

Configure the server to listen on port 80

Listening on port 4000 might be fine if it's behind a load balancer, otherwise we need to make the app available on port 80. There are two ways to do this:

Port forwarding using iptables (see Port forwarding with iptables)
Reverse proxy using a web server, (see Serving your Phoenix app with Nginx)

After you complete this step, you should be able to access your website in the browser by navigating to your droplet's public IP address.

SSL

The steps necessary to get SSL set up with your Phoenix application depend on the approach that you took in the previous step. If you are forwarding ports using iptables, then you should set up SSL in your application's endpoint, as described in Phoenix docs. You can get an SSL certificate for free from Let's Encrypt.

If you are running behind an Nginx reverse proxy, you should instead set up SSL in Nginx. The necessary steps are described in Digital Ocean's tutorials.

How to prepare your Phoenix app for deployment

Following are the steps used to set up this repo. You can do the same to add it to your own project. This repo is built as a series of git commits, so you can see how it works step by step.

Generate Phoenix project

mix phx.new your_app
mix deps.get
cd assets && npm install && node node_modules/webpack/bin/webpack.js --mode development

Add mix.lock to git
Add package-lock.json to git

Configure releases

Elixir 1.9 has built in support for creating releases. For earlier versions, use the Distillery library.

Generate initial config files in the rel dir:

mix release.init

Check the rel directory into git.

In mix.exs, tell Mix not to include Windows executables in releases. In the main project configuration, add the option :releases:

def project do
  [
    app: :mix_deploy_example,
    version: "0.1.0",
    elixir: "~> 1.9",
    elixirc_paths: elixirc_paths(Mix.env()),
    compilers: [:phoenix, :gettext] ++ Mix.compilers(),
    start_permanent: Mix.env() == :prod,
    aliases: aliases(),
    deps: deps(),

    # add this line:
    releases: releases()
  ]
end

Then, in the same file, add a private function that returns your project's release configuration:

defp releases do
  [
    # change this to your application name
    mix_deploy_example: [
      include_executables_for: [:unix]
    ]
  ]
end

Tune the OS for performance

For optimal performance, it is recommended that you increase the default limits for open TCP ports and file handles. Follow the instructions in the post Tuning TCP ports for your Elixir app.

Add runtime config files

Loading runtime configuration from Elixir source files in Elixir 1.9 is very straightforward. The file config/releases.exs is copied into your release and evaluated at startup, and you can load your Elixir configuration file from within that file. Edit config/releases.exs:

import Config
import_config "/etc/mix-deploy-example/config.exs"

Create a config/prod.secret.exs.sample that you can use to generate production configuration files on the build server:

import Config

# Change these identifiers to ones specific to your application
config :mix_deploy_example, MixDeployExample.Repo,
  username: "CHANGEME",
  password: "CHANGEME",
  database: "CHANGEME",
  hostname: "CHANGEME",
  port: 5432,
  ssl: true,
  pool_size: 15

config :mix_deploy_example, MixDeployExampleWeb.Endpoint,
  secret_key_base: "CHANGEME2"

Add migrator module (optional)

In order for the bin/deploy-migrate script to work properly, you need to add a migrator module to your project. The instructions on how to do so are described in the post Running Ecto migrations in a release.

If your build server and production server are the same machine, you can also skip this step and just run your migrations with MIX_ENV=prod mix ecto.migrate.

Set up ASDF

Create a .tool-versions file in the root of your project, describing the versions of OTP, Elixir, and Node that you will be building with:

erlang 21.3
elixir 1.9.0
nodejs 10.16.0

Install mix_deploy and mix_systemd

Add libraries to deps from Hex:

{:mix_systemd, "~> 0.5.0"},
{:mix_deploy, "~> 0.5.0"}

Or from GitHub:

{:mix_systemd, github: "cogini/mix_systemd", override: true},
{:mix_deploy, github: "cogini/mix_deploy"},
end

Add rel/templates and bin/deploy-* to .gitignore:

echo '/rel/templates' >> .gitignore
echo '/bin/deploy-*' >> .gitignore

Copy build and utility scripts into your repo

Copy shell scripts from the bin/ directory of the mix-deploy-example repo to the bin/ directory of your project.

These scripts build your release or install the required dependencies:

build
build-install-asdf
build-install-asdf-deps-centos
build-install-asdf-deps-ubuntu
build-install-asdf-init
build-install-asdf-macos
build-install-deps-centos
build-install-deps-ubuntu

This script verifies that your application is running correctly:

validate-service

Check these scripts into git.

Configure for running in a release

In config/prod.exs, uncomment or add this line so that Phoenix can run correctly in a release:

config :phoenix, :serve_endpoints, true

In the same file, configure mix_deploy and mix_systemd to run your application as the app user. This step is mandatory:

config :mix_deploy,
  app_user: "app",
  app_group: "app"

# Minimal
config :mix_systemd,
  app_user: "app",
  app_group: "app"

Still in prod.exs, configure your application's endpoint to fetch port number from environment variables. The corresponding variable will be set by systemd:

config :your_app_name, YourAppNameWeb.Endpoint,
  http: [:inet6, port: System.get_env("PORT") || 4000],
  # ...

Confirm that everything compiles by building the app:

mix deps.get
mix deps.compile
mix compile

You should be able to run the app locally with:

# Create development database
mix ecto.create

# Compile assets with production settings
(cd assets && npm install && npm run deploy)

mix phx.server
open http://localhost:4000/

If everything seems to work, you can proceed with deployment just like you did with the mix-deploy-example sample application.

Deploying complex apps to AWS with Terraform, Ansible, and Packer

2020-01-11T00:00:00+08:00

Recently we helped a client migrate a set of complex Ruby on Rails applications to AWS, deploying across multiple environments and regions.

They have a half-dozen SaaS products which they have built over the last decade. They had been running them on a set of shared physical servers, with lots of weird undocumented relationships between components.

There were problems with reliability and performance, as well as overall complexity. They were deploying using Capistrano to push releases directly to the production servers, so they didn't have an enforced release process between dev, staging and production environments.

The big driver for improvement, however, was the need to separate customer data by country/region. GDPR compliance is easier if they keep European data hosted in Europe. Customers in China were also complaining about poor network performance crossing the "Great Firewall".

They needed multiple environments for each app: dev, staging, production in US/Canada/EU/China, plus a "demo" environment where customers can try out the app in a sandbox.

They needed a controlled release process with automated tests and ability to roll back in case of problems. They had heavy load during certain parts of the year, and underutilization the rest of the time, so they needed autoscaling. They needed robust monitoring, metrics and alerting.

The solution

They had tried to containerize their apps, but after months of poor progress, they gave up. It was too disruptive to their development team. They had to make a lot of changes at once, and everyone was having to become deployment experts. We came in and designed a new system which required minimum changes to their apps and workflow while solving their deployment issues.

One of the most important design considerations was handling all their apps and environments with a common framework. If there are too many special cases, the system becomes unmanageable. It's a false economy to optimize one environment with custom code, but increase the cost of managing the overall system. It is better to have a standard template with configuration options. On the other hand, we can't have too much configurability, it must be "opinionated".

The apps are all similar, following the standard structure for large Rails apps: front end web, background job processing, Redis or Memcached for caching, Elasticsearch, MySQL or PostgreSQL. They need to run slightly differently in dev, staging, demo and multiple production environments. AWS China in particular has many differences from standard AWS due to missing services and lack of encryption.

They needed an automated CI/CD pipeline, running unit tests before release, standard deployment and rollback, production monitoring and centralized logging.

With a good pipeline, developers mostly don't care about deploy time, it happens in the background. Time to get a change deployed can become very important, however, when dealing with production problems. If each iteration takes a half hour to deploy, you are going to make a bad day even worse.

Running in China, with a very slow connection to the outside world, means that we have to cache things like gems and OS packages locally, rather than downloading them every time from the network. Otherwise it can take ages to build an AMI.

Architecture

Multiple AWS accounts

Running multiple apps in the same environment requires additional layers of abstraction and configuration in your automation, making things more complex.

AWS accounts are free, so we use AWS Organizations to set up a master AWS account for billing, then an AWS account per environment (dev, staging, prod). Next we create a VPC per app in each environment. That removes layers, making the deployment scripts simple and consistent between the different environments.

If we have the same ops team handling prod for multiple apps, then we can put all the prod VPCs in the same AWS account. For extra security, we can easily make a separate AWS account per app + environment. The scripts stay the same, we just change the AWS_PROFILE to point to the right place.

Sharing resources between accounts

The downside to having multiple AWS accounts is that we need permissions to share resources across accounts. While we can do this with IAM, it may be better to duplicate work to improve security, reduce coupling, and keep the configuration consistent.

For example, if we are managing the DNS domain for the app in Route53 in the prod account, then scripts in dev need cross-account permissions to create host entries. Instead, we can use a different domain per environment, e.g. example.com for production, example-dev.com for development. This is more secure and keeps mistakes from affecting production. It also makes it easy to use consistent subdomains for e.g. api.example.com or per-customer subdomains.

Similarly, we could build an AMI in one environment and use it everywhere, but it may be better to just build it once per env. There are definitely good reasons for having immutable artifacts, running exactly the same AMI in QA and prod, but saving resources is not the main motivation.

Structure

Following is a standard AWS structure:

The app runs in a Virtual Private Cloud (VPC) across multiple Availability Zones (AZs) for high availability. The VPC is split into two networks, public and private.

All the data is in the private network, with controlled access. All traffic from the outside world goes through the Application Load Balancer (ALB), which proxies HTTP requests to the application instances.

The app runs in an Auto Scaling Group (ASG), which dynamically changes the number of running instances according to load. It also easily ensures high availability, because it will automatically start instances in different data centers (AZs) in case of problems.

We use the AWS Relational Database Service (RDS) service for the database, which handles high availability across multiple AZs. Similarly, we can use Elasticsearch for full text search and Elasticache for caching. Simple Storage Service (S3) stores application file data.

Bastion hosts allow users to remotely access instances in the private subnet. Users connect via ssh to the bastion, which forwards the connection to back end servers. This function can now be handled better by Amazon's new AWS Systems Manager Session Manager service.

Similarly, the NAT GW allows application servers in the private network to connect to services on the public network, e.g. partner APIs.

DevOps servers are EC2 instances used for deployment and management functions within the VPC, e.g. running Jenkins CI or ops tools like Ansible. In this case, we used the DevOps server to help ease the transition to the cloud, handling deployment processes. As we automated the application deployment, we switched to AWS CodeBuild.

Meeting legacy apps half way

Ideally, modern cloud applications do not store data on the app server. They keep all application data in a separate shared location, either RDS database or S3 file store. This allows us to automatically start and stop multiple instances according to load or as part of the deployment process.

Making complex legacy apps do this can be a lot of work, though. For this client, we used EFS to allow apps to share temporary files between servers. One server can write a file and another server can read it back on a subsequent request. This allows the application to use temp files as if it was only running on one server, without needing to make user sessions "sticky" to one server.

Deploying with CodeDeploy and Capistrano

The client originally deployed using Capistrano, pushing code directly to production systems via ssh. That doesn't work when there are multiple servers in an ASG, though, as we would need to push to all of them. It's also fragile and uncoordinated, so we risk having the system in an inconsistent state during a deploy or rollback.

The continuous integration / continuous deployment (CI/CD) system automates the process of building and deploying code. It watches the code repository for changes, runs tests, builds releases and automatically deploys them to production. It monitors the success of the deployment, rolling it back in case of problems.

In this case, just getting the app building in CI/CD was a big task, so we first implemented a hybrid system. Developers continued to use Capistrano to deploy the app using cap deploy. Instead of deploying directly to the prod server, however, they pushed code to a DevOps server. At the last step, custom rake tasks packaged the code and turned it into a CodeDeploy release, which we deployed into production.

Blue / Green deployment makes systems more reliable by taking advantage of how easy it is to start temporary servers in the cloud. Instead of updating code on existing servers, we can start new servers and deploy new code to them. Once we are sure it works, we switch traffic to the new servers and shut down the old ones.

Encryption

Applications with sensitive user data such as health care and financial services require high security, and encryption has become a baseline requirement for all applications. In this design, we used encryption everywhere: all data at rest is encrypted, and all data in transit is encrypted. That means turning on encryption for S3, RDS and EBS disk volumes, and using SSL on the ALB for external traffic, between the ALB and the app, and between the app and RDS and Elasticsearch.

Show me the code!

Here is a complete example of deploying an app to AWS.

It supports multiple components, e.g. web front end, background job handler, periodic jobs, a separate server to handle API traffic or web sockets connections. It uses RDS for database, Redis or Memcached, Elasticsearch, CDN for static assets, SSL, S3 buckets, encryption.

The app can run in an autoscaling group and use a CI/CD pipeline to handle blue/green deployment. It supports multiple environments: dev, staging, prod, demo, with slight differences for each.

It's built in in a modular way using Terraform, Ansible and Packer. We have used it to deploy multiple complex apps, so it handles many things that you will need, but it's also flexible enough to be tweaked when necessary for special requirements. It represents months of work.

These modules cover the following scenarios:

Minimal EC2 + RDS

VPC with public, private and database subnets
App runs in EC2 instance(s) in the public subnet
RDS database
Route53 DNS with health checks directs traffic to app
Data stored in S3

This is good for a simple app, and is also a stepping stone when deploying more complex apps. The EC2 instance can be used for development or as a canary instance.

CloudFront for assets

Store app assets like JS and CSS in CloudFront for performance

CodePipeline for CI/CD

Whenever code changes, pull from git, build in CodeBuild, run tests and deploy automatically using CodeDeploy. Run tests against resources such as RDS or Redis.

Auto Scaling Group and Load Balancer

App runs in an ASG in the private VPC subnet
Blue/Green deployment
SSL using Amazon Certificate Manager
Spot instances to reduce cost

Worker ASG

Worker runs background tasks in an ASG, with its own build and deploy pipeline.

Multiple front end apps

Load Balancer routes traffic between multiple front end apps

Shared S3 buckets

Share data between S3 buckets
Use signed URLs to handle protected user content

Static website

Build the public website using a static site generator in CodeBuild, deploying to CloudFront CDN. Use Lambda@Edge to rewrite URLs.

Elasticache

Add Elasticache Redis or Memcached for app caching.

Elasticsearch

Add Elasticsearch for the app.

DevOps

Add a DevOps instance to handle deployment and management tasks.

Bastion host

Add Bastion host to control access to servers in the private subnet. Or use with AWS SSM Sessions.

Prometheus metrics

Add Prometheus for application metrics and monitoring

SES

Use SES for email.

How it works

It uses Terraform to create the infrastructure, Ansible and Packer to set up instances and AMIs. It uses AWS CodePipeline/CodeBuild/CodeDeploy to build and deploy code, running the app components in one or more autoscaling groups running EC2 instances.

The base of the system is Terraform and Terragrunt. Common Terraform modules can be enabled according to the specific application requirements. Similarly, it uses common Ansible playbooks which can be modified for specific applications. If an app needs something special, we can easily add a custom module for it.

We use the following terminology:

Apps are under an org, or organization, e.g. a company. org_unique is a globally unique identifier, used to name e.g. S3 buckets
An env is an environment, e.g. dev, stage, or prod. Each gets its own AWS account
An app is a single shared set of data, potentially accessed by multiple front end interfaces and back end workers. Each app gets it's own VPC. A separate VPC, generally one per environment, handles logging and monitoring using ELK and Prometheus
A comp is an application component

We have three standard types of components: web app, worker and cron.

Web apps process external client requests. Simple apps consist of only a single web app, but complex apps may have more, e.g. an API server, admin interface or instance per customer.

Workers handle asynchronous background processing driven by a job queue such as Sidekiq, SQS or a Kafka stream. They make the front end more responsive by offloading long running tasks. The number of worker instances in the ASG depends on the load.

Cron servers handle timed batch workloads, e.g. periodic jobs. From a provisioning perspective, there is not much difference between a worker and a cron instance, except that cron instances are expected to always be running so that they can schedule jobs. Generally speaking, we prefer to move periodic tasks to Lambda functions where possible.

We normally run application components in an auto scaling group, allowing them to start and stop according to load. This also provides high availability, as the ASG will start instances in a different availability zone if they die. This makes it useful even if we normally only have one instance running.

Running in an ASG requires that instances start from a "template" image AMI and be stateless, storing their data in S3 or RDS. We can also run components in standalone EC2 instances, useful for development and earlier in the process of migrating the app to the cloud.

We can also deploy the app to containers via ECS as part of the same system. Everything is tied together with a common ALB, so it's just a question of routing traffic.

When possible, we utilize managed AWS services such as RDS, ElastiCache, and Elasticsearch. When managed services lack functionality, are immature or are expensive at high load, we can run our own.

The system makes use of CloudFront to host application assets as well as static content websites or "JAM stack" apps using tools like Gatsby.

We deploy the application using AWS CodeDeploy using a blue/green deployment strategy. The CodeDeploy releases can be built using CodePipeline or a DevOps EC2 instance.

By default we use Route53 for DNS and ACM for certificates, though it can work with external DNS, certs and other CDNs like CloudFlare.

Terraform structure

Using Terragrunt, we separate the configuration into common modules, app configuration and environment-specific variables.

Under the terraform directory is the modules directory and a directory for each app, e.g. foo:

terraform
    modules
    foo
    bar

For many apps, the recommended Terragrunt structure in Keep your Terraform code DRY and example works fine. It uses a directory hierarchy like:

aws-account
    env
        region
            resources

e.g.

dev
    stage
        us-east-1
            asg

In this case, we may have multiple prod environments in different regions, each potentially with its own AWS account. We use a flatter structure combined with environment vars which determine which config vars to load.

Under the app directory are:

terragrunt.hcl
common.yml
dev.yml
prod.yml
dev
prod

terragrunt.hcl is the top level config file. It loads configuration from YAML files based on the environment, starting with common settings in common.yml and overriding them based on the environment, e.g. dev.yml.

Configure common.yml to name the app you are building, e.g. org, app and set the region it will run in.

Next configure the resources for the environment, e.g. dev. Each resource has a directory which defines its name and a terragrunt.hcl which sets dependencies and variables.

Dirs for each environment define which modules will be used. For example, this defines a single web app ASG behind a public load balancer, SSL cert, Route53 domain, RDS database, CodePipeline building in a custom container image, deploying with CodeDeploy, using KMS encryption keys:

acm-public
asg-app
codedeploy-app
codedeploy-deployment-app-asg
codepipeline-app
ecr-build-app
iam-codepipeline
iam-codepipeline-app
iam-instance-profile-app
iam-s3-request-logs
kms
launch-template-app
lb-public
rds-app
route53-delegation-set
route53-public
route53-public-www
s3-app
s3-codepipeline-app
s3-request-logs
sg-app-private
sg-db
sg-lb-public
sns-codedeploy-app
target-group-default
vpc

Modules are named by the AWS component plus a component-name suffix, e.g. asg-api for an autoscaling group for a web component handling API requests. Each component is a directory and a Terragrunt config file specifying the module and any necessary variables. For example:

terraform {
  source = "${get_terragrunt_dir()}/../../../modules//asg"
}
dependency "vpc" {
  config_path = "../vpc"
}
dependency "lt" {
  config_path = "../launch-template-api"
}
dependency "tg" {
  config_path = "../target-group-api"
}
include {
  path = find_in_parent_folders()
}

inputs = {
  comp = "api"

  min_size = 1
  max_size = 3
  desired_capacity = 1
  wait_for_capacity_timeout = "2m"
  wait_for_elb_capacity = 1

  health_check_grace_period = 30
  health_check_type = "ELB"

  target_group_arns = [dependency.tg.outputs.arn]
  subnets = dependency.vpc.outputs.subnets["private"]
  launch_template_id = dependency.lt.outputs.launch_template_id
  launch_template_version = "$Latest" # $Latest, or $Default

  spot_max_price = ""
  on_demand_base_capacity = 0
  on_demand_percentage_above_base_capacity = 0
  override_instance_types = ["t3a.nano", "t3.nano"]
}

The source identifies the Terraform code, in this case asg-app, and the dependencies. The second part sets variables, e.g. AMI and instance type, ASG size and health check parameters.

Outputs of one module are stored in the state, and we can then use them as inputs for other modules. The system is flexible, using separate modules identified by name and path. This makes it straightforward to define multiple front end or worker components or customize modules when necessary. This is a key advantage of Terraform over CloudFormation. When CloudFormation config gets large, it becomes hard to manage and extend. Terraform also supports multiple providers, not just AWS.

The configuration for dev is normally roughly the same as prod, but with e.g. smaller instances. It's possible, however, to have different structure as needed.

Ansible structure

Ansible is used to set up AMIs, perform tasks like creating db users, and generate config files in S3 buckets from templates. We may use the Ansible vault to store secrets or put them into SSM Parameter Store.

The structure generally follows the approach in "Setting Ansible variables based on the environment".

playbooks contains common and app-specific playbooks.

manage-users.yml
files
foo
    app-ssm.yml
    bastion.yml
    bootstrap-db-mysql.yml
    bootstrap-db-pg.yml
    bootstrap-db-ssm.yml
    config-app-https.yml
    config-app.yml
    devops.yml
    packer-app.yml

files has common files used by the playbooks, e.g. ssh public keys used by manage-users.yml.

vars contains configuration for each env. Most configuration is done here, not in the inventory, due to the need to manage multiple environments.

foo
    dev
        app-https.yml
        app-secrets.yml
        app.yml
        bastion.yml
        common.yml
        db-app.yml
        devops.yml

Following is an example playbook used to provision an AMI, playbooks/$APP/packer-$COMP.yml:

- name: Install base
  hosts: '*'
  become: true
  vars:
    app_name: foo
    comp: app
    tools_other_packages:
      - chrony
      # Parse cloud-init
      - jq
      # Sync config from S3
      - awscli
  vars_files:
    - vars/{{ app_name }}/{{ env }}/common.yml
    - vars/{{ app_name }}/{{ env }}/app.yml
    - vars/{{ app_name }}/{{ env }}/ses.yml
    - vars/{{ app_name }}/{{ env }}/ses.vault.yml
    - vars/foo/{{ env }}/elixir-release.yml
  roles:
    - common-minimal
    - tools-other
    - cogini.users
    - iptables
    - iptables-http
    - codedeploy-agent

    - cronic
    - postfix-sender
    - mesaguy.prometheus
    - postgres-client
    - cogini.elixir-release

It loads its config using vars_files from the vars directory, then runs a series of roles.

Packer structure

To actually build the AMI, we use Packer.

Makefile
README.md
builder
    build.sh
    build_centos.yml
    build_ubuntu.yml
foo
    dev
        build_app.sh
        build_cron.sh
        build_worker.sh
set_env.sh

Set OS environment vars and run:

./${APP}/${ENV}/build_app.sh

This launches an EC2 instance and runs the playbooks/foo/packer-app.yml Ansible playbook to configure it. The result is an AMI ID which you put into e.g. the image_id var in terraform/foo/dev/launch-template-app/terragrunt.hcl.

Need help?

Need help deploying your complex app? Get in touch!

A new approach to deploying Elixir apps: mix_deploy

2020-01-07T00:00:00+08:00

There has been a lot of action lately in Elixir to integrate releases into the core system and improve the process of configuring the application. We still need, however, some way to deploy the releases into production and pull configuration from the environment.

I have written two new libraries which focus on the level below the Erlang VM. mix_systemd generates a systemd unit file to run the app, and mix_deploy generates scripts to deploy the app to servers, interface with systemd and handle the deployment lifecycle of systems like AWS CodeDeploy.

They are used when deploying to virtual machines or dedicated servers, not a container system like Kubernetes.

These libraries are essentially a collection of opinionated templates which work together: they look at the configuration of the app in mix.exs and config/prod.exs and generate a systemd unit file which calls shell scripts.

They also automate the process of managing config files with production secrets, as well as runtime configuration from cloud-init to configure a cluster.

The focus is on three main scenarios:

Building and deploying on the same server
Building on a CI system like AWS CodeBuild and deploying to the cloud with a CD system like AWS CodeDeploy
Building on a CI server and deploying to dedicated servers using Ansible

The same approach scales from a $5/month Digital Ocean server to AWS running a load balancer and autoscaling group, to fleets of dedicated servers. It provides a simple and cheap way to get started. Elixir's excellent concurrency support means that it works great with a small number of servers with more cores rather than slicing things up.

Documentation is here:

mix_systemd GitHub project
mix_deploy GitHub project
A complete sample app
A step by step tutorial for beginners: Deploying an Elixir app to Digital Ocean with mix_deploy
Deploying Elixir apps with Ansible
Best practices for deploying Elixir apps

We have been using these libraries to deploy a number of production applications. The goal is that you can simply add the libraries to your project and they will do the right thing, while still being customizable to handle different situations. Contributions are welcome!

I am reachfh on Freenode #elixir-lang IRC channel, jakemorrison on on the Elixir Slack and Discord. I am in Taiwan, though, so look for me in that timezone :-).

Best practices for deploying Elixir apps

2019-12-31T00:00:00+08:00

Figuring out how to deploy your Elixir app can be confusing, as it's a bit different from other languages. It's very mature and well thought out, though. Once you get the hang of it, it's quite nice.

Our mix_deploy and mix_systemd libraries help automate the process. This working example puts the pieces together to get you started quickly. This post gives more background and links to advanced topics.

Summary

Big picture, we deploy Erlang "releases" using systemd for process supervision. We run in cloud or dedicated server instances. We build and deploy on the same server or build on a continuous integration server and deploy using Ansible or AWS CodeDeploy.

We make healthcare and financial apps, so we are paranoid about security. We run apps that get large amounts of traffic, so we are careful about performance. And we deploy to the cloud, so the apps need to be stateless, dynamically scaled under the control of a system like AWS CodeDeploy.

Locking dependency versions

The process starts in your dev environment. When you run mix deps.get, mix fetches the dependencies listed in the mix.exs, but they are normally only loosely specified, e.g. {:plug_cowboy, "~> 2.0"} will actually install the latest compatible version, 2.6.3.

Mix records the specific versions that it fetched in the mix.lock file. Later, on the build machine, mix uses the specific package version or git reference in the lock file to build the release.

This makes a release completely predictable and reproducible. It does not depend on the version of libraries installed on the server, and one app doesn't affect another. It's like Ruby's Gemfile.lock or Node's package-lock.json files. This locking happens automatically as part of the standard mix process, just make sure you check the mix.lock file into source control.

Managing Erlang and Elixir versions

For simple deployments, we can install Erlang and Elixir from binary packages. Instead of using the packages that come with the OS, which are generally out of date, use the packages from Erlang Solutions.

One disadvantage of OS packages is that only one version can be installed at a time. If different projects need different versions, then we have a conflict. Similarly, when we upgrade Erlang or Elixir, we need to first test the code with the new version, moving it through dev and test environments, then putting it into production. If anything goes wrong, we need to be able to roll back quickly. To support this, we need to precisely specify runtime versions and keep multiple versions installed so we can switch between them.

When building a release for production, Elixir is just another library dependency as far as Erlang is concerned. We can also package the Erlang virtual machine inside the release, so it's not necessary to install Erlang on the prod machine globally at all. Just install the release and it includes the matching VM.

That lets us upgrade production systems with no drama. We have apps which have been running continuously for years on clusters of servers, upgrading through multiple Elixir and Erlang versions with no downtime.

ASDF manages multiple versions of Erlang, Elixir and Node.js. It is a language-independent equivalent to tools like Ruby's RVM or rbenv.

The .tool-versions file in the project root specifies the versions to use:

erlang 22.2
elixir 1.9.4
nodejs 10.15.3

ASDF looks at the .tool-versions file and automatically sets the path to point to the correct version. The build script for the project runs asdf install to install the matching Erlang, Elixir and Node.js versions.

See Using ASDF with Elixir and Phoenix for details.

Building and testing

We normally develop on macOS and deploy to Linux. The Erlang VM mostly isolates us from the operating system, and mix manages library dependencies tightly, so we don't find it necessary to use Docker or Vagrant. It is necessary, however, to build the release with an Erlang VM executable that matches your target system. You can't just build the release on macOS and use it on a Linux server.

For simple projects, we build on the same server that runs the app: check out the code from git, build a release, then deploy it locally running under systemd.

In larger projects, a CI/CD server checks out the code, runs tests, then builds a release. We then deploy to the cloud using AWS CodeDeploy or deploy the release using Ansible.

Like your dev machine, the build server runs ASDF. When it makes a build, it automatically uses the versions of Erlang and Elixir specified in the .tool-versions file, which is in sync with the code. These build scripts handle the setup and build process.

Erlang releases

The most important part of the deployment process is using Erlang "releases". A release combines the Erlang VM, your application, and the libraries it depends on into a tarball, which you deploy as a unit. The release has a script to start the app, launched and supervised by the OS init system (e.g. systemd). If it dies, the system restarts it.

Releases handle a lot of the details you need to run things reliably in production, e.g.:

Packaging
Configuration
Running migrations
Getting a console on a running app
Upgrades

Building releases

Since Elixir 1.9, mix has built in support for creating releases. For earlier versions, use the Distillery library.

Configure your release in mix.exs.

def project do
  [
    app: :foo,
    releases: [
      prod: [
        include_executables_for: [:unix],
        steps: [:assemble, :tar]
      ]
    ]
  ]
end

rel/vm.args.eex sets Erlang VM startup arguments. We normally tune it to increase TCP ports for high volume apps.

Generate a template in your project under rel:

mix release.init

Edit it as needed, then build the release:

MIX_ENV=prod mix release

This creates a tarball with everything you need to deploy:

_build/prod/foo-0.1.0.tar.gz

Running database migrations

In the deployed system, we don't have mix. The release command script allows us to call an Elixir function to run migrations from a release.

/srv/foo/current/bin/foo eval "Foo.Release.migrate"

Configuration

There are four different kinds of things that we may want to configure:

Static information about application layout, e.g. file paths. This is the same for all machines in an environment, e.g. staging or prod.
Information specific to the environment, e.g. the hostname of the db server.
Secrets such as db passwords, API keys or TLS keys.
Dynamic information such as the IP address of the server or other machines in the cluster.

Elixir has a couple of mechanisms for storing configuration. When you compile the release, it converts Elixir-format config files like config/prod.exs into an initial application environment (sys.config) that is read by Application.get_env/3. That's fine for simple, relatively static apps. It's better to keep secrets separate from the release, though.

Elixir 1.9 releases support dynamic configuration at runtime. You can run the Elixir file config/releases.exs when it boots or use the shell script rel/env.sh.eex to set environment vars. With these you can theoretically do anything. In practice, however, it can be more convenient and secure to process the config outside of the app. That's where mix_systemd and mix_deploy come in.

Environment vars

The simplest way to configure your app is via OS environment variables. You can set them via the systemd supervisor or container runtime. Your application then calls System.get_env/1 in config/releases.exs or application startup. Note that these environment vars are read at runtime, not when building your app.

mix_systemd supports reading environment vars from files, e.g.

/srv/foo/etc/environment
/etc/foo/environment
/run/foo/environment

This lets you set config defaults in the release, then override them in the environment or at runtime.

Config providers

At a certain point, making everything into an environment var becomes annoying. It's verbose and vars are simple strings, so you have to encode values safely and convert them back to lists, integers or atoms.

Config providers load data on startup, merging it with the default application environment before starting the VM. This lets us keep secrets outside of the release file and change settings depending on where the app is running. We also keep secrets out of the build environment, e.g. a shared CI system.

They support standard formats like TOML:

[foo."Foo.Repo"]
url = "ecto://foo_prod:Sekrit!@db.foo.local/foo_prod"
pool_size = 15

[foo."FooWeb.Endpoint"]
secret_key_base = "EOdJB1T39E5Cdeebyc8naNrOO4HBoyfdzkDy2I8Cxiq4mLvIQ/0tK12AK1ahrV4y"

Add the TOML config provider to mix.exs:

defp releases do
   [
     foo: [
       include_executables_for: [:unix],
       config_providers: [
         {TomlConfigProvider, path: "/etc/foo/config.toml"}
       ],
       steps: [:assemble, :tar]
     ]
   ]
 end

The startup scripts read the initial application environment compiled into the release, parse the config file, merge the values, write it to a temp file then start the VM. Because of that, they need a writable directory. That is configured using the RELEASE_TMP environment var, which you can set to the app's runtime_dir, e.g. /run/foo.

Copying files

This config file approach is simple, but effective. The question is how to get the environment files onto the server. When deploying a simple app on the same server, we can just copy the prod.secret.exs or environment file to /etc/foo. When deploying to dedicated servers, we can generate the config file using Ansible and push it to the server.

In cloud environments, we may run from a read-only image, e.g. an Amazon AMI, which gets configured at start up based on the environment by copying the config from an S3 bucket. See deploy-sync-config-s3 in mix_deploy.

Config servers and vaults

You can also store config params in an external configuration system and read them at runtime. An example is AWS Systems Manager Parameter Store.

Set a parameter using the AWS CLI:

aws ssm put-parameter --name '/foo/prod/db/password' --type ‘SecureString’ --value 'Sekrit!"

While it's possible to read params in config/releases.exs, it's tedious. Better is to grab all of them at once and write them to a file, then read it in with a Config Provider like aws_ssm_provider.

Application initialization

Instead of doing a lot of work in your config/releases.exs file, keep it focused on getting the data. Handle application config in your Application.start/2 or Supervisor.init/1. This leverages the supervision structure of OTP, allowing components to fail and be restarted with the right configuration.

Supervising your app

In the Erlang OTP framework, we use supervisors to start and stop processes, restarting them in case of problems. It's turtles all the way down: you need a supervisor to make sure your Erlang VM is running, restarting it if there is a problem.

Ignore the haters, systemd is the best supervisor we have right now, and all the Linux distros are standardizing on it. We might as well take advantage of it. Systemd handles all the things that "well behaved" daemons need to do. Instead of scripts, it has declarative config that handles standard situations. It sets up the environment, handles logging and controls permissions.

mix_systemd generates a systemd unit file for your app and mix_deploy generates the scripts it needs to start and configure it.

Permissions and directories

For security, following the principle of least privilege, we limit the app to only what it really needs to do its job. If the app is compromised, the attacker can only do what the app can do.

We use one OS user (deploy) to upload the release files, and another (e.g. foo) to run the app. This means that the app only needs to have read-only access to its own source code and config. The app user account does not need permissions to restart the app, that's handled by the deploy user or systemd.

We make use of systemd features and cloud services. Instead of writing our own log files, we send them to journald, which sends them to CloudWatch Logs or ELK. When running in the cloud, the app should be stateless. Instead of putting files on the disk, it keeps state in an RDS database and uses S3 for file storage.

The result is that many apps can run without needing write access to anything on the disk, improving security.

Deploying the app

So now we have a release tarball and some config files, time to put them on a server.

Build and deploy to the same server
Build with CodeBuild and deploy with CodeDeploy doc and example
Build on a build server and deploy using Ansible

Connecting to the outside world

The app isn't much use if we can't talk to it. There are two options for how to receive traffic, direct or via a proxy.

You can serve your Phoenix app with Nginx, but listening direct gives you lower latency and overall lower complexity. Erlang can handle lots of load with no problems. For example, Heroku's routing layer is based on Erlang. We have apps that handle a billion requests a day, including DDOS attacks. You can handle 3000 requests per second on a simple $5/month Digital Ocean droplet.

In a modern cloud app running running behind a load balancer, then listening on port 4000 is fine, just tell the load balancer to use that port. For a freestanding app, we need to listen to port 80 and/or port 443 for SSL. We normally redirect traffic from port 80 to 4000 in the firewall using iptables.

You may need to set some HTTP options that Nginx was dealing with, e.g.:

config :foo, FooWeb.Endpoint,
  http: [
    compress: true,
    protocol_options: [max_keepalive: 5_000_000]
  ],

Things you probably don't need right now

While they are cool, you don't initially need to worry about:

Hot code updates
Distributed Erlang

Additional topics

Running Ecto migrations in a release

2019-06-30T00:00:00+08:00

We do this by defining a helper function which we can then call from the release startup script:

/srv/foo/current/bin/foo eval "MyApp.ReleaseTasks.migrate.run"

The main point is that we need to initialize the database connection information the same way as your application normally does, e.g. by loading a runtime file like /etc/foo/config.toml or setting environment variables. If you use TOML configuration files, you will need to install the package toml_config_provider from Hex.

Following is a working example. Where you see CHANGEME, use the name of your project.

defmodule Foo.ReleaseTasks.Migrate do
  @moduledoc "Mix task to run Ecto database migrations"

  # CHANGEME: Name of app as used by Application.get_env
  @app :foo
  # CHANGEME: Name of app repo module
  @repo_module Foo.Repo

  def run(args \\ [])

  def run(_args) do
    ext_name = @app |> to_string |> String.replace("_", "-")
    config_dir = Path.join("/etc", ext_name)

    # Read config.exs if present
    config_exs = Path.join(config_dir, "config.exs")
    app_env =
      case File.exists?(config_exs) do
        true ->
          IO.puts("==> Loading config file #{config_exs}")
          Config.Reader.merge([], Config.Reader.read!(config_exs))

        _ ->
          app_env
      end

    # Read TOML config if present (requires `toml_config_provider` package)
    config_toml = Path.join(config_dir, "config.toml")
    app_env =
      case File.exists?(config_toml) do
        true ->
          IO.puts("==> Loading config file #{config_toml}")
          TomlConfigProvider.load(config, config_toml)

        _ ->
          app_env
      end

    Application.put_env(@app, @repo_module, app_env[@app][@repo_module])

    # Start requisite apps
    IO.puts("==> Starting applications..")

    for app <- [:crypto, :ssl, :postgrex, :ecto, :ecto_sql] do
      {:ok, res} = Application.ensure_all_started(app)
      IO.puts("==> Started #{app}: #{inspect(res)}")
    end

    # Start repo
    IO.puts("==> Starting repo")
    {:ok, _pid} = apply(@repo_module, :start_link, [[pool_size: 2, log: :info, log_sql: true]])

    # Run migrations for the repo
    IO.puts("==> Running migrations")
    priv_dir = Application.app_dir(@app, "priv")
    migrations_dir = Path.join([priv_dir, "repo", "migrations"])

    opts = [all: true]
    config = apply(@repo_module, :config, [])
    pool = config[:pool]

    if function_exported?(pool, :unboxed_run, 2) do
      pool.unboxed_run(@repo_module, fn ->
        Ecto.Migrator.run(@repo_module, migrations_dir, :up, opts)
      end)
    else
      Ecto.Migrator.run(@repo_module, migrations_dir, :up, opts)
    end

    # Shut down
    :init.stop()
  end
end

Deploying Elixir apps with Ansible

2019-06-16T00:00:00+08:00

Elixir runs great on bare metal, as it easily takes advantage of all the cores on the machine. We can get a machine with 24 CPU cores, 32 GB of RAM and 10TB of traffic for under $100 month. Try that in the cloud. You can cut your hosting bills by 90%.

When deploying to bare metal, we use Ansible, an easy-to-use standard tool for managing servers. It has reliable and well documented primitives to handle logging into servers, uploading files and executing commands: just what we need to deploy an Elixir app. We also use it when deploying to AWS, setting up AMIs for use in auto-scaling groups. This guide shows how to set up and run a Phoenix app, talking to a Postgres database. It is based on this working template and the principles in "Best practices for deploying Elixir apps".

Install Ansible

On your local dev machine, install Ansible with the Python pip command:

python -m pip install ansible

See the Ansible docs for other options.

Clone the example application

git clone https://github.com/cogini/mix-deploy-example.git

Set SSH alias and Ansible host list

Ansible uses ssh to talk to the server. On your local dev machine, add an ssh host alias in the ~/.ssh/config file so you can reference the server using its name:

Host web-server1
    HostName 123.45.67.89

You can use any name you like, but it needs to match the name in the Ansible "inventory", e.g. ansible/inventory/hosts. This file puts hosts into groups so we can manage them together, e.g.:

[web_servers]
web-server1
web-server2

[db_servers]
db-server1
db-server2

[web_servers] is a group of web servers. web-server1 is a hostname from the Host line in your .ssh/config file.

The configuration variables defined in inventory/group_vars/all apply to all hosts in your project. They are overridden by vars for more specific groups like inventory/group_vars/web_servers or for individual hosts, e.g. inventory/host_vars/web-server.

Configure the Elixir app

We use our elixir-release Ansible role to set up and deploy the Elixir app.

Under the ansible dir, edit inventory/group_vars/all/elixir-release.yml to match the Elixir app you are deploying:

# External name of the app, used to name directories and the systemd unit
elixir_release_app_name: mix-deploy-example

See the docs for the elixir-release Ansible role for more options.

Configure OS accounts

We use our users Ansible role to manage the accounts used to deploy and run the app, as well as control access for system administrators and developers.

For security, we use separate accounts to deploy the app and to run it. The deploy account owns the code and config files, and has rights to restart the app. We normally use a separate account called deploy. The app runs under a separate account with the minimum permissions it needs at runtime. We normally create a name matching the app, e.g. foo or use a generic name like app.

Configure users and associated ssh keys in inventory/group_vars/all/users.yml. The following defines a user jake, getting the ssh keys from their GitHub profile, and a user ci for a CI/CD server account, getting the ssh key from a file.

users_users:
  - user: jake
    name: "Jake Morrison"
    github: reachfh
  - user: ci
    name: "CI server"
    key: ci.pub

users_deploy_users defines users that are allowed to log into the deploy account on the server via ssh, e.g.:

users_deploy_users:
  - jake
  - jenkins

users_global_admin_users defines admin users, e.g. for your ops team. The following creates a separate account called bob on the server with sudo:

users_global_admin_users:
  - bob

See the docs for the users Ansible role for more options.

We split the deployment into two phases, setup and deploy. In the setup phase, we do the tasks that require elevated permissions, e.g. creating user accounts, creating app dirs, installing OS packages, and setting up the firewall.

In the deploy phase, we push the latest code to the server and restart it. The deploy doesn't require admin permissions, so it can run from a regular user, e.g. the build server.

Set up the web server

Once things are configured, run Ansible to do initial server setup, creating users and configuring the firewall.

From your local dev machine, run this playbook:

ansible-playbook -u root -v -l web_servers playbooks/setup-web.yml -D

The -u flag specifies the user for bootstrapping, after that you would normally use your own admin user. The user needs to be root or have sudo permissions. Depending on the hosting provider's provisioning process, that might be root or a default user like centos or ubuntu with your keypair installed. See playbooks/manage-users.yml for other connection options, e.g. specifying a root password manually.

The -v flag controls verbosity, you can add more v's to get more debug info. The -D flag shows diffs of the changes Ansible makes on the server. If you add --check to the Ansible command, it will show you the changes it is planning to do, but doesn't actually run them.

This playbook uses the iptables and iptables-http roles to set up the base firewall and port forwarding to redirect port 80/443 to the non-privilged port the app listens on.

Set up the app

Next, set up the server to run the app, creating directories and configuring the systemd unit. The mix_systemd Elixir library creates a systemd unit file for the app, and the mix_deploy library generates utility scripts.

When deploying to a single server, you can build on the server and run the scripts to set it up. When deploying to more servers, we use the elixir-release Ansible role. It creates directories and copies the generated files from the local project to the server.

From your local dev machine, run this playbook:

ansible-playbook -u $USER -v -l web_servers playbooks/deploy-app.yml --skip-tags deploy -D

This runs all the tasks in the role except those tagged with deploy, which are instead run from the CI/CD server.

Deploy the app config

Instead of baking secrets like db passwords into the release file, we create a config file and copy it to the app config dir under /etc.

Here we use the Ansible vault to manage app secrets.

First, configure the db server settings in inventory/group_vars/all/db.yml db_password in inventory/group_vars/all/secrets.yml and secret_key_base in inventory/group_vars/web_servers/secrets.yml.

From your local dev machine, run this playbook to generate a TOML config file with the secrets and push it to the web servers:

ansible-playbook -u $USER -v -l web_servers playbooks/config-web.yml -D

See templates/app.config.toml.j2.

Deploy the app to web server

Finally, we are ready to deploy the app. We build the release on a build server, then push it to prod servers and restart.

On a CI/CD server, run the following playbook:

ansible-playbook -u deploy -v -l web_servers playbooks/deploy-app.yml --tags deploy --extra-vars ansible_become=false -D

-u deploy specifies that the CI server should connect to the target server as the deploy user using the ssh key we set up before.

Other tasks

Ansible is useful for all sorts of admin tasks.

This playbook sets up the build server, installing ASDF version manager and checking out the app from git:

ansible-playbook -u root -v -l build_servers playbooks/setup-build.yml -D

See inventory/group_vars/build_servers/vars.yml, particularly app_repo for the URL of the git repo.

You can install Ansible on the build machine with:

ansible-playbook -u $USER -v -l web_servers playbooks/setup-ansible.yml -D

The following playbook sets up a Postgres database:

ansible-playbook -u $USER -v -l db_servers playbooks/setup-db.yml -D

Configuration is in inventory/group_vars/db_servers/postgresql.yml.

Managing user accounts with Ansible

2019-05-24T00:00:00+08:00

As part of developing and deploying web applications, we need to be able to manage OS user accounts and control access for developers and systems admins.

To do this, we wrote an Ansible role to manage users. It is basically an opinionated wrapper on the Ansible user module. This post describes how it works.

At a high level, we create an OS account for the app to run under and an account to deploy the app. We may also create accounts for specific users. We then add ssh user keys to control access to these accounts. The ssh keys for users can be stored locally or pulled from GitHub.

User types

We separate users into different types:

Global system admins / ops team

When we provision a server, we automatically create accounts for our system admin team, independent of the project.

These users have their own accounts on the server with sudo permissions. We add them to the wheel or admin group, then allow them to run sudo without a password.

Project admins / power users

These users have the same rights as global admins, but are set up on per-project or per-server basis, controlled with inventory host/group vars. Normally the tech lead for the project would be an admin.

Deploy account

This user account is used to deploy the application to the server. It owns the application software files and has write permissions to the deploy and config directories.

The app and deploy accounts do not have sudo permissions. We may, however, configure sudo to allow them to run specific commands to e.g. restart the app by running systemctl restart foo. That is handled by the role that installs and configures the app, not this role.

For example, make a file like /etc/sudoers.d/deploy-foo:

deploy ALL=(ALL) NOPASSWD: /bin/systemctl start foo, /bin/systemctl stop foo, /bin/systemctl restart foo, /bin/systemctl status foo

Another option is to configure systemd to look for a flag file and restart the app if it changes. See mix_deploy for an example.

App account

The application runs under this user account.

For better security, we limit what this account can do. It has write access to the directories it needs at runtime, e.g. logs, and has read-only access to its code and config files.

Developers

Developers may need to access the deploy account to deploy or the app user account to look at the logs and debug it. We add the ssh keys for developers to the accounts, allowing them to log in via ssh.

For systems with sensitive data like health care or financial services, we tightly control access to production servers, but give more access to dev servers.

Project users

These users have their own OS user accounts like admins, but don't have sudo. An example might be an account for a customer who needs to be able to log in and run queries against the db. You can give them permissions to e.g. access the log files for the app by adding them to the app group and setting file permissions.

Configuration

By default, this role does nothing.

For it to do something, you need to define variables in group vars like inventory/group_vars/app-servers, host vars like inventory/host_vars/web-server or a vars section in a playbook.

You can have different settings on a host or group level to e.g. give developers login access in the dev environment but not on prod.

App accounts

users_deploy_user specifies the account that deploys the app. Optional, if not specified the deploy user will not be created. users_deploy_group specifies the group, defaults to users_deploy_user.

users_deploy_user: deploy
users_deploy_group: deploy

users_app_user specifies the account that runs the app. Optional, if not specified the app user will not be created. users_app_group specifies the group, defaults to users_app_user.

users_app_user: foo
users_app_group: foo

User accounts

users_users defines OS account names and ssh keys for users. It is simply a list of users, the accounts are not created until they are referenced.

It is a list of dicts with four fields:

user: Name of the OS account
name: User's name. Optional.
key: ssh public key file. Put them in e.g. your playbook files directory. Optional.
github: The user's GitHub id. The role gets the user keys from https://github.com/{{ github }}.keys. Optional.

Example:

users_users:
  - user: jake
    name: "Jake Morrison"
    github: reachfh
  - user: ci
    name: "CI server"
    key: ci.pub

Lists of users

After you have defined users, you add them to groups for the specific servers, specifying the id used in the user key. By default, these are empty, so if you don't specify users, they will not be created.

Global admin users with a separate OS account and sudo permissions.

users_global_admin_users:
 - jake

Project level admin users with a separate OS account and sudo permissions.

users_admin_users:
 - fred

Project users with a separate OS account but no sudo permission.

users_regular_users:
 - bob

Users (ssh keys) who can access the deploy account.

users_deploy_users:
 - ci

Users (ssh keys) who can access the app account.

users_app_users:
 - fred

Group configuration

You can specify additional groups which the different types of users will have. By default these lists are empty, but you can use it to fine tune access to the app.

We normally configure ssh so that a user account must must be a member of a sshusers group, or ssh will not allow anyone to log in.

Add this to /etc/ssh/sshd_config

AllowGroups sshusers

Then add sshusers to the users_admin_groups, e.g.

users_admin_groups:
  - sshusers

Admin user groups

Additional groups that admin users should have.

The role will always be added the wheel or admin group, depending on the platform, independent of the OS groups specified here. If there are admin users defined, then this role sets up sudo with a /etc/sudoers.d/00-admin file so that admin users can run sudo without a password.

users_admin_groups:
  - sshusers

Regular user groups

Additional groups that regular users should have.

users_regular_groups:
  - sshusers

Deploy user groups

Additional groups that the deploy account should have.

users_deploy_groups:
  - sshusers

App user groups

Additional groups that the app account should have.

users_app_groups:
  - sshusers

Deleting users

This role puts "ansible-" in the comment when it creates users. This allows it to track when users are added or removed from the lists and delete the accounts.

You can also specify accounts in the users_delete_users list and they will be deleted. This is useful for cleaning up legacy accounts.

You can control whether to delete the user's home directory when deleting the account with the users_delete_remove and users_delete_force variables. See the Ansible docs for details. For safety, these variables are no by default, but if you are managing the system users with this role, you probably want to set them to yes.

users_delete_remove: yes
users_delete_force: yes

The role can optionally remove authorized keys from system users like 'root' or 'centos'. This is useful for security to avoid backup root keys, once you have set up named admin users.

users_remove_system_authorized_keys: true

Setup

This role is normally run as the first thing on a new instance. That creates admin users and sets up their keys so that they can run the other roles which configure the server.

A project specific role is responsible for preparing the server for the app, e.g. creating directories and installing dependencies. We normally deploy the app from a build or CI server, without sudo, using the deploy user account.

Here is a typical playbook:

- name: Manage users
  hosts: '*'
  vars:
    users_app_user: foo
    users_app_group: foo
    users_deploy_user: deploy
    users_deploy_group: deploy
    users_users:
      - user: jake
        name: "Jake Morrison"
        github: reachfh
    users_app_users:
      - jake
    users_deploy_users:
      - jake
  roles:
    - { role: cogini.users, become: true }

Add the host to the inventory/hosts file.

[web-servers]
web-server-01

Add the host to .ssh/config or a project specific ssh.config file.

Host web-server-01
    HostName 123.45.67.89

On a physical server where we start with a root account and no ssh keys, we need to bootstrap the server the first time, specifying the password with -k.

ansible-playbook -k -u root -v -l web-server-01 playbooks/manage-users.yml --extra-vars "ansible_host=123.45.67.89"

On macOS the -k command requires the askpass utility, which is not installed by default, so it falls back to Paramiko, which doesn't understand .ssh/config, so we specify ansible_host manually.

On following runs, after the admin users are set up, use:

ansible-playbook -u $USER -v -l web-server-01 playbooks/manage-users.yml

Deleting legacy users

Define legacy user accounts to delete in the users_delete_users list, e.g.:

ansible-playbook -u $USER -v -l web-servers playbooks/manage-users.yml --extra-vars "users_delete_users=[fred] users_delete_remove=yes users_delete_force=yes"

Multiple databases with Digital Ocean Managed Databases Service

2019-05-24T00:00:00+08:00

Digital Ocean's new Managed Databases service takes care of managing your database for you.

If you are creating one db cluster per app, you can use the defaultdb database and doadmin user. It's cheaper, however, to run multiple applications on the same db cluster or dev/staging/prod versions of the same app. We can create databases and users via the UI, but by default all users have full rights to all databases.

A better solution is to create users with restricted permissions. To do that we need to set permissions via SQL.

First, install the Postgres client library on your Droplet (assuming Ubuntu 18.04):

sudo apt-get install postgresql-client postgresql-client-common

Log into the database shell as the doadmin account:

psql -U doadmin -h <host> -p 25060 -d defaultdb

Create the production database and make it private by default:

create database app_prod;
revoke all on database app_prod from public;

On your dev machine, generate a strong password for the db user using e.g. pwgen:

pwgen -s 16

Create the prod db user, putting in the user password:

create user app_prod with encrypted password 'CHANGEME';

Give the app user rights to manage the prod db:

grant all privileges on database app_prod to app_prod;

Exit the shell with \q. Confirm that the db user can connect:

psql -U app_prod -h <host> -p 25060 -d app_prod

Follow the same procedure to create a test database and user.

create database app_test;
revoke all on database app_test from public;
create user app_test with encrypted password 'CHANGEME2';
grant all on database app_test to app_test;

These db users have full rights on their respective databases, e.g. they can create tables. This matches what is expected by web frameworks that run database migrations on startup. They can't, however, create databases.

Applications with higher security requirements can separate the user account which manages the database from the user which accesses the database at runtime. If an attacker manages to compromise your application, then they are limited in what they can do. Other examples are creating read-only views which hide sensitive information.

See "How to manage default privileges for users on a database vs schema" for more details.

Running Ecto migrations in production releases with Distillery custom commands

2019-05-24T00:00:00+08:00

In a dev or test environment, we execute the mix ecto.migrate command to run database migrations. When running from a release, however, the mix command is not available. Instead, we need to execute Ecto.Migrator.run/4 from code. We do this by adding a Distillery custom command called migrate which we call from the release script, e.g.:

/srv/foo/current/bin/foo migrate

The fundamentals of defining and running a migration are covered in this article. In a more complex app, however, we need a bit more in our script to initialize the environment.

First, in rel/config.exs, define the migrate command:

set commands: [
    migrate: "rel/commands/migrate.sh"
]

Next, create the script rel/commands/migrate.sh, changing Foo to match your project module name:

release_ctl eval --mfa "Foo.Tasks.Migrate.run/1" --argv -- "$@"

Finally, add the Elixir code which runs the database migrations.

Following is a working example. Where you see CHANGEME, use the name of your project.

defmodule Foo.Tasks.Migrate do
  @moduledoc "Mix task to run Ecto database migrations"

  # CHANGEME: Name of app as used by Application.get_env
  @app :foo
  # CHANGEME: Name of app repo module
  @repo_module Foo.Repo

  def run(_args) do
    ext_name = @app |> to_string |> String.replace("_", "-")
    config_dir = Path.join("/etc", ext_name)

    config_exs = Path.join(config_dir, "config.exs")
    if File.exists?(config_exs) do
      IO.puts "==> Loading config file #{config_exs}"
      Mix.Releases.Config.Providers.Elixir.init([config_exs])
    end

    config_toml = Path.join(config_dir, "config.toml")
    if File.exists?(config_toml) do
      IO.puts "==> Loading config file #{config_toml}"
      Toml.Provider.init([path: config_toml])
    end

    repo_config = Application.get_env(@app, @repo_module)
    repo_config = Keyword.put(repo_config, :adapter, Ecto.Adapters.Postgres)
    Application.put_env(@app, @repo_module, repo_config)

    # Start requisite apps
    IO.puts "==> Starting applications.."
    for app <- [:crypto, :ssl, :postgrex, :ecto, :ecto_sql] do
      {:ok, res} = Application.ensure_all_started(app)
      IO.puts "==> Started #{app}: #{inspect res}"
    end

    # Start repo
    IO.puts "==> Starting repo"
    {:ok, _pid} = apply(@repo_module, :start_link, [[pool_size: 2, log: :info, log_sql: true]])

    # Run migrations for the repo
    IO.puts "==> Running migrations"
    priv_dir = Application.app_dir(@app, "priv")
    migrations_dir = Path.join([priv_dir, "repo", "migrations"])

    opts = [all: true]
    config = apply(@repo_module, :config, [])
    pool = config[:pool]
    if function_exported?(pool, :unboxed_run, 2) do
      pool.unboxed_run(@repo_module, fn -> Ecto.Migrator.run(@repo_module, migrations_dir, :up, opts) end)
    else
      Ecto.Migrator.run(@repo_module, migrations_dir, :up, opts)
    end

    # Shut down
    :init.stop()
  end
end

Tuning TCP ports for your Elixir app

2019-05-24T00:00:00+08:00

Elixir is great at handling lots of concurrent connections. When you actually try to do this, however, you will bump up against the default OS configuration which limits the number of open filehandles/sockets. You may also run out of TCP ephemeral ports.

The result is poor application performance, e.g. timeouts. If you are running behind Nginx, you may see it as 503 errors, with your application taking five seconds to respond. When you look at the logs, however, the application response time is fine.

What is happening is that the client talks to Nginx, then Nginx talks to your app, but there are not enough filehandles available, so Nginx queues the request. You may start with 1024 by default, which is pitifully small. You will need to raise that at each step in the config, e.g. systemd unit file, Nginx, and Erlang VM.

First, make sure that open file limits are increased at each step in the chain.

OS account limits

The OS account running the app has default limits:

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3841
max locked memory       (kbytes, -l) 16384
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 3841
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Here we can see the open file limit of 1024 by default.

You can see the same for a running process by looking up limits for the process id, cat /proc/<pid>/limits:

# cat /proc/800/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             3841                 3841                 processes
Max open files            65535                65535                files
Max locked memory         16777216             16777216             bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       3841                 3841                 signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Create a /etc/security/limits.d/foo-limits file for the account running the app:

foo soft    nofile          1000000
foo hard    nofile          1000000

Systemd

Systemd doesn't use the limits file, though, so you need to set the limit with variables like LimitNOFILE in the unit file for your app:

LimitNOFILE=65536

Erlang VM options

In rel/vm.args for your release, increase the number of ports on the command line. In newer Erlang versions, the setting is +Q 65536. In older ones, use -env ERL_MAX_PORTS 65536.

Nginx open files

If you are running Nginx in front of your app, make sure it has enough open ports. See "Serving your Phoenix app with Nginx".

Ephemeral TCP ports

After that, you may run into lack of ephemeral TCP ports. This hits you very hard running behind Nginx as a proxy, but can also hit you on the outbound side when you are talking to a small number of back end servers.

In TCP/IP, a connection is defined by the combination of source IP + source port + destination IP + destination port. In this situation, all but the source port is fixed: 127.0.0.1 + random + 127.0.0.1 + 4000. There are only 64K ports. The TCP/IP stack won't reuse a port for 2 x maximum segment lifetime, which by default is 2 minutes.

Doing the math:

1024 ports / 120 sec = 8.53 requests per second with default file handle limit
60,000 / 120 = 500 requests per sec

If you are getting limited talking to back end servers, then it's useful to give your server multiple IP addresses. Tell your HTTP client library to use an IP from a pool as its source when making the request. Then the equation turns into "source IP from pool" + random port + target IP + 80.

You may be able to reuse outbound connections, with HTTP pipelining, if the back ends support it. At a certain point, the back end servers may be the limit. They may benefit from having more IPs as well.

DNS

DNS lookups can become an issue on outbound connections. We have had hosting providers block us because they thought we were doing a DOS attack on their DNS. Google DNS rate limits to 100 requests per second by default. Run a local caching DNS for your app.

Kernel tuning

You can tune the OS kernel settings to reduce the maximum segment lifetime.

In /etc/sysctl.conf (or a file in /etc/sysctl.d/):

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15
# Recycle and Reuse TIME_WAIT sockets faster
net.ipv4.tcp_tw_reuse = 1

Load the new settings:

sudo sysctl -p

There are other kernel TCP settings you can tune as well, e.g.

sysctl -w fs.file-max=12000500
sysctl -w fs.nr_open=20000500
ulimit -n 20000000
sysctl -w net.ipv4.tcp_mem='10000000 10000000 10000000'
sysctl -w net.ipv4.tcp_rmem='1024 4096 16384'
sysctl -w net.ipv4.tcp_wmem='1024 4096 16384'
sysctl -w net.core.rmem_max=16384
sysctl -w net.core.wmem_max=16384

See https://phoenixframework.org/blog/the-road-to-2-million-websocket-connections and https://www.rabbitmq.com/networking.html#dealing-with-high-connection-churn

See this presentation on tuning Elixir performance.

Configure ssh to connect to a server

2019-05-01T00:00:00+08:00

This article describes how to configure ssh to connect to a server using an ssh key for access. Using ssh keys is more secure than passwords, and makes it easier to automate systems using tools like Ansible.

First, create an ssh key, if you don't have one already.

Configure your ssh config file

If your server only has an IP address, you can make a host alias to make it easier to use. Create a file on your local machine called ~/.ssh/config. Add the server to it:

Host web-server
    HostName 123.45.67.89

The ssh config file supports a lot more options. For example, you can specify the userid to use on the remote server, the port, or the key.

Set the file permissions on ~/.ssh/config:

chmod 600 ~/.ssh/config

ssh is picky about file permissions. For security, the files and directories need to only be readable by you, and ssh will refuse to work if they are wrong.

Test it by connecting to the server:

ssh user@web-server

If it doesn't work, run ssh with -v flags to see what the problem is. You can add more verbosity, e.g. -vvvv if you need more detail.

ssh -vv user@web-server

For example:

chown $USER:staff ~/.ssh
chmod 700 ~/.ssh

chown $USER:staff ~/.ssh/id_rsa
chmod 600 ~/.ssh/id_rsa

Using ASDF with Elixir and Phoenix

2019-03-26T00:00:00+08:00

For simple deployments, we can install Erlang and Elixir from binary packages. Instead of using the packages that come with the OS, which are generally out of date, you should use the packages from Erlang Solutions.

One disadvantage of the OS packages is that we can only have one version installed at a time. If different projects have different versions, then we have a problem. Similarly, when we upgrade the Erlang or Elixir version, we should first test the code with the new version, moving a version through dev and test environments, then putting it into production. If anything goes wrong, we need to be able to roll back quickly. To support this, we need to precisely specify runtime versions and keep multiple versions installed so we can quickly switch between them.

This is mainly useful for dev and build environments. For production, use releases. Releases bundle the VM with the code, so we don't need to install Erlang on the prod machine at all. We just install the release and it includes the matching VM that we tested with.

The ASDF version manager lets us manage multiple versions of Erlang, Elixir and Node.js. It is a language-independent equivalent to tools like Ruby's RVM or rbenv.

ASDF is safe to install on your machine beside the Erlang packaged with your OS. It won't conflict with anything else, that's its whole reason for existing. You can, however, also use it to install a default global version.

It uses the .tool-versions file in your project to automatically set the path to use specific versions. The file looks like this:

erlang 21.3
elixir 1.8.1
nodejs 10.15.3

The asdf install command reads the file and installs the necessary versions on your system if necessary.

Install ASDF

These instructions assume that you are running macOS on your dev machine and Linux on your prod machine.

This script automates the process of installing ASDF on macOS. Following are step by step commands with explanation.

First, get the ASDF code:

git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.7.1

Add commands to your shell startup scripts:

echo -e '\n. $HOME/.asdf/asdf.sh' >> ~/.bash_profile
echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> ~/.bash_profile

The above commands installed from git to be more consistent with the Linux side.

You can also install via Homebrew:

brew install asdf
echo -e '\n. $(brew --prefix asdf)/asdf.sh' >> ~/.bash_profile
echo -e '\n. $(brew --prefix asdf)/etc/bash_completion.d/asdf.bash' >> ~/.bash_profile

After installing ASDF, log out of your shell and log back in to activate the scripts.

See the ASDF docs for more details.

Next, install the ASDF plugins for Elixir and Phoenix:

asdf plugin-add erlang
asdf plugin-add elixir
asdf plugin-add nodejs

Install build dependencies

ASDF builds Erlang from source, so it needs to have some build tools and libraries installed. Other packages like Node.js have dependencies as well.

On macOS, first install Homebrew, then run:

# Install common ASDF plugin deps
brew install coreutils automake autoconf openssl libyaml readline libxslt libtool

# Install Erlang plugin deps
brew install unixodbc wxmac

# Install Java. It's optional, but installing it avoids popup prompts.
# If you already have Java installed, you don't need to do this
brew cask install java

# Install Node.js plugin deps
brew install gpg

# Import node gpg keys
# This can be flaky, as it depends on network connections to the GPG key servers
# You may need to run it multiple times
bash ~/.asdf/plugins/nodejs/bin/import-release-team-keyring

See the ASDF Erlang docs for more options.

Install tools

Use ASDF to install the versions of Erlang, Elixir and Node.js specified in the .tool-versions file:

asdf install

You might need to run this twice.

Install Elixir libraries into the ASDF dir for the specific Elixir version you are running:

mix local.hex --if-missing --force
mix local.rebar --if-missing --force

# Install the Phoenix archive (optional), so that you can run e.g. `mix phx.new`
# mix archive.install hex phx_new 1.4.2

Confirm that it works by building the app the normal way:

mix deps.get
mix deps.compile
mix compile

You should be able to run the app locally with:

mix ecto.create

# Webpack (the new hotness)
(cd assets && npm install && node node_modules/webpack/bin/webpack.js --mode development)

# Brunch (old and busted)
(cd assets && npm install && node node_modules/brunch/bin/brunch build)

iex -S mix phx.server
open http://localhost:4000/

Install ASDF on Linux

Following are scripts to install ASDF in a build/CI environment:

Install on Ubuntu 18.04

First, set up the base system and utils:

echo "==> Initialize package manager and install basic utilities"

export DEBIAN_FRONTEND=noninteractive

echo "===> Updating package repos"
apt-get update -qq

echo "===> Installing locale $LANG"
LANG=C apt-get -qq install locales
locale-gen "$LANG"

echo "===> Updating system packages"
apt-get -qq upgrade

echo "===> Installing apt deps"
apt-get -qq install dialog apt-utils

echo "===> Installing utilities"
apt-get -qq install wget curl unzip make git

Next, install build deps:

echo "==> Install ASDF plugin dependencies"

echo "===> Installing ASDF common plugin deps"
apt-get -qq install automake autoconf libreadline-dev libncurses-dev libssl-dev \
    libyaml-dev libxslt-dev libffi-dev libtool unixodbc-dev

echo "===> Installing ASDF Erlang plugin deps"
apt-get -qq install build-essential libncurses5-dev libwxgtk3.0-dev libgl1-mesa-dev \
    libglu1-mesa-dev libpng-dev libssh-dev unixodbc-dev xsltproc fop

echo "===> Installing ASDF Node.js plugin deps"
apt-get -qq install dirmngr gpg

Install on CentOS 7

First, set up the base system repo and utils:

echo "==> Initialize package manager and install basic utilities"

echo "===> Installing EPEL repository"
wget --no-verbose -P /tmp https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install -q -y /tmp/epel-release-latest-7.noarch.rpm

echo "===> Updating package repos"
yum update -y -q

echo "===> Updating system packages"
yum upgrade -y -q --enablerepo=epel

echo "===> Installing utilities"
yum install -y -q wget curl unzip make git

Next, install build deps:

echo "==> Install ASDF plugin dependencies"

echo "===> Installing common ASDF plugin deps"
yum install -y -q automake autoconf readline-devel ncurses-devel openssl-devel \
    libyaml-devel libxslt-devel libffi-devel libtool unixODBC-devel

echo "===> Installing ASDF Erlang plugin deps"
groupinstall -y 'Development Tools' 'C Development Tools and Libraries'
yum install -y -q wxGTK3-devel wxBase3 openssl-devel libxslt \
    java-1.8.0-openjdk-devel libiodbc unixODBC erlang-odbc

echo "===> Installing ASDF Node.js plugin deps"
yum install -y -q install gpg perl perl-Digest-SHA

Install ASDF

Finally, install ASDF, same for Ubuntu and CentOS:

echo "==> Install ASDF and plugins"

if [ ! -d "$HOME/.asdf" ]; then
    echo "===> Installing ASDF"
    git clone https://github.com/asdf-vm/asdf.git "$HOME/.asdf" --branch v0.7.1

    echo -e '\n. $HOME/.asdf/asdf.sh' >> ~/.bashrc
    echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> ~/.bashrc
fi

source "$HOME/.asdf/asdf.sh"

if [ ! -d "$ASDF_DIR/plugins/erlang" ]; then
    echo "===> Installing ASDF erlang plugin"
    asdf plugin-add erlang
fi

if [ ! -d "$ASDF_DIR/plugins/elixir" ]; then
    echo "===> Installing ASDF elixir plugin"
    asdf plugin-add elixir
fi

if [ ! -d "$ASDF_DIR/plugins/nodejs" ]; then
    echo "===> Installing ASDF nodejs plugin"
    asdf plugin-add nodejs

    echo "===> Importing Node.js release team OpenPGP keys to main keyring"
    # This can be flaky
    bash ~/.asdf/plugins/nodejs/bin/import-release-team-keyring
fi

Install build deps

Finally, install Erlang, Elixir, etc. You would generally put it in the "build" phase of your scripts so the .tool-versions file in git controls the versions.

echo "===> Installing build deps with ASDF"
asdf install
# Run it again to make sure all the plugins ran, as there have been issues with return codes in the past
asdf install

Running a local caching DNS for your app

2019-01-04T00:00:00+08:00

When your app is acting as a proxy to back end servers, it it may need to do a DNS lookup to convert the host name of the server into an IP hundreds of times a second. DNS can become the bottleneck for requests. It also puts heavy load on hosting provider DNS servers, which may not be able to take it, or they may get unhappy with you. Hard coding IPs of back end servers is dangerous, as they may change over time.

Running a local caching DNS server on the app server machine caches results of DNS lookups to make them fast and reduce load on external DNS servers.

The local DNS server forwards requests to the upstream DNS servers and caches the results.

Here is an example /etc/named.conf:

options {
    listen-on port 53 { 127.0.0.1; };

    filter-aaaa-on-v4 yes;

    directory     "/var/named";
    dump-file     "/var/named/data/cache_dump.db";
    statistics-file "/var/named/data/named_stats.txt";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    allow-query     { localhost; };
    allow-recursion { 127.0.0.1; };
    allow-transfer { none; };
    notify no;

    minimal-responses yes;

    dnssec-enable yes;
    dnssec-validation yes;
    dnssec-lookaside auto;

    /* Path to ISC DLV key */
    bindkeys-file "/etc/named.iscdlv.key";

    managed-keys-directory "/var/named/dynamic";

    forwarders {
        8.8.8.8;
        8.8.4.4;
    };
    forward only;
};

include "/etc/named/rndc.key";

// controls {};
controls {
    inet 127.0.0.1 allow { localhost; } keys { "rndc-key"; };
};

statistics-channels {
    inet 127.0.0.1 port {{ bind_cache_statistics_port }} allow { 127.0.0.1; };
};

logging {
    // http://www.zytrax.com/books/dns/ch7/logging.html

    channel debug_log {
        file "/var/log/named/debug.log" versions 10 size 10m;
        severity dynamic;
        // debug and above. Assume the global debug level defined by either the command line parameter -d or by running rndc trace
        print-time yes;
        print-severity yes;
        print-category yes;
    };

    channel simple_log {
        file "/var/log/named/bind.log" versions 10 size 1m;
        //severity warning;
        severity notice;
        print-time yes;
        print-severity yes;
        print-category yes;
    };

    channel query_log {
        file "/var/log/named/query.log" versions 10 size 1m;
        severity dynamic;
        //severity warning;
        print-time yes;
        print-severity yes;
        print-category yes;
    };

    category default {
        simple_log;
        // Enable this to get debug logging
        //debug_log;
    };

    category queries {
        // Enable this to get query logging, it's not on for category default
        // query_log;
    };
};

view "localhost" {
    match-destinations { 127.0.0.1; };

    zone "." IN {
        type hint;
        file "named.ca";
    };

    include "/etc/named.rfc1912.zones";
    include "/etc/named.root.key";
};

You should also configure /etc/resolv.conf to specify the caching DNS first, followed by your upstream DNS servers:

nameserver 127.0.0.1
nameserver 8.8.8.8
nameserver 8.8.4.4

Here is an Ansible role that sets up the caching DNS server. It's also in Ansible Galaxy.

Creating an ssh key

2019-01-01T00:00:00+08:00

Instead of using passwords, it's more secure to use ssh keys to control access to servers. It also makes it easier to automate systems using tools like Ansible.

To create an ssh key, on macOS or Linux, open up a terminal and type:

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

Set a pass phrase to protect access to your key (optional but recommended). the desktop will remember your pass phrase in the keyring when you log in so you don't have to enter it every time.

Your ssh key is also used to control access to GitHub. See the GitHub docs.

Elixir and embedded programming presentation

2018-08-15T00:00:00+08:00

Here are the slides for the presentation on Elixir and embedded programming I gave to the Elixir LA user's group.

It introduces embedded programming and how Elixir is a good match for a new generation of embedded systems.

Running Nerves on Amazon EC2

2018-08-04T00:00:00+08:00

I have been looking into the best way to deploy Elixir in the cloud. As part of that, I have been building various AMIs with only the minimum needed to run an Elixir app.

Nerves is a framework for building embedded systems in Elixir. Instead of running a general purpose operating system, it does as much as possible in Elixir. It boots to an init process which starts an Erlang VM, which is then responsible for starting the system.

The traditional Linux boot process has a lot of legacy cruft: shell scripts calling C programs which call kernel APIs to configure the network. With Nerves, we do this in Elixir, using helper programs where necessary.

When Elixir is in charge, we need some way to combine system-level code with the application code on the same VM, handling system updates. That is handled by the Shoehorn library.

This post shows how you can run a Nerves application on EC2.

Nerves on EC2

I created a Nerves "system" for EC2, nerves_system_ec2. It is based on nerves_system_x86_64, adding the drivers needed for EC2 to the kernel and configuring the boot process for the EC2 environment.

I created nerves_init_ec2 to bring up the system, similar to nerves_init_gadget. AWS provides EC2 instance metadata to the running system, accessed via HTTP calls to a special IP address.

nerves_init_ec2 uses this information at runtime to configure the instance. The most important part is configuring the ssh console to use the SSH key pair to access the system remotely.

Building an app for EC2

Following are instructions for how to get a Nerves app running on EC2. I created a simple Nerves app which does all this, hello_nerves_ec2.

Set up build server

I am building the system via an instance in EC2. The build server runs Ubuntu 18.04 on a t2.xlarge instance to have more resources for building the custom nerves system. The build generates a lot of files, so I added a 100GB gp2 EBS volume mounted under my home directory.

To deploy, I write the Nerves firmware image to the disk, then take a snapshot of the volume, turn it into an AMI and launch an instance with it. I attach a 1GB EBS volume to it for the Nerves system under /dev/xvdn.

Install build deps

sudo apt install build-essential automake autoconf git squashfs-tools ssh-askpass
sudo apt install libssl-dev libncurses5-dev bc m4 unzip cmake python xsltproc
sudo apt install libmnl-dev

wget https://github.com/fhunleth/fwup/releases/download/v1.2.3/fwup_1.2.3_amd64.deb
sudo dpkg -i fwup_1.2.3_amd64.deb

Set up ASDF for Erlang and Elixir

git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.5.1
echo -e '\n. $HOME/.asdf/asdf.sh' >> ~/.bashrc
echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> ~/.bashrc

asdf plugin-add erlang
asdf plugin-add elixir

asdf install erlang 21.0
asdf install elixir 1.6.6

asdf global erlang 21.0
asdf global elixir 1.6.6

mix local.hex
mix local.rebar

mix archive.install hex nerves_bootstrap

Get the nerves system

Check out nerves_system_ec2:

git clone https://github.com/cogini/nerves_system_ec2

Create a new Nerves project

mix nerves.new hello_nerves_ec2
cd hello_nerves_ec2

In mix.exs, reference the new nerves system:

defp system("ec2"), do: [{:nerves_system_ec2, path: "../nerves_system_ec2", runtime: false, nerves: [compile: true]}]

Add nerves_init_ec2

Add nerves_init_ec2 to mix.exs deps:

defp deps(target) do
[
  {:nerves_runtime, "~> 0.4"},
  {:nerves_init_ec2, github: "cogini/nerves_init_ec2"},
] ++ system(target)
end

In config/config.exs, add nerves_init_ec2 to the list of applications loaded by Shoehorn:

config :shoehorn,
  init: [:nerves_runtime, :nerves_init_ec2],
  app: Mix.Project.config()[:app]

Configure nerves_init_ec2 if you like. The defaults will bring up a system with an IEx console accessible via ssh on port 22.

Build the project

export MIX_TARGET=ec2
mix deps.get
mix firmware

Burn the firmware to the EBS volume mounted on the build server. Be careful about the device name. In my life I have messed up my system by overwriting my build server disks with firmware files more times than I would like to admit...

mix firmware.burn -d /dev/xvdn

nerves_runtime will initialize the root partition on startup, but we may only get one boot out of an AMI. Create the filesystem in the build environment:

sudo mkfs.ext4 /dev/xvdn4

Configure AWS

At this point, the new Nerves system is all set up on the EBS volume. Now we need to launch an instance from it. We do that using the AWS API, so it can run from anywhere. I normally run it from my dev machine, but you can do it from the build server as well.

In order to talk to the API, we need permissions. When you create an AWS account, you get a "root" account with full permissions, but you should not use it for for everyday operations. You should create an admin user for yourself and a role for your app to run under which gives it access to specific resources.

Go to IAM in the AWS console.

Create a group called Admins and attach policy AdministratorAccess, giving members full access.

Create a user for yourself, e.g. cogini-jake. Under "Access type," check "Programmatic access" and "AWS Management Console access." Set your login password. Click "Next: Permissions" and then "Add user to group", selecting the Admins group. Record the "Access key id" and "Secret access key" now, this is your only chance.

On your local dev machine, set up an AWS profile in ~/.aws/credentials with the keys:

[nerves-dev]
aws_access_key_id = XXX
aws_secret_access_key = YYY

Most AWS client tools will automatically look up the access keys using the profile, so you can control keys on a per-project basis by setting the profile in the environment.

export AWS_PROFILE=nerves-dev

Install the AWS Command Line Interface:

pip install awscli

Create an ssh key pair. Run create-key-pair.sh

bin/create-key-pair.sh nerves

Copy the output to ~/.ssh/nerves.pem and chmod 0600 nerves.pem.

Create an AWS security group (like a firewall) which allows access to the ports on the instance from the Internet. create-security-group.sh opens port 22 for the IEx console and port 80 for HTTP.

bin/create-security-group.sh nerves

Launch the instance

launch-instance-from-volume.sh takes a snapshot of the volume, builds an AMI, then launches an EC2 instance with it.

Edit the script to match your details:

# Name of security group
SECURITY_GROUP=nerves
# Name of instance to create
NAME=hello_nerves_ec2
KEYPAIR=nerves
# Tag instance with owner so admins can clean up stray instances
TAG_OWNER=jake

Run the script, specifying your volume:

bin/launch-instance-from-volume.sh vol-abc123

The script will print the IP of the new instance, or you can get it from the AWS console.

Connect to the instance

ssh -i ~/.ssh/nerves.pem 123.45.67.89

To exit the SSH session, type ~.

You can view the console output in the AWS Management Console EC2 Dashboard with "Actions | Instance Settings | Get System Log". The graphical instance screenshot is available immediately, but the text log takes a few minutes to appear.

Creating nerves_system_ec2

Following is the process I used to create nerves_system_ec2. I basically followed the Nerves documentation for customizing the system.

Make a copy of nerves_system_x86_64 and modify it

git clone https://github.com/nerves-project/nerves_system_x86_64 nerves_system_ec2
cd nerves_system_ec2/
git remote rename origin upstream
git remote add origin git@github.com:cogini/nerves_system_ec2.git
git push origin master

Configure the Nerves system

mix nerves.system.shell

make menuconfig
make savedefconfig

make linux-menuconfig
make linux-update-defconfig

I used same kernel config as for my minimal EC2 system with Buildroot.

Modify the grub.cfg config

The kernel options are the same as nerves_system_x86_64, with a few additions.

Since we can't manually respond to a panic, we just reboot.

set cloud_opts=panic=1 boot.panic_on_fail

Configure hardware options

# https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/nvme-ebs-volumes.html#timeout-nvme-ebs-volumes
set hardware_opts=nvme.io_timeout=4294967295

Set up a serial console, allowing output to be captured in text with "Actions | Instance Settings | Get System Log" or the AWS CLI command aws ec2 get-console-output.

set console_opts=console=tty1 console=ttyS0

This is the resulting kernel command

linux (hd0,msdos2)/boot/bzImage root=PARTUUID=04030201-02 rootwait $console_opts $cloud_opts $hardware_opts

Modify /etc/erlinit.config

Use the serial console:

-c ttyS0
-s "/usr/bin/nbtty"

Modify fwup.conf/fwup-revert.conf

Reduce the size of the user filesystem to match the 1GB volume. In the cloud, we should not be storing data on the instance, everything should be in S3 or a database.

define(APP_PART_COUNT, 1013248)

Servers for beginners

2018-07-06T00:00:00+08:00

Spinning up a server is easy enough, just go to Digital Ocean and push a button. But now you are responsible for your server. What does that mean?

Once you have a server running, you need to take care of a few things:

Data and databases
Monitoring
Security
Maintenance

Data and databases

Cloud servers and automation tools like Ansible make it very easy to set up a server. If the server crashes, we can just build another one in a few minutes. The important thing is protecting the data. Managing that data and making sure it doesn't get lost or stolen is the biggest driver for hosting.

We typically have two kinds of data, uploaded files and databases. In the old days, we would keep data files on the local hard disk, and either install a database locally or on a dedicated database server.

Object storage

These days, it's common to put uploaded files in cloud "object" storage like Amazon S3 or Digital Ocean Spaces. These services are typically very reliable, and we can be comfortable that if we have written the data, it won't be lost in a hard disk crash.

Backups

There is a saying, "The one way to guarantee that you will get fired as a systems administrator is to fail to make backups."

While writing to an object store can generally be considered reliable, we still need to be careful that a program bug doesn't accidentally delete everything. DevOps tools make it almost as easy to delete an S3 bucket as to create it.

It's useful to make periodic snapshots. If you are in AWS, it's probably sufficient to periodically sync data to another S3 bucket, perhaps in a different region. Another option is to sync data to local disk partition and then back it up to Tarsnap or rsync.net.

An important part of backups is testing that they work. It's sadly common to think that you have been making daily backups, then find that they are bad when you need them.

Security

Make sure that the permissions on your S3 buckets are locked down. It's easy to accidentally make them world writable. Enable encryption, so if someone gets access to the data, they still can't read it.

There is a saying that, "Backups are a way of reliably violating file system access permissions at a distance." Make sure that your backups are as secure as your online systems.

Databases

Running a database on the app server machine is easy. For simple applications where the data doesn't change often, you can set up a cron job to make a daily export of the data and back it up to file storage, as described above. You risk losing up to a day of data written since the last backup.

It's generally better to separate the app server from the db server. It makes it easier to upgrade the app server without worrying about the data and provides a bit more security.

If you need higher availability, the traditional solution is to use replication to synchronize the data from your primary database to a backup database. If the primary fails, you promote the backup to primary and point the application to it. You may lose a transaction or two that is in flight, but generally this is quite safe.

Cloud hosting environments like AWS and GCP have database hosting services that take care of the database for you. If you are running in the cloud, that's what you should use. You can run a single database and make periodic snapshot backups, similar to the export process described above. You can also set up a hot backup for high availability. This typically uses the magic of the cloud service's block storage system to replicate data instead of the database native protocol, so it is less complex and performs better.

Monitoring and alerting

You need some way to make sure your server is working other than waiting for your customers to call. You also need to get notified when it is struggling, e.g. having application errors.

Monitoring

There are two sides to monitoring: external and internal.

External monitoring periodically checks that your service is up, e.g. by making an HTTP request to your home page or (better) a special health check URL.

Internal monitoring checks things that are not visible from outside, e.g. disk space usage.

Logging

Your application and other services on the server will generate logs. You need to monitor them for cries for help.

Alerting

When something goes wrong, your monitoring system should notify you. This might be email, SMS, or a post to a Slack group.

Solutions

While you can set up your own home-grown monitoring system, it's easier to use a service like Datadog. In AWS, you can use AWS CloudWatch.

Security

When you put a server on the internet, people will immediately start attacking it. I have seen machines hacked within five minutes because someone set the root password to abc123.

Minimizing surface area

Minimize the amount of software that you are running. Do things in a simple way. If you are not running a piece of software, you don't have to worry about bugs in that software being exploited to hack your system.

Run a firewall that restricts access to the system. For a straightforward application, that means opening port 22 (or an alternative port) for ssh and port 80/443 for web. You might run a mail server like postfix to facilitate sending mail (though a service like AWS SES or Sendgrid would probably be better). Block inbound mail, and you don't have to worry about the server being hacked.

Tighten up defaults

Only allow remote access with ssh keys, not passwords. Don't allow remote logins as root. Only allow ssh access for your own user account, not that of the application. Use strong passwords everywhere. Use a password manager like LastPass to manage them.

Keep up to date

It's tempting to say, "If it ain't broke, don't fix it." Security vulnerabilities are found all the time, though, so it's important to keep your system up to date, or you will be caught by some script kiddie scanning for known vulnerabilities. Apply updates periodically. The cloud makes this easier than it used to be. Run a stable version of Linux, e.g. CentOS or Ubuntu LTS, and you can safely update packages. For major OS updates, spin up a new instance to try it out. This is where having your db separate from your app server makes things easier.

Run separate dev, staging and prod environments

The cloud makes it easy to make new instances. Keep each environment separate, simple and consistent. That way you can keep production data safe.

Avoid vulnerable software

Keep your custom app separate from third party code, or lock it down as much as you can.

Some software is common and easy to hack. WordPress is the biggest example. It is open source, which means that attackers can inspect it to find vulnerabilities. There are lots of free, poorly written plugins. Millions of people run WordPress, so it's easy to scan the internet looking for vulnerable servers.

If you need to run WordPress, use a managed hosting service, e.g. Bluehost. If you just need a simple blog, look at static site generators, e.g. Jekyll or Pelican. Run your blog out of an S3 bucket and there is nothing to hack or to fail.

Principle of least privilege

There are lots of other things that you can do to lock down your app.

Maintenance

If you have the above done right, there is not much that you need to do to keep your server running. The main thing is paying attention to disks filling up, often from log files. Set up log file rotation, e.g. with logrotate.

Don't worry

This may seem a bit overwhelming, but it's not hard once you get the hang of it.

It's like learning to fix your own car. It may not make sense to change your oil every time, but learn how things work. You won't get stuck on the side of the road with a flat tire or ripped off by mechanics.

The Law of Leaky Abstractions says that we need to understand the layers below our application. If you want to be an architect, you need to understand the laws of physics for hosting. Learning how things work allows you to build better applications and save lots of money.

You are now on your way through the Five Stages of Hosting. I got to stage 4.5, running my own VPS and dedicated server hosting business, before I ascended to the cloud. :-)

The impact of network latency, errors, and concurrency on benchmarks

2018-07-06T00:00:00+08:00

The goal of benchmarking is to understand the performance of our system and how to improve it. When we are making benchmarks, we need to make sure that they match real world usage.

In my post on Benchmarking Phoenix on Digital Ocean, changing the concurrent connections and network latency had a big effect on the results. This post goes into more details on why.

Most websites have connections from multiple clients. That's difficult to simulate, so we often make a lot of connections from a small number of clients, maybe just one. It's an unrealistic way to benchmark if we want to test how the server can handle load. It ends up being limited by latency of the connection between the client and the server.

Latency

There is a direct relationship between the latency and the number of requests the server can handle. It is basically just math.

If the server takes 1 ms to handle a request, then the fastest we can do is 1000 requests per second on a single connection, done serially.

1000 ms/sec / 1 ms/request = 1000 requests/sec

That's limited by the server, it is working as hard as it can to process requests. This assumes that network latency is negligible.

If we are testing from a client in the same data center, and round trip latency is 4ms, then each request from the client takes 1 ms for the request and 4 ms waiting for the network.

1000 ms/sec / 5 ms/request = 200 requests/sec

With one client, the server is sitting idle, it could be handling five times as many requests if it had more clients talking with it.

If we add more network latency, say 50ms, then it has a dramatic affect:

1000 ms/sec / 51 ms/request = 19.6 requests/sec

Client concurrency

In order to accurately benchmark what the server can do, we have to add more clients.

We do that in wrk by adding more concurrency:

wrk -t200 -c200 -d60s --latency -s wrk.lua "http://159.89.197.173"

At a certain point, the client performance starts to affect the benchmark. This is particularly true if we benchmark on the same server from localhost. The client takes up resources that the server needs to handle the requests, skewing the results.

Server concurrency

If we only have one CPU, then the server can actually only do one thing at a time. In practice, servers are often waiting on network connections, so we can use a non-blocking I/O framework and do work on other requests when we are waiting. Single-threaded frameworks with non-blocking I/O can handle a lot of clients, as long as they are not doing CPU heavy work. It's efficient and easy to reason about. As processing work increases, they start to have problems because requests have to wait. This is the way that Node.js or Twisted Python work.

More advanced languages like Java, C++ or Go use threads. Each request gets its own independent thread, and the operating system schedules them. Programming can be complex, though, with race conditions accessing shared data. Shared memory means that one thread can step on another thread's memory. There are also limits to how many OS threads can be active at a time, as each takes up system resources.

Server side concurrency is where Elixir really shines. The Erlang VM handles the low level networking using non-blocking I/O. It also schedules Elixir processes across a moderate number of OS threads, allowing it to handle millions of lightweight processes efficiently. This is particularly useful when the server has multiple cores, Elixir will transparently take advantage of them all.

TCP/IP connections and reliability

In that benchmark post, I set max_keepalive: 5_000_000, which told Phoenix to keep the network connection open between requests. This allows the client to send multiple requests on the same TCP/IP connection, avoiding the work and latency from the TCP three-packet handshake (SYN/SYN-ACK/ACK).

This is realistic for interactive web use, where a single user may request multiple pages or other assets. For an API, keep-alive may not be relevant.

Packet loss can have a big effect on the worst case time. If the connection loses a packet and it needs to be retransmitted, then that request will be extra slow. For benchmarking, once the connection loses a packet, it can block the rest of the pipeline, giving poor results, and you may get overall better throughput without it.

The real world can be nasty, though, and tuning according to benchmarks in a clean network can lead to poor application behavior.

Mobile connections may have very high packet loss (> 50%) and/or very high latency (> 500 ms). Retransmits happen both at the cellular-network level and the TCP level, causing pathological network behavior. Running a reliable protocol on top of another reliable protocol results in conflicts with timing of acknowledgements, retries, and windowing.

With 50% packet loss, every time we send a packet, it may get lost. That goes for DNS packets, TCP handshake packets, request packets, response packets, etc. For an API, we are probably best minimizing the number of API requests, putting more data in each request. We rely on the automatic network retransmits to get the data there. If the connections completely die, it can take minutes to give up. We may be better off with a connection per request again.

We can also run out of TCP/IP sockets as we handle more simultaneous requests. As you get more traffic, it's important to configure each part of the system to make sure you have enough sockets, or it will fundamentally limit your application. The defaults are surprisingly small. This is particularly important when we are using web sockets, as described in The Road to 2 Million Websocket Connections in Phoenix

Tuning the bottlenecks

The most important thing to look at when performance tuning is not the average, it's the worst case. That indicates where the problems lie. Sometimes, as described above, it's lost network packets. Other times, it's waiting on a shared bottleneck, typically the database. Sometimes it's a garbage collection pause. The Erlang VM manages memory on a fine-grained per-process basis, so it's particularly good at this.

See this performance tuning presentation for more details.

Managing app secrets with Ansible

2018-06-16T00:00:00+08:00

In web applications we usually have a few things that are sensitive, e.g. the login to the production database or API keys used to access a third party API. We need to be particularly careful about how we manage these secrets, as they may allow attackers to access data without going through the application itself.

There are trade-offs in managing secrets, depending on the size of the organization.

For a small team of developers who are also the admins, then we implicitly trust our devs. We may store the secrets on our dev machine and push them from there to the app servers. It's better not to have secrets in the build environment, though, particularly if it's a third party CI service.

We need to keep the secrets separate from the build, loading them separately on the target system. That might mean putting them in a separate file or reading them at runtime from an external source like an S3 bucket or AWS Parameter Store. Access is controlled by IAM instance roles.

For secure applications like health care or finance, we need to tightly control access to production systems. We can restrict access to your ops team. Ideally nobody would log into production systems, and if they do, there is an audit log.

Ansible vault

The Ansible automation tool a vault function which we can use to store keys. It automates the process of encrypting variable data so we can check it into source control, and only people with the password can read it. It's great for simple deployments with small teams. It has very few moving parts and dependencies, while being reasonably secure.

The following shows describes how you can use the vault.

Configuring the vault key

First, generate a vault key and put it in the file vault.key:

openssl rand -hex 16

You can specify the password when you are running a playbook with the --vault-password-file vault.key option, or you can make the vault password always available by setting it in ansible.cfg:

vault_password_file = vault.key

Defining secrets

There are two ways to store secrets in Ansible variable files. Either we can encrypt the file as a whole, or we can embed encrypted data inline.

To create an encrypted variable file:

ansible-vault create --vault-id=vault.key inventory/group_vars/web_servers/secrets.yml

Add variables normally. When you save the file, it will be encrypted. Later, edit the file like this:

ansible-vault edit --vault-id=vault.key inventory/group_vars/web_servers/secrets.yml

To encrypt a single variable inline:

openssl rand -hex 32 | ansible-vault encrypt_string --vault-id=vault.key --stdin-name 'db_pass'

That generates encrypted data like:

db_pass: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          64346139623638623838396261373265666363643264333664633965306465313864653033643530
          3830366538366139353931323662373734353064303034660a326232343036646339623638346236
          39623832656466356338373264623331363736636262393838323135663962633339303634353763
          3935623562343131370a383439346166323832353232373933613363383435333037343231393830
          35326662353662316339633732323335653332346465383030633333333638323735383666303264
          35663335623061366536363134303061323861356331373334653363383961396330386136636661
          63373230643163633465303933396336393531633035616335653234376666663935353838356135
          36323866346139666462

Copy that into a standard variable file.

Generating templates

Now, you can use Ansible's template function to create a config file template. The template file includes the variables, and Ansible will automatically decrypt vault variables and insert them into the template.

For example, here is a template which configures an Elixir Phoenix app:

[{{ elixir_app_name }}."{{ elixir_app_module }}Web.Endpoint"]
secret_key_base = "{{ secret_key_base }}"

[{{ elixir_app_name }}."{{ elixir_app_module }}.Repo"]
username = "{{ db_user }}"
password = "{{ db_pass }}"
database = "{{ db_name }}"
hostname = "{{ db_host }}"
ssl = {{ db_ssl }}
pool_size = {{ db_pool_size }}

The Ansible task could generate it on the production server:

- name: Create config.toml
  template: src=etc/app/config.toml.j2 dest=/etc/app/config.toml owner={{ app_user }} group={{ app_group }} mode=0644

Generating config files to S3

When deploying in the cloud, you can generate the app config file to an S3 bucket. When the app starts up, it can then sync the config file from the S3 bucket.

---
# Generate app config and upload to S3 bucket
#
# ansible-playbook -v -u $USER --extra-vars "env=$ENV" playbooks/$APP/config-app.yml -D

- name: Generate config file from template and upload to S3
  hosts: localhost
  gather_facts: no
  connection: local
  vars:
    app_name: foo
    comp: app
    file_format: toml
    input_template: ../../templates/{{ app_name }}/{{ comp }}/config.{{ file_format }}.j2
    output_file: config.{{ file_format }}
  vars_files:
    - ../../vars/{{ app_name }}/{{ env }}/common.yml
    - ../../vars/{{ app_name }}/{{ env }}/db-app.yml
    - ../../vars/{{ app_name }}/{{ env }}/app.yml
    - ../../vars/{{ app_name }}/{{ env }}/app-secrets.yml
  tasks:
    - name: Create tempfile
      tempfile:
        state: file
      register: temp_file

    # - debug: var=temp_file.path

    - name: Fill template to tempfile
      template:
        src: "{{ input_template }}"
        dest: "{{ temp_file.path }}"
      no_log: true

    - name: Put config to S3
      aws_s3:
        bucket: "{{ config_bucket }}"
        object: "{{ config_bucket_prefix }}/{{ output_file }}"
        src: "{{ temp_file.path }}"
        mode: put

    - name: Delete tempfile
      file:
        state: absent
        path: "{{ temp_file.path }}"

Improving app security with the principle of least privilege

2018-06-11T00:00:00+08:00

The security principle of "least privilege" means that apps should only have the permissions that they need to do their job, nothing more. If an attacker compromises your app, then they can't do anything outside of what the app would normally do. They may be able to break the application, but they can't use it as a stepping stone to attack other systems.

For example, a normal web application might need to be able to respond to HTTP requests, query and update a database, handle file uploads, and write log messages.

Separate app user and deploy user

The app has certain things that it needs to do at runtime, e.g. read and write files. Create a separate user for it to run under which only has those permissions. Use a different user account to deploy and manage the app.

The app account is not shared with anything else, so we can use user permissions to control which files the app can read and limit the system resources it can use. The app needs to be able to read its own source or binary files, but it doesn't need to be able to write them. Have the files owned by the deploy user and use group permissions to give the app access to them. One app should not be able to read and write another app's files. Tighten up permissions on directories to disallow read access to world.

When deploying the app, we need to be able to restart it. Give those permissions to the deploy user, not the app. Even better, instead of giving sudo to the deploy user, use file triggers to restart. This lets you deploy from e.g. a CI/CD system with limited trust.

Restrict permissions with systemd

By default, users are allowed to do quite a lot, e.g. open outbound network connections. If you don't need these rights, restrict them on a per-user or per-app basis. Systemd has many features to restrict what apps are allowed to do. It provides a relatively easy declarative interface to traditional Unix features like chroot(2). It also supports modern Linux features which provide more ways to restrict access in a fine grained way. You can use SELinux to restrict even more.

File uploads

In order to handle file local uploads, the app needs to be able to write to a location on the local disk. We can create a directory like /var/lib/myapp/uploads and make it writable by the app user. If all we need to do is receive data, it's not too bad, but we usually need to show data to other users. An attacker may try to upload a file to that directory, then convince the system to run it as code.

If our filters are not good, they might be able to upload a standalone PHP script. More tricky, if you allow users to upload an avatar image, then the attacker might embed some PHP code in their image file, then have web server execute it. Configure the web server to handle user content only as data.

The OS can help as well. Set the umask to keep the app from being able to create executable files. Make a separate file system for the uploads, then mount it with the noexec option, and the OS will stop executable files from running.

There are plenty of additional attacks where the data from one user can be used to attack another user, e.g. by embedding JavaScript, but that's at a different layer in the stack.

Using a Content Delivery Network

Even better is if we don't serve static files from the app server at all. Upload the files to an S3 bucket, then use a CDN like CloudFront to deliver the files. That is more secure, plus it's faster and cheaper than serving static files from an app server.

Signed URLs

Instead of giving the app permission to write files to the S3 bucket, give the user a signed URL which allows them to upload data directly to S3. The app doesn't need to process the files, so it doesn't need permission to read or write them at all. Again, faster and cheaper as well.

Database access

The app needs an account and password to be able to connect to the database. Some apps might just need read only access, but most need to update tables as well. We can still use the database permissions system to restrict what the app db user can do.

With PostgreSQL, remove role permissions like SUPERUSER, CREATEDB, CREATEROLE. Make the db schema owned by a different database user from the app user, then restrict the app user from creating or altering tables. When deploying the app, run database migrations as the user that owns the schema, not the app user.

In enterprise apps, it's common to share a database between multiple apps. Use database views to create read-only views on data, restricting data from sensitive columns.

Writing logs

In order to write a request log, we normally give the app user write access to a directory like /var/log/myapp. That also means that an attacker can read potentially sensitive information from the logs or overwrite them, covering his or her tracks.

Instead of writing our own log files, use journald to manage logs. The app writes its logs to standard out, and systemd redirects them to the journal. The app can also use the journald logging API to write structured log messages, including metadata. We then pull that data out in real time and send it to a log aggregation system like Elasticsearch/Logstash/Kibana and generate real-time alerts on application errors or attacks.

Listening on a non-privileged port

Listening on TCP/IP ports below 1024 requires root permissions. In the early days of the Internet, it was common to have programs start as root, bind to the port, then (hopefully) drop privileges. The result was e.g. the "Morris worm" which exploited a buffer overflow in the "sendmail" mail server, then turned around to attack other machines.

We normally run our apps behind a proxy like Nginx. The proxy is still running with elevated permissions, though. Better is if we redirect traffic from port 80 to the app in the firewall using an iptables rule. Nothing runs as root.

Egress filtering

We normally use firewalls and security groups to restrict inbound traffic, but we can also use them to restrict outbound traffic. Set up firewall rules which allow only the intended activity of the app. If someone hacks your system and gets access to data, make it hard to get it out. Make it impossible to use your app server to probe other internal systems looking for vulnerabilities or attack other sites on the internet. Whitelist IPs to make sure that requests are coming from the correct place.

Restrict access with IAM roles

When your app is running in a cloud environment like AWS, it needs to access various resources like S3 buckets. Instead of putting AWS keys on your instance where they can be stolen, assign an instance role which implicitly gives it access to resources at runtime. Amazon is making it possible to access databases using IAM roles. You can restrict access by source IP, so even if an attacker gets your keys, they can't access resources from outside the instance. The IAM instance role can give it access to encryption keys in KMS, giving it access to encrypted S3 buckets or API keys for third party services from Parameter Store.

Putting it into practice

As you can see, there are lots of ways to restrict the deployment environment to make life harder for attackers. The key is to think about exactly what your application needs to do, then try to find ways to make sure that it can only do that.

There is much more to security than just the app itself. This is a "defense in depth" approach. Security issues are inevitable, we need to limit their impact. Attackers must break multiple layers in order to compromise your app and make use of data and credentials that they obtain.

Similarly, we need to monitor systems so that we know when they are attacked or parts have been compromised. Add an audit trail to identify the source of attacks, e.g. compromised user accounts or hostile internal users. Restrict access to data so that there is a data breach, we can identify exactly which data was leaked.

Running special versions of Erlang with ASDF and kerl

2018-06-07T00:00:00+08:00

If you want to try out the new features in Erlang 21 before it's released, you will have to build it yourself, as there is no package available. Same thing if you want to run a patch release or configure Erlang with special options.

The ASDF version manager lets you have multiple versions of Erlang, Elixir and Node.js on your machine at one time, choosing the version based on the .tool-versions config file.

Our Elixir deploy template has instructions for setting up ASDF.

List the available versions:

asdf list-all erlang

Install the release candidate:

asdf install erlang 21.0-rc2

As pointed out by Jared Smith (@sublimecoder, you can also point to a git ref in the code base on GitHub:

asdf install erlang ref:master

You can use any branch SHA or tag on the repo, or switch to your local fork. See the ASDF Erlang docs or the kerl docs for more info.

Deploying your Phoenix app to Digital Ocean for beginners

2018-05-26T00:00:00+08:00

This is a gentle introduction to getting your Phoenix app up and running on a $5/month server at Digital Ocean. It starts from zero, assuming minimal experience with servers. It assumes you are running macOS. If you have any questions, open an issue on GitHub or ping me on the #elixir-lang IRC channel on Freenode, I am reachfh.

It is based on this working template and the principles in "Best practices for deploying Elixir apps". The README for the template is very similar, but goes into more depth.

It starts with a default Phoenix project with PostgreSQL database. First get the template running, then add the changes to your own project. This guide works with CentOS 7, Ubuntu 16.04, Ubuntu 18.04 and Debian 9.4. If you are not sure which distro to use, choose CentOS 7. The approach here works file dedicated servers and cloud instances as well.

It uses Ansible, which is an easy-to-use standard tool for managing servers. Unlike edeliver, it has reliable and well documented primitives to handle logging into servers, uploading files and executing commands.

Overall approach

Set up the web server
Set up a build environment on the server
Check out code on the server from git and build a release
Deploy the release to the web server

The actual work of checking out and deploying is handled by simple shell scripts which you run on the build server or from your dev machine via ssh, e.g.:

# Check out latest code and build release on server
ssh -A deploy@build-server build/deploy-template/scripts/build-release.sh

# Deploy release
ssh -A deploy@build-server build/deploy-template/scripts/deploy-local.sh

Set up dev machine

Check out the project from git on your local dev machine, same as you normally would:

git clone https://github.com/cogini/elixir-deploy-template

Install build tools

Install Erlang, Elixir and Node.js according to the instructions on the Elixir website.

The code is tested with Erlang version 20.3, Elixir 1.6.6 and Node.js 8.2.1, so installing those versions is best. I generally recommend using ASDF to make your build environment more isolated and consistent, but it's not mandatory.

Confirm that it works by building the app the normal way:

mix deps.get
mix deps.compile
mix compile

You should be able to run the app locally with:

mix ecto.create
(cd assets && npm install && node node_modules/brunch/bin/brunch build)

iex -S mix phx.server
open http://localhost:4000/

Install Ansible

Install Ansible on your dev machine. On macOS, use pip, the Python package manager:

sudo pip install ansible

If pip isn’t already installed, run:

sudo easy_install pip

See the Ansible docs for other options.

Generate an ssh key

We use ssh keys to control access to servers instead of passwords. This is more secure and easier to automate.

Generate an ssh key if you don't have one already:

ssh-keygen -t rsa -b 4096 -C "your_email@example.com"

Set a pass phrase to protect access to your key (optional but recommended). macOS and modern Linux desktops will remember your pass phrase in the keyring when you log in so you don't have to enter it every time.

Add the ~/.ssh/id_rsa.pub public key file to your GitHub account. See the GitHub docs.

Set up a server

Go to Digital Ocean (affiliate link) and create a Droplet (virtual server).

Choose an image: If you are not sure which distro to use, choose CentOS 7
Choose a size: The smallest, $5/month Droplet is fine
Choose a datacenter region: Select a data center near you
Add your SSH keys: Select the "New SSH Key" button, and paste the contents of your ~/.ssh/id_rsa.pub file
Choose a hostname: The default name is fine, but a bit awkward to type. Use "web-server" or whatever you like.

The defaults for everything else are fine. Click the "Create" button.

Add the host to the ~/.ssh/config file on your dev machine:

Host web-server
    HostName 123.45.67.89

The file permissions on ~/.ssh/config need to be secure or ssh will be unhappy:

chmod 600 ~/.ssh/config

Test it by connecting to the server:

ssh root@web-server

If it doesn't work, run ssh with -v flags to see what the problem is. You can add more verbosity, e.g. -vvvv if you need more detail.

ssh -vv root@web-server

File permissions are the most common cause of problems with ssh. Another common problem is forgetting to add your ssh key when creating the Droplet. Destroy the Droplet and create it again.

Configure Ansible

Add the hosts to the groups in the Ansible inventory ansible/inventory/hosts file in the project:

[web-servers]
web-server

[build-servers]
web-server

[db-servers]
web-server

[web-servers] is a group of web servers. web-server is the hostname from the Host line in your .ssh/config file. [build-servers] is the group of build servers. It can be the same as your web server. [db-servers] is the group of database servers, which can be the same.

If you are using Ubuntu or Debian, add the host to the [py3-hosts] group, and it will use the Python 3 interpreter that comes by default on the server.

The repo has multiple hosts in the groups for testing different OS versions, comment them out.

Set Ansible variables

The configuration variables defined in inventory/group_vars/all apply to all hosts in your project. They are overridden by vars in more specific groups like inventory/group_vars/web-servers or for individual hosts, e.g. inventory/host_vars/web-server.

Ansible uses ssh to connect to the server. These playbooks use ssh keys to control logins to server accounts, not passwords. The users Ansible role manages accounts.

The inventory/group_vars/all/users.yml file defines a global list of users and system admins. It has a live user (me!), change it to match your details:

users_users:
  - user: jake
    name: "Jake Morrison"
    github: reachfh

users_global_admin_users:
 - jake

The inventory/group_vars/all/elixir-release.yml file specifies the app settings:

# External name of the app, used to name directories and the systemd process
elixir_release_name: deploy-template

# Internal "Elixir" name of the app, used to by release to name things
elixir_release_name_code: deploy_template

# Name of your organization or overall project, used to make a unique dir prefix
elixir_release_org: myorg

# OS user the app runs under
elixir_release_app_user: foo

# OS user for building and deploying the code
elixir_release_deploy_user: deploy

# Port that Phoenix listens on
elixir_release_http_listen_port: 4001

The inventory/group_vars/build-servers/vars.yml file specifies the build settings.

It specifies the project's git repo, which the Ansible playbook will check out on the build server:

# App git repo
app_repo: https://github.com/cogini/elixir-deploy-template

Set up web server

Run the following Ansible commands from the ansible dir in the project.

Do initial server setup:

ansible-playbook -u root -v -l web-servers playbooks/setup-web.yml -D

In this command, web-servers is the group of servers, but you could also specify a specific host like web-server. Ansible allows you to work on groups of servers simultaneously. Configuration tasks are generally written to be idempotent, so we can run the playbook against all our servers and it will make whatever changes are needed to get them up to date.

The -u flag specifies which user account to use on the server. We have to use root to do the initial bootstrap, but you should generally use your own user account, assuming it has sudo. The -v flag controls verbosity, you can add more v's to get more debug info. The -D flag shows diffs of the changes Ansible makes on the server. If you add --check to the Ansible command, it will show you the changes it is planning to do, but doesn't actually run them. These scripts are safe to run in check mode, but may give an error during the play if required OS packages are not installed.

Set up the app (create dirs, etc.):

ansible-playbook -u root -v -l web-servers playbooks/deploy-app.yml --skip-tags deploy -D

Configure runtime secrets, setting the $HOME/.erlang.cookie file and generate a Conform config file at /etc/deploy-template/deploy_template.conf:

ansible-playbook -u root -v -l web-servers playbooks/config-web.yml -D

For ease of getting started, this generates secrets on your local machine and stores them in /tmp. See below for discussion about managing secrets.

At this point, the web server is set up, but we need to build and deploy the app code to it.

Set up build server

We assume this is the same as the web server, but it can be different.

Set up the server:

ansible-playbook -u root -v -l build-servers playbooks/setup-build.yml -D

This sets up the build environment, e.g. install ASDF.

Install PostgreSQL, assuming we are running the web app on the same server.

ansible-playbook -u root -v -l build-servers playbooks/setup-db.yml -D

Configure config/prod.secret.exs on the build server:

ansible-playbook -u root -v -l build-servers playbooks/config-build.yml -D

Again, see below for discussion about managing secrets.

Build the app

Log into the deploy user on the build machine:

ssh -A deploy@build-server
cd ~/build/deploy-template

The -A flag on the ssh command gives the session on the server access to your local ssh keys. If your local user can access a GitHub repo, then the server can do it, without having to put keys on the server. Similarly, if your ssh key is on the prod server, then you can push code from the build server using Ansible without the web server needing to trust the build server.

Build the production release:

scripts/build-release.sh

That script runs:

echo "Pulling latest code from git"
git pull

echo "Updating versions of Erlang/Elixir/Node.js if necessary"
asdf install
asdf install

echo "Updating Elixir libs"
mix local.hex --if-missing --force
mix local.rebar --if-missing --force
mix deps.get --only "$MIX_ENV"

echo "Compiling"
mix compile

echo "Updating node libraries"
(cd assets && npm install && node node_modules/brunch/bin/brunch build)

echo "Building release"
mix do phx.digest, release

asdf install builds Erlang from source, so the first time it runs it can take a long time. If it fails due to a lost connection, delete /home/deploy/.asdf/installs/erlang/20.3 and try again. You may want to run it under tmux.

Deploy the release locally

If you are building on the web web server, then you can use the custom mix tasks in lib/mix/tasks/deploy.ex to deploy locally.

In mix.exs, set deploy_dir to match Ansible, i.e. deploy_dir: /opt/{{ org }}/{{ elixir_release_name }}:

deploy_dir: "/opt/myorg/deploy-template/",

Deploy the release:

scripts/deploy-local.sh

That script runs:

MIX_ENV=prod mix deploy.local
sudo /bin/systemctl restart deploy-template

The build is being done under the deploy user, who owns the files under /opt/myorg/deploy-template and has a special /etc/sudoers.d config which allows it to run the /bin/systemctl restart deploy-template command.

Verify it works

Make a request to the app supervised by systemd:

curl -v http://localhost:4001/

Have a look at the logs:

systemctl status deploy-template
journalctl -r -u deploy-template

Make a request to the machine over the network on port 80 through the magic of iptables port forwarding.

You can get a console on the running app by logging in as the foo user the app runs under and executing:

/opt/myorg/deploy-template/scripts/remote_console.sh

You can also deploy to a remote machine using Ansible.

Managing secrets with Ansible

Ansible has a vault function which you can use to store keys. It automates the process of encrypting variable data so you can check it into source control, so only people with the password can read it.

There are trade-offs in managing secrets.

For a small team of devs who are also the admins, then you trust your developers and your own dev machine with the secrets. It's better not to have secrets in the build environment, though. You can push the prod secrets directly from your dev machine to the web servers. If you are using a third-party CI server, then that goes double. You don't want to give the CI service access to your production keys.

For secure applications like health care or finance, we need to tightly control access to production systems. Ideally nobody would log into production systems, and if they do, it should be logged. You can restrict vault password access to your ops team, or use different keys for different environments.

You can also set up a build/deploy server in the cloud which has access to the keys and configure the production instances from it. When we run in an AWS auto scaling group, we build an AMI with Packer and Ansible, putting the keys on it the same way. Even better, however, is to not store keys on the server at all. Pull them when the app starts up, reading from an S3 bucket or Amazon's KMS, with access controlled by IAM instance roles.

The one thing that really needs to be there at startup is the Erlang cookie, everything else we can pull at runtime. If we are not using the Erlang distribution protocol, then we don't need to share it, it just needs to be secure.

The following shows describes how you can use the vault.

Generate a vault password and put it in the file ansible/vault.key:

openssl rand -hex 16

You can specify the password when you are running a playbook with the --vault-password-file vault.key option, or you can make the vault password always available by setting it in ansible/ansible.cfg:

vault_password_file = vault.key

The ansible/inventory/group_vars/web-servers/secrets.yml file specifies deploy secrets.

Generate a cookie for deployment and copy it into the secrets.yml file:

openssl rand -hex 32 | ansible-vault encrypt_string --vault-id vault.key --stdin-name 'erlang_cookie'

That generates encrypted data like:

erlang_cookie: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          64346139623638623838396261373265666363643264333664633965306465313864653033643530
          3830366538366139353931323662373734353064303034660a326232343036646339623638346236
          39623832656466356338373264623331363736636262393838323135663962633339303634353763
          3935623562343131370a383439346166323832353232373933613363383435333037343231393830
          35326662353662316339633732323335653332346465383030633333333638323735383666303264
          35663335623061366536363134303061323861356331373334653363383961396330386136636661
          63373230643163633465303933396336393531633035616335653234376666663935353838356135
          36323866346139666462

Generate secret_key_base for the server the same way:

openssl rand -base64 48 | ansible-vault encrypt_string --vault-id vault.key --stdin-name 'secret_key_base'

Generate db_pass for the db user:

openssl rand -hex 16 | ansible-vault encrypt_string --vault-id vault.key --stdin-name 'db_pass'

This playbook configures the production server, setting the $HOME/.erlang.cookie file on the web server and generates a Conform config file at /etc/deploy-template/deploy_template.conf with the other vars:

ansible-playbook --vault-password-file vault.key -u $USER -v -l web-servers playbooks/config-web.yml -D

This playbook configures config/prod.secret.exs on the build server.

ansible-playbook --vault-password-file vault.key -u root -v -l build-servers playbooks/config-build.yml -D

Database

Most apps use a database. The Ansible playbook playbooks/setup-db.yml creates the database for you.

Whenever you change the db schema, you need to run migrations on the server.

After building the release, but before deploying the code, update the db to match the code:

scripts/db-migrate.sh

That script runs:

MIX_ENV=prod mix ecto.migrate

Surprisingly, the same process also works when we are deploying in an AWS cloud environment. Create a build instance in the VPC private subnet which has permissions to talk to the RDS database. Run the Ecto commands to migrate the db, build the release, then do a Blue/Green deployment to the ASG using AWS CodeDeploy.

Changes

Following are the steps used to set up this repo. You can do the same to add it to your own project.

It all began with a new Phoenix project:

mix phx.new deploy_template

Set up release

Generate initial files in the rel dir:

mix release.init

Modify rel/config.exs and vm.args.eex.

Set up ASDF

Add the .tool-versions file to specify versions of Elixir and Erlang.

Configure for running in a release

Edit config/prod.exs

Uncomment this so Phoenix will run in a release:

config :phoenix, :serve_endpoints, true

Add Ansible

Add the Ansible tasks to set up the servers and deploy code, in the ansible directory. Configure the vars in the inventory.

This repository contains local copies of roles from Ansible Galaxy in roles.galaxy. To install them, run:

ansible-galaxy install --roles-path roles.galaxy -r install_roles.yml

Add mix tasks for local deploy

Add lib/mix/tasks/deploy.ex

Add Conform for configuration

Add Conform to deps in mix.exs:

 {:conform, "~> 2.2"}

Generate schema to the config/deploy_template.schema.exs file.

MIX_ENV=prod mix conform.new

Generate a sample deploy_template.prod.conf file:

MIX_ENV=prod mix conform.configure

Integrate with release by adding plugin Conform.ReleasePlugin to rel/config.exs:

release :deploy_template do
  set version: current_version(:deploy_template)
  set applications: [
    :runtime_tools
  ]
  plugin Conform.ReleasePlugin
end

Benchmarking Phoenix on Digital Ocean

2018-05-18T00:00:00+08:00

Just for fun, I decided to benchmark the performance of the elixir deploy template running on a $5/month Digital Ocean Droplet.

Following Saša Jurić's post, I set up wrk on my Mac and on some Digital Ocean instances in the same data center.

I made a simple request function in wrk.lua:

request = function()
  wrk.method = "GET"
  return wrk.format(nil, "/")
end

This is just reading the default Phoenix home page. There is no actual logic performed, or more importantly, database calls.

With zero tuning of Phoenix, from my Mac in Taiwan going to Digital Ocean in Singapore, I got the following results:

wrk -t12 -c12 -d60s --latency -s wrk.lua "http://159.89.197.173"
Running 1m test @ http://159.89.197.173
  12 threads and 12 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    73.06ms   78.84ms   1.18s    96.43%
    Req/Sec    15.63      5.09    20.00     63.53%
  Latency Distribution
     50%   60.49ms
     75%   62.98ms
     90%   72.32ms
     99%  454.03ms
  11099 requests in 1.00m, 24.06MB read
Requests/sec:    184.77
Transfer/sec:    410.20KB

The response time is all driven from the network latency:

ping 159.89.197.173
PING 159.89.197.173 (159.89.197.173): 56 data bytes
64 bytes from 159.89.197.173: icmp_seq=0 ttl=49 time=56.037 ms
64 bytes from 159.89.197.173: icmp_seq=1 ttl=49 time=55.475 ms
64 bytes from 159.89.197.173: icmp_seq=2 ttl=49 time=55.455 ms

The actual processing time of Phoenix is in the microsecond range:

May 16 10:07:05 elixir-deploy-template deploy-template[29275]: 10:07:05.777 request_id=5d6bcb89q0jv834pu7d39fmaaq828opg [info] GET /
May 16 10:07:05 elixir-deploy-template deploy-template[29275]: 10:07:05.777 request_id=5d6bcb89q0jv834pu7d39fmaaq828opg [info] Sent 200 in 142µs
May 16 10:07:05 elixir-deploy-template deploy-template[29275]: 10:07:05.781 request_id=mbrp97u5btbvike3057ho7h4ao9j4jfd [info] GET /
May 16 10:07:05 elixir-deploy-template deploy-template[29275]: 10:07:05.781 request_id=mbrp97u5btbvike3057ho7h4ao9j4jfd [info] Sent 200 in 251µs

The machine itself is not working hard at all. The worst case latency is driven by occasional network glitches, e.g. lost packets.

I did a bit of tuning, changing the log level so that we are not writing each request to the log twice, and also changing the max_keepalive so that we run multiple requests on the same connection. Reusing connections is realistic if you have users doing multiple things on your site. It mainly affects the max latency stats as compared to the average latency.

git diff
diff --git a/config/prod.exs b/config/prod.exs
index 1cf2f5a..7acf2d7 100644
--- a/config/prod.exs
+++ b/config/prod.exs
@@ -14,12 +14,16 @@ use Mix.Config
 # manifest is generated by the mix phx.digest task
 # which you typically run after static files are built.
 config :deploy_template, DeployTemplateWeb.Endpoint,
+  http: [port: 4001,
+    protocol_options: [max_keepalive: 5_000_000]
+  ],
   load_from_system_env: true,
   url: [host: "example.com", port: 80],
   cache_static_manifest: "priv/static/cache_manifest.json"

 # Do not print debug messages in production
-config :logger, level: :info
+#config :logger, level: :info
+config :logger, level: :warn

Cranking up the concurrency makes things more interesting:

wrk -t200 -c200 -d60s --latency -s wrk.lua "http://159.89.197.173"
Running 1m test @ http://159.89.197.173
  200 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency    90.91ms  106.45ms   1.12s    95.02%
    Req/Sec    14.06      5.31    20.00     73.44%
  Latency Distribution
     50%   67.35ms
     75%   73.49ms
     90%   83.67ms
     99%  681.16ms
  162932 requests in 1.00m, 353.22MB read
Requests/sec:   2710.91
Transfer/sec:      5.88MB

Now we are actually making the server work, and it's CPU bound during the run.

Next I ran the same test from another Droplet in the same data center:

wrk -t12 -c12 -d60s --latency -s wrk.lua "http://159.89.197.173"
Running 1m test @ http://159.89.197.173
  12 threads and 12 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     3.54ms    0.91ms  21.62ms   82.76%
    Req/Sec   284.03     29.73   414.00     74.49%
  Latency Distribution
     50%    3.35ms
     75%    3.95ms
     90%    4.48ms
     99%    6.53ms
  203770 requests in 1.00m, 441.75MB read
Requests/sec:   3393.59
Transfer/sec:      7.36MB

ping 159.89.197.173
PING 159.89.197.173 (159.89.197.173) 56(84) bytes of data.
64 bytes from 159.89.197.173: icmp_seq=1 ttl=61 time=1.36 ms
64 bytes from 159.89.197.173: icmp_seq=2 ttl=61 time=0.439 ms
64 bytes from 159.89.197.173: icmp_seq=3 ttl=61 time=0.431 ms
64 bytes from 159.89.197.173: icmp_seq=4 ttl=61 time=0.457 ms

On the whole, it's quite impressive performance for such a cheap instance.

SaaS pricing: users are not all the same

2018-05-18T00:00:00+08:00

It's popular these days to use hosted applications instead of running your own infrastructure. It's frustrating as a customer, though, when the pricing model is not sophisticated enough to match your actual usage.

In a SaaS product, your pricing should scale with the value the customer gets from the product. You want to charge big customers more while keeping it affordable for small customers, and you need to enable early stages of usage in big companies such as pilots or "viral" adoption.

If the value scales with the number of users, then charging per user makes sense. If you are making an app to help manage fleet of vehicles, then charging per vehicle may make more sense.

You also want natural "price breaks" which separate your entry level users from the features in the "pro" or "enterprise" plans. People who have 50 users have different problems than ones that have five, and you can charge them more if you are providing more total value.

Users are not all the same

There is a SaaS industry trend towards per-user pricing.

The fundamental problem comes when you have users within a customer account which are getting different amounts of value from your product. There is also a danger of alienating the small customers and technical users who have the potential to give them viral growth.

For example, the GitHub code hosting and collaboration platform announced a change in their pricing model that makes things difficult for consulting companies like ours. Instead of charging for the number of projects hosted, they charge per user per month. If the users are all active developers, then customers are getting value from the platform no matter how many projects there are, and it scales naturally with the size of the customer's organization.

The new change, however, increases our costs significantly without increasing value because we have inactive users. If we host a client's project in our GitHub organization, then we need accounts for our developers and one or two for client staff. When we are actively developing the project, then that's fine. If the project launches and switches to a maintenance phase, then we end up paying too much.

If the project is not active, paying for our developers is fine, as they will be working on other projects, but we are also paying for the inactive client accounts. If we take away client access to their source code, then they (reasonably) freak out. If we archive the repo, then it's same thing, and it's an administrative hassle for maintenance.

If the client has their own organization, then it's worse. They pay for their own inactive users and also for our inactive users. So they can remove our users and save the money, but that's a pain if we need to do some maintenance. When you don't have a lot of activity, it's like every bug fix costs $10 vig to GitHub to add the user for a month.

We see similar issues with production incident management products. I am ok with paying $50/month per on-call ops person, as we are getting value from the product. I don't like paying $50/month per developer who might potentially need to deal with a problem that gets escalated, or a product owner who might need to see what the status is. These users do not have the same level of interaction with the product.

Failure to get this right can result in people setting up their system to minimize costs rather than using it "properly." For security, users should have their own logins with roles that allow them to do their job. A sign that your pricing model has problems is when you see people sharing sharing account logins. For example, if you use GitHub's issue tracker, then anyone in your company might need to be able to create a bug report. Paying $10/user/month for that makes no sense, so we end up with a shared account. That's inconvenient, though, e.g. users can't get emails notifying them when the developer responds.

In your SaaS product, you may separate the management of "billable" resources, giving the owner control over how much money is being spent. For example, the owner can create resources that cost money each month, while the primary users manage them.

Sometimes the SaaS pricing seems like its own bubble world and we forget that there are other ways of doing the same thing. If I was to buy a little time tracking app that ran on my computer, I would expect to pay say $19.95, not $120 per year. For a while we used Harvest for time tracking, but it was $12 per user per month. The ROI on us making our own solution was a month or two, and we got to integrate it with processes like invoicing and accounting to make it work better for us.

Avoiding GenServer bottlenecks

2018-05-17T00:00:00+08:00

GenServers are the standard way to create services in Elixir. They are very useful, but when used incorrectly they can cause unnecessary problems. This is particularly an issue for developers coming from object oriented languages, who attempt to treat GenServers as object instances. Instead we should think in functional terms of data and transformation, shared state and concurrent access.

At it's heart, a GenServer is a separate process that receives a message, does some work, updates process state, then sends back a response. If that matches your problem, great. It's important to recognize, though, that a GenServer only handles one request at a time, and they can become a bottleneck for your system. The Erlang system has other tools available which may work better.

Following are some examples of how GenServers became bottlenecks in high volume systems, and how we resolved them.

Example: Geoip lookups

In a web application we needed to determine which country the request was coming from based on the IP address. MaxMind has various databases related to IP addresses. They are a binary file format which supports efficient querying by network prefix.

The database file we were using is about 65MB in size. Rather than read the data from disk on every request, our initial design was to put it in a GenServer. When the application starts, the GenServer loads the data file into its state. After that, the processes handling HTTP requests send it a "call" request with the IP address. It loads the data from state, looks up the country and returns it to the caller.

That worked fine for a while, but at a certain point, we started getting timeouts. A GenServer only handles a single request at a time, so it had become the bottleneck. We were effectively forcing all the requests in the system to line up and go through the GenServer process one by one.

To avoid that, we switched to a process pool. Using the Episcina library, we ran multiple instances of the GenServer. The process handling a request would check out a server from the pool, call the server to get the data, then put the process back in the pool.

That worked for a while, but eventually the lookups became the bottleneck for the system again. We added more and more servers to the queue, but it didn't help. At first we thought it was the queue manager, as it was a GenServer, too. We needed to send multiple messages: one to check out the GenServer, one to run the request, then another check it back in. The message passing is actually quite fast, though.

The bigger issue was how many processes we should have in the pool and how to manage them. Our peak load was driven by traffic spikes, particularly DDOS attacks. We would get sustained traffic of 5-10K requests per second, with spikes above that.

If the pool starts with a small number of processes, then there is a delay launching new processes as we read the data from the disk. Sometimes we would launch hundreds of processes at once in response to demand, all of them fighting for the same disk. If we pre-loaded lots of processes, then we would use a lot of RAM, and our startup time was poor.

The solution to this, like a lot of Elixir performance issues, was to use ETS. ETS stands for "Erlang Term Storage." It is an in-memory key/value database built into the Erlang virtual machine and highly optimized for concurrent access between multiple processes. It works on Erlang "terms," i.e. data structures, so there is no serialization overhead. Lookup times in ETS are less than one microsecond, which makes them 1000 times faster than something like Redis.

On startup, we load the geoip data into an ETS table. Then, in the process that handles the HTTP request, we load the data from the ETS table and do the lookup on the data blob. You might think that would be inefficient due to copying data around, but the Erlang virtual machine has optimized the process of sharing binary data. If a binary is larger than 64 bytes, it gets stored in a shared binary heap. In fact, we are just passing around a reference to the binary data between ETS and the process. This is a case where immutable data is a big win.

After this optimization, our worst case geoip lookups were taking five microseconds, and our memory usage dropped a lot. That was pretty good, but when we are under DDOS attack, we may get a lot of requests from the same IPs. We added a second ETS table to cache the results of the lookup, getting the time to less than one microsecond.

Principles

This is a good example of the principle of "model the natural concurrency of your application." We had created a lot of processes to manage the geoip data and lookups, and we had overhead talking to them. The number of processes was different from the number of requests.

For each HTTP request, we have a Cowboy process that does the work, then goes back into a pool. The right answer was to do all the work associated with the request in this process. We don't have the overhead and latency of dealing with the queue manager or sending messages to the GenServer.

Another principle is that we should restrict load at the edge of the system. If we don't have enough resources to handle a request, we should reject it rather than overloading the bottleneck and making the system fail (see below). When we use the HTTP thread, it's possible (though not required) in Cowboy to limit the number of acceptor processes. So if we can handle 1000 requests per second, we can limit it at the HTTP layer, causing the requests to be queued by the kernel in the TCP/IP layer. That in turn gives backpressure to clients of the system.

One common case where we really do need to limit concurrent access is when we are talking to a database like PostgreSQL. The database works best with a relatively small number of simultaneous requests, any more causes problems with locking. The fundamental bottleneck in the system is concurrency of the db. Once again, ETS can be a solution by caching db results that don't change.

Example: logging

In a real time bidding system we needed to write a transaction log for each request for accounting purposes. This is not a traditional text error/debug log, it is a CSV file.

We originally implemented this as an Erlang gen_event handler. Under the hood, though, these handlers are GenServers.

The event handler received events from multiple HTTP request processes, formatting them and writing them to a log file in an orderly way. This makes sense, as having multiple processes independently opening and writing to log files would cause a lot of conflict. The problem is that the GenServer once again became the bottleneck for the whole system. We were making all requests line up to go through the GenServer one by one. It got overloaded and timed out as disk I/O became an issue under load.

We could have played the same game of splitting things up into multiple GenServers. Instead, we followed a rule of Erlang: "Ericsson probably ran into this problem at British Telecom 20 years ago and solved it." So we went looking into the Erlang libs and found disk_log.

disk_log is very full featured, designed for exactly this situation. Telecom systems produce Call Detail Records, CDRs. Every time a switch touches a call, it records the information about who called whom and how long they talked. It then sends the records to a central server for "mediation," where they calculate the bill from the various pieces.

disk_log can handle 100K writes per second, using low level Erlang I/O features to support current writes from multiple processes. It supports error handling, log rotation, and reading back from logs. It's great, but the docs are limited, all you have is the man page. I am sure Ericsson has lots of examples from their products, but those are not open source. So we needed to find some examples and make some prototypes, but it solved the problem the right way.

This is one of the things I love about Erlang. The platform is very mature and has good solutions for the real problems we have. It's not magic, the laws of physics still apply, but it is a lot better than starting from scratch as we would with systems like Golang or Node.js.

Backpressure and load

One big problem with gen_event is that it doesn't have a mechanism for backpressure. It's really designed for low message volumes, and is now deprecated in Elixir. Instead you should generally use GenStage, which uses a pull model to avoid overload.

Lack of backpressure is a fundamental issue with the way Erlang process mailboxes work. If you send more data to a GenServer than it can handle, the mailbox will fill up, and eventually you will get a timeout. If you are having performance problems, look for processes with overloaded mailboxes and deal with them.

You may wonder why the Erlang system hasn't fixed this. The reason is that the current mechanism has low overhead. If we had to acknowledge every message, it would double the message load on the system. It also fits with the unreliable nature of real world systems. If we send a message and don't get a response back, then we try again. That handles messages that get lost due to network problems, crashes and overload with the same mechanism.

Another reason is that telecom systems are sold according to the amount of load they can handle. As part of the product development process, they identify what the bottlenecks are, then they limit the inbound load to what the system can handle, rejecting anything beyond that. If you have more, you need to buy another telephone switch. If a process mailbox is filling up in production, it means you have a bug or some other resource problem, e.g. failing hardware.

With systems connected to the Internet, we can't control our load, but we can be smart about how we deal with it. Have a look at Ulf Wiger's jobs framework and his presentation at the Erlang User Conference for more info on load regulation. (Another rule of Erlang is "pay attention to anything Ulf Wiger does.")

The standard Elixir Logger framework is based on gen_event and has a similar issue. It monitors its own mailbox and if it gets too full, it drops messages and applies backpressure by switching to "synchronous" mode. That makes things slower, though, so it can cause your system to crash, basically kicking it when it's down. If you are limiting load at the edge, you could use the mailbox size of your bottleneck as part of your load check when determining whether you accept requests.

There is some more discussion about logging in this performance tuning presentation

Database migrations in the cloud

2018-05-16T00:00:00+08:00

Database migrations are used to automatically keep the database in sync with the code that uses it. Elixir apps should be deployed as releases, supervised by systemd. Here is an example of how to run migrations when deploying Elixir releases.

It's tempting to automatically run database migrations when the app starts up. Once we get into more complex deployment scenarios, however, that can cause more problems than it solves. It's better to run migrations separately.

For production systems, it's important to be able to roll back code in case of problems. It's a lot easier to roll back code than data, though.

Whenever possible we make all our database changes work with old and new code releases. For example, we first run a db change which adds a column to a db, make sure that it's working properly with the old code, then deploy new code which uses the new column. If we need to roll back, then we know that the old code will still work with the new db schema.

We may also be running code in AWS in an Auto Scaling Group, deploying via a Blue/Green process with AWS CodeDeploy. This means that the db might be shared between old and new code at runtime as we upgrade the cluster.

Our normal process is to run the db migrations from a "Operations and Maintenance" instance in the deploy environment, e.g. the build server. In the test environment, we check out the code and run migrations against the db. We verify that it's working properly, then we deploy the new code and verify it works. Then we run the db updates in the production environment and deploy the code.

Deploying Elixir apps without sudo

2018-05-16T00:00:00+08:00

We normally deploy Elixir apps as releases, supervised by systemd.

After we have deployed the new release, we restart the app to make it live:

sudo /bin/systemctl restart foo

The user account needs sufficient permissions to restart the app, though. Instead of giving the deploy account full sudo permissions, you can make a user-specific sudo config file which specifies what commands it can run, e.g. /etc/sudoers.d/deploy-foo:

deploy ALL=(ALL) NOPASSWD: /bin/systemctl start foo, /bin/systemctl stop foo, /bin/systemctl restart foo

That works ok, but it would be better if we didn't require sudo permissions at all. One option is to take advantage of the supervision provided by systemd to restart the app.

When we deploy a new release, the deploy user uploads the new code, sets up the symlink, then touches a flag file. Systemd notices and restarts the app.

See mix_deploy for examples.

Getting the client public IP address in Phoenix

2018-05-16T00:00:00+08:00

When your app is running behind a proxy like Nginx, then the request will look like it's coming from Nginx, i.e. the IP will be 127.0.0.1. Similarly, If Nginx is behind a CDN, then all the requests will come from the IP of the CDN.

In order to log the request properly and make decisions like rate limiting, we need to get the IP address from a header set by the CDN.

As described in the Plug docs, we are expected to overwrite the remote_ip field in the Conn.

Following is a plug that reads the X-Forwarded-For HTTP header. You can do something similar with HAProxy’s PROXY protocol.

Add it to your app's Endpoint:

  plug MyApp.Plug.PublicIp

defmodule MyApp.Plug.PublicIp do
  @moduledoc "Get public IP address of request from x-forwarded-for header"
  @behaviour Plug
  @app :my_app

  def init(opts), do: opts

  def call(%{assigns: %{ip: _}} = conn, _opts), do: conn
  def call(conn, _opts) do
    process(conn, Plug.Conn.get_req_header(conn, "x-forwarded-for"))
  end

  def process(conn, []) do
    Plug.Conn.assign(conn, :ip, to_string(:inet.ntoa(get_peer_ip(conn))))
  end
  def process(conn, vals) do
    if Application.get_env(@app, :trust_x_forwarded_for, false) do ip_address = get_ip_address(conn, vals) # Rewrite standard remote_ip field with value from header
      # See https://hexdocs.pm/plug/Plug.Conn.html
      conn = %{conn | remote_ip: ip_address}

      Plug.Conn.assign(conn, :ip, to_string(:inet.ntoa(ip_address)))
    else
      Plug.Conn.assign(conn, :ip, to_string(:inet.ntoa(get_peer_ip(conn))))
    end
  end

  defp get_ip_address(conn, vals)
  defp get_ip_address(conn, []), do: get_peer_ip(conn)
  defp get_ip_address(conn, [val | _]) do
    # Split into multiple values
    comps = val
      |> String.split(~r{\s*,\s*}, trim: true)
      |> Enum.filter(&(&1 != "unknown"))          # Get rid of "unknown" values
      |> Enum.map(&(hd(String.split(&1, ":"))))   # Split IP from port, if any
      |> Enum.filter(&(&1 != ""))                 # Filter out blanks
      |> Enum.map(&(parse_address(&1)))           # Parse address into :inet.ip_address tuple
      |> Enum.filter(&(is_public_ip(&1)))         # Elminate internal IP addreses, e.g. 192.168.1.1

    case comps do
      [] -> get_peer_ip(conn)
      [comp | _] -> comp
    end
  end

  @spec get_peer_ip(Plug.Conn.t) :: :inet.ip_address
  defp get_peer_ip(conn) do
    {ip, _port} = conn.peer
    ip
  end

  @spec parse_address(String.t) :: :inet.ip_address
  defp parse_address(ip) do
    case :inet.parse_ipv4strict_address(to_charlist(ip)) do
      {:ok, ip_address} -> ip_address
      {:error, :einval} -> :einval
    end
  end

  # Whether the input is a valid, public IP address
  # http://en.wikipedia.org/wiki/Private_network
  @spec is_public_ip(:inet.ip_address | atom) :: boolean
  defp is_public_ip(ip_address) do
    case ip_address do
      {10, _, _, _}     -> false
      {192, 168, _, _}  -> false
      {172, second, _, _} when second >= 16 and second <= 31 -> false
      {127, 0, 0, _}    -> false
      {_, _, _, _}      -> true
      :einval           -> false
    end
  end
end

Port forwarding with iptables

2018-05-16T00:00:00+08:00

In order to listen on a TCP port less than 1024, an app traditionally needs to be started as root. Over the years this has resulted in many security problems.

A better solution is to run the application on a normal port such as 4000, and redirect traffic in the firewall from e.g. port 80 to 4000 using iptables.

Port forwarding with ufw

UFW is Ubuntu's "Uncomplicated Firewall".

Enable ufw

sudo ufw enable

Allow traffic to the app port

sudo ufw allow 4000/tcp

UFW doesn't have an easy command to do port forwarding, unfortunately, so we need to add a raw iptables rule.

Edit /etc/ufw/before.rules. At the top of the file, add the following:

*nat
:PREROUTING ACCEPT [0:0]
-A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 4000
COMMIT

Port forwarding with raw iptables

First, open up access to the app port:

sudo iptables -A INPUT -p tcp --dport 4000 -j ACCEPT

We can also open the port with rate limiting, useful for dealing with DDOS attacks. The following command allows five requests per minute from a single IP address, with a burst of 10:

sudo iptables -A INPUT -p tcp --dport 4000 -m state --state NEW \
    -m hashlimit --hashlimit-name HTTP --hashlimit 5/minute \
    --hashlimit-burst 10 --hashlimit-mode srcip --hashlimit-htable-expire 300000 -j ACCEPT

Finally, redirect port 80 to port 4000:

sudo iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-port 4000

You can see the rules with:

sudo iptables -L -n -v
sudo iptables -t nat -L -n -v

We need to make the rules persistent so that they will still be there when the machine is rebooted.

sudo apt install netfilter-persistent iptables-persistent
sudo netfilter-persistent save

Configuring iptables with Ansible

This example project for deploying Phoenix apps has Ansible tasks to set up iptables for port forwarding.

Rate limiting Nginx requests

2018-05-16T00:00:00+08:00

Any popular service may be the unfortunate recipient of a DDOS attack. We find that DDOS load ends up driving capacity planning, as it can easily be 10x the normal load.

You can rate limit at multiple levels. You might use a service such as CloudFlare, filtering provided by your hosting provider, a firewall on the local machine, Nginx rate limiting, or the application itself. If you are getting attacked regularly, you will probably end up limiting at all levels, with different thresholds.

The earlier you limit, the fewer resources are used, but the less information you have about the attack and the less control you have over how you respond. If you are running e.g. an API endpoint it's important to be able to distinguish an attack from a legit client gone wild.

Getting the user's IP address in Nginx

If Nginx is behind a CDN, then all the requests will come from the IP of the CDN. In order to log the request and make decisions like rate limiting, we need to get the IP address from a header set by the CDN.

Add this to nginx.conf, telling Nginx to get the IP address from the X-Forwarded-For header:

real_ip_header X-Forwarded-For;
set_real_ip_from 0.0.0.0/0;

Note that this is not entirely trustworthy, as an attacker can still send a request direct to the site with anything they like in the header.

See Serving your Phoenix app with Nginx for complete config file examples.

Similarly, when Nginx proxies requests to the app, we need to pass the request information through to the app.

proxy_set_header Host               $host;
proxy_set_header X-Real-IP          $remote_addr;
proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
proxy_set_header Refrerer           $http_referer;
proxy_set_header User-Agent         $http_user_agent;

Getting the user's IP address in Phoenix

Configure Phoenix to read the user's IP from the x-forwarded-for HTTP header.

Rate limiting in Nginx

In nginx.conf set up a rate limiting zone:

limit_req_zone $binary_remote_addr zone=foo:10m rate=1r/s;
limit_req_status 429;

In the vhost, limit requests for the zone:

limit_req zone=foo burst=5 nodelay;

Serving your Phoenix app with Nginx

2018-05-16T00:00:00+08:00

It's common to run web apps behind a proxy such as Nginx or HAProxy. Nginx listens on port 80, then forwards traffic to the app on another port, e.g. 4000.

Following is an example nginx.conf config:

user nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

worker_rlimit_nofile 65536;

events {
  worker_connections 65536;
  use epoll;
  multi_accept on;
}

http {
  real_ip_header X-Forwarded-For;
  set_real_ip_from 0.0.0.0/0;
  server_tokens off;

  include       /etc/nginx/mime.types;
  default_type  application/octet-stream;

  log_format  main '$remote_addr - $remote_user [$time_local] $host "$request" '
                   '$status $body_bytes_sent "$http_referer" '
                   '"$http_user_agent" "$http_x_forwarded_for" $request_time';

  access_log  /var/log/nginx/access.log main;

  limit_req_zone $binary_remote_addr zone=foo:10m rate=1r/s;
  limit_req_status 429;

  include /etc/nginx/conf.d/*.conf;
}

Here is a vhost for the app, e.g. /etc/nginx/conf.d/foo.conf:

server {
  listen       80 default_server;
  # server_name  example.com;
  root        /opt/foo/current/priv/static;

  access_log  /var/log/nginx/foo.access.log main;
  error_log   /var/log/nginx/foo.error.log;

  location / {
    index   index.html;
    # first attempt to serve request as file, then fall back to app
    try_files $uri @app;
    # expires max;
    # access_log off;
  }

  location @app {
    proxy_set_header Host               $host;
    proxy_set_header X-Real-IP          $remote_addr;
    proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
    proxy_set_header Refrerer           $http_referer;
    proxy_set_header User-Agent         $http_user_agent;

    limit_req zone=foo burst=5 nodelay;

    proxy_pass http://127.0.0.1:4000;
  }
}

Proxy settings

The main setting that does the forwarding is proxy_pass.

You can set additional options depending on usage, e.g. if it's an API endpoint, then you can reduce various buffers and timers to give better response vs the defaults, which are for more generic web serving:

proxy_intercept_errors on;
proxy_buffering on;
proxy_buffer_size 128k;
proxy_buffers 256 16k;
proxy_busy_buffers_size 256k;
proxy_temp_file_write_size 256k;
proxy_max_temp_file_size 0;
proxy_read_timeout 300;

High load

Once you start pushing Nginx hard, you will see issues. One of the first problems is OS limits on the number of open sockets. The sign of this is clients see a 5-second delay to responses (or a 503 error), but the app logs look fine, responding in milliseconds.

What is happening is that the client talks to Nginx, then Nginx talks to your app. When there are not enough filehandles available to open a connection to the app, Nginx queues the request.

You may start with 1024 by default, which is pitifully small. You will need to raise that at each step in the config, e.g. systemd unit file, Nginx, and Erlang VM.

Create /etc/systemd/system/nginx.service.d/override.conf with the following contents:

[Service]
LimitNOFILE=65536

systemctl daemon-reload

In the nginx config file, set worker_rlimit_nofile, smaller or equal to LimitNOFILE:

worker_rlimit_nofile 65536;

systemctl restart nginx

You can verify that the limits have been increased for the process by running:

cat /proc/<nginx-pid>/limits

Running out of TCP ports

After that, you may run into lack of TCP ports. In TCP/IP, a connection is defined by the combination of source IP + source port + destination IP + destination port. In this proxy situation, all but the source port is fixed: 127.0.0.1 + random + 127.0.0.1 + 4000. There are only 64K ports. The TCP/IP stack won't reuse a port for 2 x maximum segment lifetime, which by default is 2 minutes.

Doing the math:

60000 ports / 120 sec = 500 requests per sec

You can tune the global kernel config to reduce the maximum segment lifetime, e.g.:

# Decrease the time default value for tcp_fin_timeout connection
net.ipv4.tcp_fin_timeout = 15

# Recycle and Reuse TIME_WAIT sockets faster
net.ipv4.tcp_tw_reuse = 1

The HTTP client may keep the connection open, assuming that there will be another request. Depending on your use case (e.g. for an API endpoint), that may not be needed. Shut it down immediately by adding the "Connection: close" HTTP header. This is particularly useful for abuse, e.g. DDOS attacks.

See rate limitmiting Nginx requests for details about the rate limiting config in this example.

Nginx also has some complex behavior when it runs into errors when proxying.

It can be hard to figure out what is going on, as you don't get visibility. The Nginx business model is to hide the detailed proxy metrics unless you buy their Nginx Plus product, which costs thousands of dollars per server per year. A dedicated proxy server like HAProxy gives more visibility and control over the process.

Listening directly

At a certain point, you wonder what value you were getting from the local proxy. If you are only running a single app on your instance, common in cloud deployments, you can listen directly to HTTP traffic in Phoenix. That will give you lower latency and overall lower complexity. This works fine: we have Phoenix applications which handle billions of requests a day on the internet, resisting regular DDOS attacks with no problems.

In order to listen on a TCP port less than 1024, i.e. the standard port 80, an app needs to be running as root (or have CAP_NET_BIND_SERVICE capability). Running as root increases the chance of security problems. If the application has a vulnerability, then the attacker can do anything on the system. One solution is to run the application on a normal port such as 4000, and redirect traffic from port 80 to 4000 in the firewall using iptables.

Running multiple applications together

One place where running Nginx in front of the application makes sense is when you are using Nginx to glue together multiple apps, e.g. using Phoenix to improve performance of a Rails app. The first step is configuring Nginx to route certain URL prefixes to Phoenix, e.g. http://api.example.com/ or /api.

Beyond that, we need to integrate the applications, e.g. sharing a login session between Phoenix and Rails. This depends on the specific authentication frameworks used by each app.

We can also implement the UI and navigation on Phoenix to match a Rails app, allowing users to seamlessly work between both apps. The only thing the user will notice is that the Phoenix pages are 10x faster :-)

See this blog post on migrating legacy apps or this presentation for details.

Another option is to have Phoenix handle the routing, e.g. with Terraform.

Serving Phoenix static assets from a CDN

2018-05-11T00:00:00+08:00

Phoenix is fast, but you can improve performance by serving requests for static files like images, CSS and JS from Nginx or a content delivery network (CDN). This allows your app to focus on dynamic content.

Serving static assets from Nginx

If you are running your app behind Nginx, configure Nginx to serve the static files. Set the root dir in the vhost to point to the priv dir in the release:

root /opt/myorg/foo/current/priv/static;

Serving assets from a CDN

A better choice for production apps is to use a CDN. In addition to offloading requests, it also caches your content in servers close to your customers, improving network latency. A CDN like CloudFlare can also protect your app from DDOS attacks.

Some CDNs work as a read-through cache. If the CDN gets a request for a file that is not in its cache, then it contacts the app to get it, then caches it. With others, you have to separately upload assets to the CDN when deploying.

In a simple app deployed to a single server, we deploy the app and its assets together, so the assets are always in sync with the code.

With HTTP-level caching, the cache will by definition serve an old version of the file. As you make changes to your static assets, you need to make sure that the application uses the version of the assets corresponding to the version of the code that's running.

If you deploy a new version, the code should use the new assets. If you roll back, it should go back to the old version. Similarly, if you are doing a Blue / Green rolling deploy of your app in an auto scaling group, you will have a mix of old and new app versions sharing the CDN.

Building static assets in Phoenix

Fortunately, Phoenix handles the process of generating unique names for your assets. It maps a generic name like /js/app.js in your template into a versioned name like /js/app-8f0317e89884de8b7b3a685928bee5e7.js.

When we update the assets, they get a unique id, and the client will load the new version. The unique filenames mean that multiple versions of the same file can coexist in the cache.

We can also set the cache lifetime to infinity everywhere, as they will never change. This lets the browser and other caches keep them around longer for better performance.

The first step is to compile your application assets for production

mix deps.get --only prod
MIX_ENV=prod mix compile
(cd assets && webpack --mode production)
MIX_ENV=prod mix phx.digest

This builds assets under priv/static. When we deploy a new release to production, we need to copy the new asset files into the CDN.

For example, we can sync the files to the S3 bucket backing your CloudFront distribution.

aws s3 sync priv/static s3://$CDN_ASSETS_BUCKET

Configuring DNS

In order to use the CDN, Phoenix needs to generate URLs that point to it. First, we set up a DNS CNAME record pointing http://assets.example.com/ to the URL of your CDN.

If you are using Amazon AWS and CloudFront CDN, then you should use Amazon Route53 for your DNS, as it supports a special ALIAS record that works like a CNAME, but follows the underlying resource if it changes. Route53 also allows you to alias bare domains, e.g. http://example.com/, which is otherwise not allowed.

For high volume sites, we want to reduce the amount of data transferred. Make a separate domain like mycdn.com to serve your static assets, and it will not have the cookies associated with your main domain. This is also better for security, as session cookies are not sent to the CDN.

Configuring Phoenix to use the CDN

Configure your app to use the CDN by setting the static_url parameters in the endpoint config:

config :foo, FooWeb.Endpoint,
    url: [host: "example.com", scheme: "https", port: 443],
    http: [:inet6, port: {:system, "PORT"}],
    static_url: [scheme: "https", host: "assets.example.com", port: 443],
    cache_static_manifest: "priv/static/cache_manifest.json"

See this article for more details.

Getting the user's public IP

When you are running behind a CDN, requests will look like they come from the CDN. In order to get the user's actual IP, we need to look at the headers set by the CDN.

See "Serving your Phoenix app with Nginx" and "Getting the client public IP address in Phoenix" for details.

Advantages of Elixir vs Golang

2018-05-08T00:00:00+08:00

A prospect recently asked me what the advantages are of Elixir over Golang.

The simple answer is productivity. You get the best of both worlds: the productivity of a high level language with the scaling power of the mature Erlang platform.

Go is a low level language, and performance is good, but it lacks the productivity features of modern languages. It was developed for making relatively low-level services at Google's scale, e.g. HTTP routing infrastructure. When you are operating at their size, you need performance, but the complexity of C++ is hard to deal with. You need concurrency, but multi-threaded network programming is error prone. I spent years making VoIP applications in C++, so I know this pain.

It is also a reaction to the tendency for smart C++ and Java programmers to create abstractions which make systems harder to maintain over time. The layers make it hard to jump into a big codebase and fix a problem. This is a particular issue for SREs who are not just programmers, they also have to keep on top of the challenges of cloud infrastructure, networking, storage, etc.

Higher level scripting languages like Python are hard to operate at scale. They suffer from poor performance and lack of concurrency. Dynamic typing makes it hard to avoid errors at runtime, requiring lots of testing. Dependencies make them difficult to deploy, so the ability to simply copy a binary to a server is very attractive.

Go is basically a simplified version of C++, a kind of "blue collar" language. The Go Programming Language describes it this way:

"The Go project includes the language itself, its tools and standard libraries, and last but not least, a cultural agenda of radical simplicity. As a recent high-level language, Go has the benefit of hindsight, and the basics are done well: it has garbage collection, a package system, first-class functions, lexical scope, a system call interface, and immutable strings in which text is generally encoded in UTF-8. But it has comparatively few features and is unlikely to add more. For instance, it has no implicit numeric conversions, no constructors or destructors, no operator overloading, no default parameter values, no inheritance, no generics, no exceptions, no macros, no function annotations, and no thread-local storage. The language is mature and stable, and guarantees backwards compatibility: older Go programs can be compiled and run with newer versions of compilers and standard libraries."

There is a certain "embrace the suck" attitude in Go. Deploying large systems at scale sucks, so we choose simple tools that will always work and push through the problems. I can understand this perspective, but I am not so cynical. It ignores the improved productivity and safety that we can get through modern programming language features.

Ericsson was seeing similar issues when they created Erlang, the basis for Elixir. They had been building their telephone switches in low level languages and it was getting out of control. The systems were complex, buggy, and expensive to develop. Their solution was a combination of a low-level runtime to handle the problems of networking and concurrency once and for all, combined with a high level language and framework to make programming easier.

Erlang's distinguishing features are concurrency and fault tolerance. The lightweight process model makes it straightforward to create systems which scale to millions of stateful connections, e.g. WhatsApp.

The platform has great depth of tools to create, debug and manage large production systems. The OTP framework standardizes patterns for building services out of components. The platform includes features like an in-memory key/value store, process registry, etc., as well as built-in solutions for issues like production system tracing, high volume logging, alerting and metrics.

Elixir starts with the mature Erlang platform and adds powerful language features like lisp-style macros and protocols. We get the ease of use of object oriented languages, without the tight coupling between components. Functional programming gives us generic algorithms which work across all data. Pattern matching makes logic simpler. Binary matching syntax makes it easy to implement network protocols reliably with high performance. Immutability and lack of side effects make systems easier to reason about and debug. Unlike academic functional languages like Haskell, the language is focused on practical industrial programming, not type theory. Data structures are straightforward and easy to understand. Everyone talks about the concurrency, because it's special, but the language itself is legitimately a joy to program in.

As an example, error handling in Go looks like this:

c, err := foo.Client()
if err != nil {
    panic(err)
}
msg, err := c.Request(foo, bar)
if err == nil {
    fmt.Printf("Message on %s: %s\n", msg.Topic, string(msg.Value))
} else {
    fmt.Printf("Client error: %v (%v)\n", err, msg)
    break
}

The equivalent code in Elixir would simply be:

{:ok, conn} = Foo.Client.connect()
{:ok, result} = Foo.Client.request(conn, foo, bar)

Elixir's pattern matching lets us program for the success case. If we get an error return (e.g. {:error, reason}), then the unhandled match will fail, and the thread will exit. It will write a backtrace to the log with all the context of the call so we can duplicate the problem in our dev environment.

A supervisor monitors the process and manages all errors, including ones we may have missed. This is different from exceptions, as it allows us to actually deal with errors, e.g. retrying a call on connection timeout.

Elixir is taking the opportunity to rethink and refine a well established system. Mature languages accumulate cruft over time. For example, Java has multiple date-time classes (java.util.Date, java.sql.Date, Calendar) and we have to convert between them. Functions have parameters in various orders, so we have to keep looking at the docs. When Microsoft created C# and .NET, they had the benefit of learning from Java, which helped them to quickly create a full, consistent standard library and virtual machine. Elixir does that for Erlang, but can also take advantage advantage of all the existing Erlang libraries.

Similarly, a lot of the Elixir community comes from Rails, as the creator of Elixir, José Valim, was a Rails committer. He started with the mature Rails system and did it again, better, focusing on the problems he had maintaining large Rails projects that had evolved over time. The platform has better performance and reliability, but also takes a step back on "magic", as some of the features which make simple projects easy end up causing problems when they grow bigger. In addition to standard MVC, the Phoenix web framework (http://phoenixframework.org/) provides a "channels" abstraction which makes it easy to create stateful web applications, e.g. web chat systems. There is a GraphQL server as well which integrates with Phoenix.

As with Ruby on Rails, programmer productivity and ease of use are a primary focus for the community. Everything works well out of the box, and there are standard, well integrated tools for testing, asset pipeline, deployment, etc. People have taken the libraries that they loved from Ruby and implemented them for Elixir. There are also cutting edge tools for static analysis and property testing coming from the academic community.

The sweet spot for Go

The most interesting applications of Golang for me are network services which need to be very fast, i.e. they are CPU bound. Garbage collection avoids a major class of errors, making it safer. Go has built in concurrency using the CSP model, and it can call into C libraries very efficiently. Good applications are things like deep content inspection or running machine learning models. When you are operating at huge scale, cost of hardware starts to actually be more important than programmer time.

The concurrency crisis

Most of us are not writing applications at the scale of Google, but we still need to efficiently use hardware we have. To do that, we need languages that can handle concurrency, but the current crop of languages and platforms face challenges. It is hard to make existing languages and libraries safe, as it breaks programmer assumptions. Network communication libraries need to be rewritten to be non-blocking or use threads. Worse, shared data needs locking to protect concurrent access. This has made it very difficult for existing scripting languages like Python, Ruby and PHP to support concurrency. Node.js is built around non-blocking IO, but can't take advantage of more than one CPU without hacks like multiple processes. There is too much exposed machinery, and the language lacks type safety. Java uses a similar thread-based approach to concurrency as C++, but pervasive object orientation creates potential locking problems for every object. Rust has potential as a safe replacement for C to do systems programming, with concurrency. It is too low level for general application development productivity, though.

Elixir has had support for concurrency from day one, and it has 30+ years of tooling developed for Erlang. While the absolute performance is less than compiled languages, it is easy to parallelize tasks to make use of the machine. If that's not enough, we can deploy the app to work across a cluster of servers with few changes. This is the difference between speed and scalability. Go focuses on low level performance and relies on systems like Kubernetes to scale. That has its own nest of complexity to deal with, though.

It is easier to add libraries to Elixir for practical web programming tasks than it is to make other systems concurrent and reliable. We are using it to build large systems today, and we know it works. Instead of fighting to retrofit concurrency to existing languages, we can get on building the next generation of systems.

Yield optimization vs customer service

2018-05-08T00:00:00+08:00

Whenever we try to squeeze the last bit of utilization out of a system, there is a danger that it will have a big negative impact on the user experience. A great example of this is overbooking in the airline industry. Usually a bad customer experience does not involve getting beaten and dragged from an aircraft, but avoiding over-optimization definitely provides better service.

An example is a recent experience I had with a hotel. I flew to Shanghai to work with a customer, taking a morning flight and arriving at their office in the early afternoon. We got right to work with product design meetings. We went to dinner, then then they dropped me off at my hotel, arriving at 8:10 PM.

When I checked in, the lady working the front desk informed me that they had canceled my reservation at 8 PM. She was not exactly rude, but had clearly had the same conversation multiple times. She had no ability to help me and was tired of this situation.

My customer had booked the room from a portal website, and there was no indication about check in time. In this case, we were completely flexible. The hotel could have contacted us before the reservation expired, and we would have checked in before dinner. They had no contact information for us, though, and no mechanism to do it via the portal.

So now I had to find another room. We went back to the portal, and found the same hotel showing rooms available. Just for fun, we booked a room, then told the staff. She said it doesn't work like that. In fact, after someone makes a booking on line, the portal notifies the hotel, and they accept or reject the booking. There is no real-time information about whether rooms are available. Most hotels sell their rooms through multiple channels, so it's common for them to be double booked. They don't update the room status on the portals in real time. The staff is used to having unhappy customers, but it's out of their hands.

China has some interesting rules about hotel prices. It's illegal to charge more for a room during peak periods, so what they do is give variable discounts. Generally speaking, if you contact a hotel directly, they have a lower discount than going through a portal. The staff in the hotel has no incentive to match an online price, especially when they are sold out. I have had hotels cancel my booking because they can get more money for the room from a new customer.

There was a big trade show going on, so it was hard to find a room. It took about an hour of calling around before we found one. This wasn't a crisis for me, but imagine someone who didn't know Chinese, landing after a long international flight for the first time in a new city, with no local support structure. What if I missed my checkin because my flight was delayed, and I had no way to communicate? Not a great experience. This kind of optimization represents a small short-term improvement for the hotel, but a huge problem for the customer. It is classic short term thinking.

In the past, hotel bookings were controlled by the hotel itself. If they didn't sell all their inventory, then that was just the way it was. Now with the portals, it's attractive to optimize yields, trying to sell every last space for whatever price possible. In order to maximize further, they overbook, expecting a certain percentage of cancellations and no-shows.

Doing better

We need to optimize intelligently to mitigate the effects on user experience.

Automation breeds inflexibility

Systems based on people and paper have the ability to deal with problems. Automated systems only behave in one way, and customer service staff can't fix them. With paper, we could put a sticky note on a form, e.g. saying that a customer called to say they will arrive late. Adding a "notes" field to your database for customers, orders, and suppliers can provide great flexibility and value.

Optimized systems may separate the decisions from the information needed to make them. When there are communications problems, it can cause the process to fail. Give your people authority to solve problems. Robots should not boss people around.

Stressing your system

Pushing for the last bit of utilization puts stress on your system, exposing weaknesses. You have to think of every contingency, and execution has to be perfect. If the hotel always has an extra room in the basement that it doesn't use, then it can solve all kinds of problems.

Sometimes it's just math. An example from queuing theory: suppose we have a bank with tellers who can serve a customer in five minutes, and customers arrive every five minutes. With no variation, there is no waiting. In practice, even a 5% variation will cause the queue to blow up at random times. Adding a little extra capacity makes the queue go away forever.

Optimization can cause catastrophic failure. The supply chain management trend to minimize inventory that most stores have little or no stock, they get replenished frequently. If there is a problem with the transportation network, we will all have no food. What is your backup plan? How will your business work if the internet goes down? Design your system to be resilient to failures.

Context

Get as much context as possible and use it to inform your decisions. Get the customer's flight information, and you know when they will arrive. Track the flight, and you know when it is delayed, and you can hold their room for them. If they are a frequent guest, then you can be pretty sure that they will show up. Treat it as a long term relationship.

When you are going through an intermediary, learn as much about the customer as you can. Get their communication details. Sell direct. When you work through partners, they control the customer. You become a commodity. Sites like booking.com control a huge percentage of hotel bookings, often over 75%. They also have a better user experience than the average hotel website. The customer has a better selection, better pricing, and they can store their credit card details to make bookings easier. It's one thing to pay a commission to be discovered the first time, but hotels lose the same commission when the customer comes back again. Make the direct, long term experience better than that of the intermediaries.

Incrementally migrating large Rails apps to Phoenix

2018-04-28T00:00:00+08:00

Here are the slides for the presentation on incrementally migrating large Rails apps to Phoenix I gave at Ruby Elixir Conf Taiwan 2018.

Is Elixir/Phoenix ready for production?

2018-03-11T00:00:00+08:00

Someone asked this question on the Elixir Forum. Following is my answer:

When I first saw Elixir, we were developing custom apps (e-commerce, CRUD, etc) with Ruby on Rails, Python, and PHP.

We were using Erlang for the “tricky bits”, e.g. IoT and real time web, and liked it a lot. In 2006, we tried to use Erlang for web development. The core was great, but the rest of the ecosystem was lacking (e.g. database interfaces, templates, automatic page loading). Productivity was not great, so we ended up making hybrid apps.

When I saw Chris McCord’s post comparing Rails and Phoenix in 2014, I was really excited. The metaprogramming capabilities of Elixir let them build a web framework that combines the power of Erlang with the ease of use of Rails.

I wanted to make sure that it would be ok to bet the company on Elixir and Phoenix. The basic productivity was great, and the Elixir and Phoenix teams focus a lot on developer experience and getting started. I knew we would be able to deliver custom development projects efficiently.

My next question was whether we would have the libraries we needed for various project requirements, e.g. interfacing with credit card payment systems. It is fine to develop a few things, but it’s hard for a startup project budget and timeline to cover development of basic libraries. As a safety valve, I looked at calling Python and Ruby from Elixir using tools like Elixir Snake and Erlport. They worked fine, and I knew that we would be ok. In practice, we haven’t needed to do that much, all the libraries we have needed have been available or we could write them quickly. We have mainly ended up using Python for things like Pandas for data analysis.

In addition to standard web development tasks, Elixir and Phoenix support the next-generation “stateful web” applications like chat that are really hard to build any other way. One thing that I really like is that a single platform can do it all, i.e. public web, back end CRUD for admin, mobile APIs, interfaces to 3rd party APIs, and real time messaging. It actually simplifies development a lot, because we don’t need to use multiple languages, multiple servers, background job queues, etc.

In the last three years, we have done all our new projects in Elixir, and it’s worked fine. You don’t have to worry about OTP when getting started, you can build apps with just Phoenix. You will want to learn it in the future, though, as it is where the platform gets a lot of its power. As for difficulty finding developers, we haven’t found it hard for devs to get up to speed on Elixir and Phoenix. Within two weeks they are fully productive, particularly if they have experienced people available to help.

There is some learning curve associated with functional development. It’s similar to learning object oriented programming. You can start programming immediately, but it takes six months before you are really thinking functionally. Then you find it difficult to use barbaric languages with mutation and miss pattern matching dearly :-).

Setting Ansible variables based on the environment

2018-03-11T00:00:00+08:00

When deploying applications, we we usually have the same basic architecture in different environments (dev, test, prod), but settings differ. Some settings are common to all the machines in the environment, e.g. the db server connection string. We need to vary the size of instances depending on the environment, and we need to keep application secrets like passwords by environment.

What we would like is to put a machine in multiple groups, setting some defaults for the whole system, then overriding them by role and environment. Unfortunately that doesn't work in Ansible. There are priority rules between var sources, but all groups are the same. The Ansible "best practice" (limitation) is that a variable should be defined in one and only one place.

If your environments are relatively static, e.g. dedicated servers, then you can do it as follows:

Use the all group to set defaults for all servers. Next set group-specific variables by server "role", which take priority over all. Finally, set host-specific vars, which take priority over group vars.

inventory/group_vars/all
inventory/group_vars/web-servers
inventory/group_vars/app-servers
inventory/host_vars/server-01

If you are using the Ansible vault (recommended), these are directories, so you end up with something like this:

inventory/group_vars/web-servers/vars.yml
inventory/group_vars/web-servers/vault.yml

Define your groups in inventory/hosts and add servers to them.

[web-servers]
server-01

[app-servers]
server-02

If you have environment-specific vars, then you can make:

inventory/group_vars/web-servers
inventory/group_vars/app-servers
inventory/group_vars/web-servers-prod
inventory/group_vars/app-servers-prod
inventory/group_vars/web-servers-stage
inventory/group_vars/app-servers-stage


[web-servers]
server-01
server-03

[app-servers]
server-02
server-04

[web-servers-prod]
server-01

[web-servers-stage]
server-03

[app-servers-prod]
server-02

[web-servers-stage]
server-04

You end up duplicating some variables to deal with the lack of hierarchy. You can also use AWS dynamic inventory to assign servers to roles using tags. It supports having lots of servers.

This system breaks down if you have lots of applications and environments, though, e.g. multiple copies of the same app in production for different customers. One of our customers has a dozen apps deployed to AWS, each running in dev/test/staging/demo/prod environments. In addition, they run production environments in multiple regions, (US, EU, China, etc.).

In this case, we use a different structure to keep the combinational explosion of variables under control.

Make a playbook that sets up the machine or app, e.g. playbooks/myapp/web-server.yml:

- name: Configure web server
  remote_user: ubuntu
  hosts: '*'
  become: true
  gather_facts: true
  vars_files:
    - vars/myapp/{{ env }}/app.yml
    - vars/myapp/{{ env }}/common.yml
    - vars/myapp/{{ env }}/datadog.yml
    - vars/myapp/{{ env }}/keys.yml
  roles:
    - {role: ubuntu-common}
    - role: datadog.datadog
      when: '"env == prod" or "env == demo"'

Then make vars files for each combination of app and env, e.g. vars/myapp/prod/app.yml.

The vars directory is relative to the playbook. So you it could be playbooks/vars/myapp/dev/app.yml or at the top level. In that case the vars_files would be ../vars/myapp/{{ env }}/app.yml.

Finally, call the playbook specifying the environment:

ansible-playbook -i "myapp-$ENV" --extra-vars "env=$ENV" playbooks/myapp/web-server.yml

Here we set an OS environment var ENV=prod which pulls in a separate inventory (which could also use a dynamic inventory script) and set the Ansible env var which loads right var files.

If you need to set up an instance per customer, you can have vars/myapp/a/app.yml and vars/myapp/b/app.yml. And you can share common vars like var/myapp/common/app.yml.

In all of these, the playbooks don't use a lot of conditional vars, though they can if necessary distinguish between e.g. dev and prod. Generally it's best to use roles with default vars set in e.g. roles/ubuntu-common/defaults/main.yml. The tasks can use something like when: '"env == prod"' or you can conditionally include a role as shown above.

What makes a language popular?

2018-01-27T00:00:00+08:00

In the recent HackerRank developer survey, we can see "Which languages do employers look for by industry?" and "Which languages are developers planning to learn next?" In terms of popularity, there is a definite swing to JavaScript and Python. In terms of mind share with language enthusiasts, not so much.

I don't really care about absolute popularity. What I need is tools so I can make good systems, combined with sufficient popularity for a healthy ecosystem. The biggest challenge for programming languages right now is how to handle concurrency and take advantage of multiple CPU cores without going insane. We are also seeing a swing from dynamic languages towards strongly typed functional languages like Haskell and OCaml.

Over the last 20 years we went from low level compiled languages like C++ to Java, then scripting languages like Perl, Ruby, and Python. Productivity of the dynamic languages was much higher, but they were slower and more susceptible to failures at runtime, so we needed lots of tests. The type inference in modern functional languages gives us a good combination of productivity, performance, and safety.

There is a balance between mass adoption and language power. In order to be successful, we need to have some combination of:

Accessibility for beginning and average programmers
A business model which drives investment in the platform
Language features which support "programming in the large" and productive frameworks
A good community of fellow programmers and jobs

JavaScript is the ultimate weak language, but it's extremely popular because it's accessible. It has a business model behind it which drives the platform. As the browser makers speed up the runtime, Node.js benefits on the server side. Google tried to get advanced features into the language, but couldn't reach consensus with the other browser platforms. JavaScript is slowly getting classes, optional typing and syntactic sugar. It won't get macros or advanced features which require strong typing, though, as it would break compatibility with the existing web and make it hard for beginners to learn. It's most interesting as a compilation target for advanced languages like Elm and Reason.

Python is one of my main languages, but it's having trouble evolving. I was disappointed when Python 3 was released, as it didn't have enough new features to justify switching large production code bases. Guido was hostile to functional style programming, e.g. map/reduce. Based on its growing success, he was probably correct, but I don't expect significant improvements in the language. It will be my scripting language of choice and useful for data science, but not for building big systems.

Java was really good at getting people to adopt it, and it wasn't by being powerful, it was by being more accessible and easier to use.

"We were not out to win over the Lisp programmers; we were after the C++ programmers. We managed to drag a lot of them about halfway to Lisp." - Guy Steele, Java spec co-author and Lisp pioneer

The Java VM has incredible amounts of engineering behind it. They started with object orientation, then added support for dynamic languages. By building on the Java VM, languages like Clojure and Scala get a good runtime and are not seen as being too risky. It is probably the future of Ruby. Java has potential as the "one VM to rule them all", unless Oracle screws it up. Microsoft .NET is the same, they have this nice VB.NET / C# / F# thing going to combine accessibility with power. It's a nice system, but too proprietary for my tastes. I have seen too much bad behavior from Microsoft in my life to invest in the platform.

Haskell is very powerful, but the academic terminology makes it difficult for beginning and workaday programmers. I am very interested in it myself, but I don't think it will be a mainstream language. It can be a nice "secret weapon", though, and it influences practical languages like Elm. OCaml gives us languages like Reason and F# on popular platforms. Its level of typing may be the right balance between strictness and getting things done. It's low level enough for systems programming, while keeping safety.

Erlang provides a very interesting language data point, as everything is there for practical reasons. It is functional because that's the way you make reliable and scalable systems. It does not do as much with types as I might like, compared to Haskell, but that makes it easy to understand.

Erlang has a solid VM with a business model supporting it. Ericsson uses it for their telecom products, and companies like WhatsApp add features that they need. It scales to use all our available CPU cores without drama. Mature tools let us manage and debug servers in production, and we can easily build highly available, highly scalable clusters. The platform has been been under development for 30 years. It's not going away, it's getting better.

Elixir brings the Erlang platform to a new generation of developers. It is easy to get started, and adds powerful features like Lisp-style macros and Clojure-style protocols. It takes the standard library functions from Erlang and makes them consistent and easy to use.

Elixir has a good chance at being a mainstream language for web and server side applications. It is a logical next step for the Rails community, as the platform hits its limits. It has a great community, and lots of libraries are getting written for web development and more advanced applications. It is the standard platform for building massive chat systems, and is popular for next generation IoT, financial, and health care systems. The Nerves platform makes it easy to build reliable embedded systems and appliances.

If Elixir stays at its current level of adoption, that that's fine. In the future I think we will see Erlang-style concurrency features in the JVM and .NET platforms. As we saw with Twisted Python, however, it is really hard to make existing code and libraries safe for concurrency. Every Java object is a potential problem. The most likely result of better concurrency primitives in the Java VM is probably Elixir being ported to it.

KYC wall of shame

2018-01-14T00:00:00+08:00

There is a saying that frustration is an occupational hazard of being a user experience designer (or an excessively logical engineer). Once you start designing processes, you see process problems everywhere, whether or not you want to. As an American living in Taiwan, I am used to being the weird guy who breaks the process. Lately, however, I have been having more than my usual share of identity confusion (no jokes, please).

Other than PayPal, which was particularly bad, I won't name and shame, because the general state of the art is pretty sad. There are lots of opportunities for startups to compete on user experience.

Bank 1

My bank was sold to another bank (for the fourth time, now). I had activated my new card at the ATM, but that apparently wasn't enough, so they gave me a call. The lady said, "For security, we need to verify your identity. What is your birthday?" I was like, "Uh, no, that's not the way that security verification works. Would you give your birthday to anyone who called you on the phone?" We compromised, she gave me the year and month, and I gave her the day.

Next she wanted to set up a new phone banking PIN. But their phone system doesn't recognize DTMF tones from mobile phones, so I couldn't do it.

Takeaways

There are some logic problems here. First, they need to get the fundamentals of authentication right. It's hard enough to train users to avoid scams, we should not make people think it's normal. Second, why call me on my mobile if your system can't handle it?

Bank 2

I opened a bank account when I first arrived in Taiwan years ago. The bank's systems required the customer's national id number. As a foreigner, I didn't have one, so they created a fake number for me from my birth date and name.

A year ago, my internet banking stopped working, and we had to switch the account to use my alien registration certificate number. That's better, but still a problem. The format of the ARC numbers is slightly different from national id card numbers, so their validation logic fails. I had to use use my wife's national id number for my login.

It took about two hours sitting in the branch, as the staff diligently made phone calls to people at the head office. At some points we almost lost hope and closed the account, but eventually it worked. Recently, though, the bank's systems changed, and the various parts of my account became dissociated. My ATM card stopped working, saying that there was no bank account (another hour to fix). Now the internet banking stopped working with a 500 error. At least the paper account book still works....

Better than this guy, I guess:

No Emojis for your bank account name... 😂😂😂😂😂 pic.twitter.com/S2wc5pZ2XZ
— Bud (@this_is_bud) May 25, 2016

Takeaways

Don't assume that everyone has a national id. Make your own unique identifier and associate it with the user's id. How do you deal with foreigners? What is the key that links different systems in your organization? Is it the customer's name? Their id number? Do you expect that number to never change? In some countries the passport number is their national id number, in others it changes when they renew their passport.

Bank 3

Another bank in Taiwan is verifying accounts for FATCA. I needed to fill out the US W-9 form with my name. Of course, I actually had to fill it out three times, with three names. One to match my US tax return, and two more for the different ways they had broken my name on my bank account and credit card, e.g. family name first, name chopped because it is too long.

At some banks, my name is "MORR", because Chinese people have a maximum of four characters in their name. I have learned that I can only make a wire transfer to one bank during the day, because matching the account names requires human attention. Otherwise, it fails with an obscure XML error.

Bank 4

We opened a bank account for my daughter, and my mother wired her some money. When the money arrived, the bank rejected it because it didn't use her full name, it had a middle initial. They wanted us to send the transaction back to the US and do it again (paying the fees again, of course). I told them that if we had to do that, we would close the account because they were too incompetent to trust with money, and they relented.

Takeaways

The Know Your Customer and Anti-Money Laundering process would be a lot easier if you let your customer actually use their real name. Of course this gets a bit challenging, e.g. different scripts or Chinese names, but it certainly makes people happy. So you should let people enter their real names, and add a field for a transliterated version if necessary. Maybe allow them to have multiple variations on their name. Is the goal to veriify that the name is correct or to make sure it's unambiguous?

Bank 4

My corporate credit card has an enhanced verification process. Sometimes when making online purchases, it bounces me to a page to verify my identity. It used to ask me for my passport number (from a previous passport, but whatever).

With no notice, the bank changed the system so that it required a code sent to my mobile phone. Their credit card system didn't have the correct mobile number, though, it had our office number in Hong Kong. There was no place on the web to see or change the phone number, I had to send them a letter by post, which takes a week to process. So the effect was that I suddenly became unable to make payments by corporate credit card.

The bank redesigned their website. The new website is prettier, but doesn't address the actual usability issues. If anything, it makes them worse, because it fits less information on a single screen.

As a business, our most fundamental issues are being able to reconcile payments we receive and making outbound payments. The only information we get in statements is "DEPOSIT" and "WITHDRAWAL". The bank only keeps 90 days of transactions online, because, I guess, lines in a database are expensive. That's a problem when we have questions on our annual accounting 18 months later, though. So every month we copy out the transactions.

They send us PDF equivalents of the paper statements, but they insist on encrypting them with a crazy Java-based system that only works on Windows. If an email doesn't get through, we have to pay them US$25 to mail us a duplicate copy. The passphrase that encrypts the files suddenly stopped working at the same time as their website changed. So now we can't open any old emails. Instead of using the password protection built into the PDF standard, they created a monster.

Recently, they were collecting more information for their KYC process. There was supposed to be a form on their website, but it disappeared in the rewrite. So I had to email 100MB of scans of documents to them. In the course of it, they asked me to provide the id of my partner, who I bought out 12+ years ago. It seems, despite notifying the bank by postal mail (twice!), it didn't work, so we will need to do it yet again. There is no acknowlegement of receipt of documents.

Takeaways

Make sure that what you are relying on for authentication actually works. They could have sent a message via SMS to the mobile numbers before switching the system over.

Let people view and change things online.

Pay attention to the fundamental things that your customers care about.

As a business, I would love to get an "API" driven bank account so I never had to go to their website at all.

Bank 5

We were trying to reconcile the bank statements for our Vietnam branch. The internet banking site has a way to export statements. Their export files had an XLS extension, but were actually HTML. When we got that figured out, we tried to parse the line items, written in Vietnamese. After the tenth regular expression variation, we realized that this was not generated by a computer, some human was entering the descriptions of the transactions for deposits and withdrawals.

Phone company 1

In our Taiwan branch, the registration lists the "responsible person" (me) and also a "branch manager." We moved to a new office, so I went to change the billing address for a mobile phone. The clerk thought that because there were two people on the registration, he had to have both managers there to approve the change of address. It took half an hour of arguing to get him to understand the difference between OR vs AND.

In practice, what do they really care what the address is, as long as someone pays the bill? Cue Mitch Hedberg

I guess I should be happy they are authenticating the request. Despite everyone adding SMS verification to their verification processes, it's not particularly secure. It's often easy for an attacker to convince a mobile phone company that I have lost my phone and need a new SIM, then they can intercept the SMS.

Phone company 2 + 3

Based on my previous professional experience with VoIP, when we set up the old office, I tried to get a VoIP phone line. I thought I had succeeded, but when they installed the system, it turned out that we had VoIP number attached to a dedicated ADSL line. The only way they could legally sell us a phone number was to also sell us a physical phone line.

When we moved, we tried to transfer the phone number to our new office. Unlike mobile numbers, it turns out that the "VoIP" phone numbers could not be ported. Same thing for our fax number. It was associated with the ADSL line, which was in a different telephone exchange, and could not be ported.

I decided that since we hadn't gotten a non-junk fax in the last year, that fax is officially over. Losing the phone number was more annoying.

Immediately after installing the new phone line, we started getting automated scam phone calls from someone pretending to be the National Health Insurance Administration, saying that there was fraud associated with my card. "Press 1 to talk with an operator." Talking with a foreigner broke the scammer's script pretty fast...

Takeaways

What happens if your user loses their phone or changes their number? How will you authenticate them?

The bank has a mobile app. Why not use that to verify my identity? I needed to talk with customer service at my bank in the US. They had a button on their app that said "Call Us." I thought, "Great! VoIP." Nope, it dialed the phone for me. Of course, I was in Taiwan, so it didn't add an international prefix. If you already have a secure login on the phone, leverage it to get a secure connection into the call center for talk. If you use chat, then you can provide rich navigation instead of voice prompts, and avoid making the customer enter their account number over and over.

What kind of permissions does your customer want to see for transactions? In my situation, any one manager should be enough. But if there were two partners, then maybe it should require both of us. How would they know the difference? How can you allow your customer to delegate responsibility for e.g. accounting or making transactions? The boss is busy...

Do your contracts specify fax as a legal notification mechanism? Does anyone have a fax number anymore?

Debugging your space probe

2018-01-03T00:00:00+08:00

Years ago we were building an embedded vehicle tracker for commercial vehicles. The hardware used an ARM7 CPU, GPS and GPRS modem, running uClinux.

We ran into a tough bug in the initial application startup process. The program that read from the GPS and sent location updates to the network was failing. When it did the console stopped working, so we could not see what was going on. Writing to a log file gave the same results.

This is unfortunately common for embedded systems. For normal programmers, if your machine won't boot up, you are having a bad day. For embedded developers, that's just a normal Tuesday, and your only debugging option may be staring at the code and thinking hard.

This board had no Ethernet and only three serial ports, one for the console, one one hard wired for the GPS and one for the cellular modem. The ROM was almost full (it had a whopping 2 MB of flash, 1 MB for the Linux kernel, 750 KB for apps and 250 KB for storage). The lack of MMU meant no shared libraries, so every binary was statically linked and huge. We couldn't install much else to help us.

A colleague came up with the idea of running gdb (the text mode debugger) over the cellular network. It took multiple tries due to packet loss and high latency, but suddenly we got a stack backtrace. It turned out printf() was failing when it tried to print the latitude and longitude from the GPS, a floating point number.

One rule for normal programming is that if you think there is a compiler bug, you are wrong -- it's a bug in your code. In this case, a few hours of debugging and Googling five year old mailing list posts turned up a patch to gcc (never applied), which fixed a bug on the ARM7 which affected uclibc.

This made me think of how the folks who make the space probes debug their problems. If you can't be an astronaut, at least you can be a programmer, right? :-)

Is it time for Lisp in DevOps?

2018-01-02T00:00:00+08:00

We have been working on a project migrating a big Rails app from physical hardware to AWS, and I have been doing a lot of automation work.

It strikes me how we are doing the same thing over and over with different tools: reading variables, templating files and running semi-declarative logic. All with more or less broken syntax, including:

Terraform / Terragrunt
Ansible
Shell scripts
Packer JSON
Jinja2
CodeDeploy appspec.yml lifecycle scripts
Capistrano
Rake
Cron specs
CloudFormation
Endless different config file syntaxes

During the dotcom days we would laugh at the guys with title "HTML Programmer". Now we are "YAML Programmers".

This makes me think about replacing it all with a Lisp (probably Scheme), following the "code is data" and "data is code" mantra. We would have lots of parentheses, but we would have real variables with sane scoping rules, real functions, proper syntax for function calls, and macros. Maybe we need the YAML/JSON equivalent of SXML

Lisp is very powerful, but has not been able to break out. I learned it after hearing smart people say, "learn Lisp, not because it's practical, but because it will expand your brain and change the way you think about programming". I have certainly found that to be true. It has minimal syntax, which is one of its biggest strengths, but tends to turn off newcomers.

I think that part of the lack of success bad timing. When Lisp was at its peak, we didn't have open source, compilers cost $5000/seat. Lisp was very powerful, and faced competition from people selling FORTRAN for defense contracts who played the game better.

Another part was the "smug Lisp weenies". Lisp traditionally attracted smart but antisocial people who loved to argue and were condescending to newbies. They made it impossible to make progress on the legitimate improvements needed to the language.

Now we have open source tools and communities. There is a lot of interesting stuff going on in Clojure and Racket, and people are a lot more welcoming.

The pendulum is swinging away from dynamic scripting languages. Go is popular in ops, but I find it too low level, and it doesn't take advantage of recent advantages in programming languages. I mainly program in Elixir, which has the power of Lisp-style macros, approachable syntax and a great community.

Maybe it's time to give Lisp another try. I could write a Scheme syntax for Terraform. Or maybe I will take a swing at a Terraform clone in Elixir...

PayPal Know Your Customer failure

2018-01-02T00:00:00+08:00

Applying for a merchant account so you can accept credit cards traditionally takes weeks. You meet with the bank, show them your financial statements, and explain your business. Then they make you an offer for e.g. 2.8% + $0.30 per transaction (plus other mystery fees that you find out about later). They may require you to keep money in your account at all times, or only pay you 30 days after your customer pays.

When we did our first SaaS product 12 years ago, our bank said, "Oh, you are doing business on the Internet, so you are in our 'high risk' category. We will charge 5% and you have to keep US$20K in your account." We were like, "Did you just tell me to go screw myself? I guess so."

The fundamental thing to understand is that because of consumer protection laws, if you fail to deliver, the bank is responsible for refunding the money to the customer. That makes them conservative.

Once you are approved, as long as your business matches what you described, you won't have problems unless you have an unusual number of returns or chargebacks. Eventually your volumes get higher, and you can renegotiate.

PayPal works differently. It's easy to get an account, you just sign up and start receiving money. It's a breath of fresh air. If their anti-fraud algorithms trigger for whatever reason, however, they lock your account and the review process starts. They ask you to explain what the money is, send them company documents, etc. The review is done at the end of the process, not the beginning, and it can get really ugly.

This has given PayPal a rocky reputation with entrepreneurs and startups. You hear horror stories about companies bumping along with a moderate amount of sales, then they get profiled in TechCrunch and PayPal shuts them down because of the "suspicious" increase in sales. They lock your money for as much as six months, and maybe just keep it forever. You are the collateral damage of PayPal's algorithms, and you can't get a human to fix it. They don't care, because it's a numbers game to them.

There are certain business that they won't accept, e.g. conference registration. There are other rules, e.g. you need to ship physical products immediately after getting the order. So if your supplier is out of stock and someone complains, you get shut down. Choosing PayPal because it's easy to get started ends up causing you a lot of trouble later.

When we did the PhD Movie sales site, we took this very seriously. Jorge was releasing a movie related to his popular PhD Comics. As soon as the announcement went out to his mailing list, we would have tens of thousands of people buying the movie immediately. Not only would the server need to handle the load, we were afraid that PayPal would shut us down. We would have an embarrassing customer experience and potentially lose the sales forever. Because of this, we implemented multiple payment processors. Fortunately things went off without a hitch, but it was nerve wracking.

Knowing Your Customer

The big story in banking these days is Know Your Customer (KYC) and Anti-Money Laundering (AML). Customers of ours like EMQ put a tremendous amount of work into getting it right. There are existential penalties from governments if they don't follow the regulations, but the process has the potential to give a really bad customer experience.

Recently all my banks have been upgrading their KYC and FATCA compliance, and I have been seeing the KYC process from the other side. As an American living in Taiwan, with company headquarters in Hong Kong, I am always a weird case.

My recent experience with PayPal shows how it can all go badly wrong.

PayPal

When I signed up for our company PayPal account in 2009, it was uneventful. We started taking payments for our hosting business and software development projects. We used PayPal to make payments. We became a business verified user.

About two years ago, PayPal started sending me emails in simplified Chinese. Not my favorite thing, but the system was working.

About six months ago, our account was suddenly restricted, and we could only transfer out $2000 per month, about 25% of our normal amount. They required us to verify our account. We sent them documents and they released the restrictions.

About a month ago, things got serious. We were restricted again. I gave them all our documents, but there was a hitch. They didn't accept our address in Hong Kong, saying that we had to enter an address in China.

One entertaining point was entering my home address in Taiwan and getting an error about an invalid postal code. I had to enter it in a different format, as if I was sending a letter from China.

I called customer service multiple times to no avail. Their compliance team team doesn't talk to customers, and their customer service team has no power to do anything. They would say that they had sent messages to the compliance team, but I was still getting an email (or 10) every day telling me that I had to verify the account or it would be shut down. I uploaded more documents and got more form emails saying that the address was not in China. I asked the compliance team questions, but never got a human response.

Finally someone investigated, and told me that the company had been set up in China at the beginning. (I think that it was actually a data migration error at some point, but that's just a guess.)

He said that our address was in their system as:

Room 1005, Allied Kajima Building
138 Gloucester Road, Wanchai
Hong Kong SAR, China

Because it ends with China, we must be a Chinese company. But simultaneously, this address was not considered a Chinese company for purposes of compliance. Is Hong Kong part of China? Is Taiwan part of China? (These questions are above my pay grade.) We had a China PayPal account, not a Hong Kong PayPal account, and it is impossible to change it (Really? Errors cannot be corrected?).

Their only solution was for us to delete the account and create it again. That would leave us as a blank slate, though, with no history. The anti-fraud systems would inevitably kick in, our account would be limited, and we would not be able to use it to process our current payment volume.

Then the tune changed, and they said that we could submit a formal affidavit from a Chinese company describing their relationship with us. I asked them what kind of relationship they wanted: landlord, customer, vendor? Should I find a company in China and pay them $100 to write a letter saying that we are, ipso facto, their customer? If I am going to be making something up, what did they want? Is this really the way KYC is supposed to work?

They would say that someone would call me back and then not do it. They were unable to call Taiwan mobile phone numbers reliably, so they would register an unsuccessful "attempt" to call but not try again (the classic failure mode of working to get a task off their todo list without actually serving the customer).

Then someone investigated more and said that when I created the account I was in Taiwan (yes!). (It's good to record the IP address when users register.) But somehow I was supposed to have intentionally created it as a China company (nope). Classic "blame the customer" approach. Of course, if it was true, wouldn't it be KYC problem if they let me create an account in China? It should be a Taiwan account, if not a Hong Kong corporate account.

In the end they restricted our account again, and we are no longer a PayPal customer. Ironically, this was a pure Know Your Customer failure on their part. The information I provided them was complete, correct, and hasn't changed since the start. They just made a mistake somewhere in their systems, and were organizationally incapable of fixing it.

Often when you see incompetence like this, it's because it's in a company's business interest to be incompetent. Not in this case, though, as they lose hundreds of dollars a month in fees. We set up wire transfer agreements with our big vendors, saving us money.

Opportunities

We may think of PayPal as not having a physical location, they live "on the Internet". This experience shows clearly, though, that PayPal is subject to the rules of different countries, but unable to deal with the real complexity of international users.

There is a big opportunity for startups to use cryptocurrencies to handle payments in a fundamentally better way. They can compete on service, dealing with country-specific KYC and fiat currency issues. It can certainly be done with a better customer experience than PayPal.

Secure web applications with GraphQL and Elixir

2017-12-30T00:00:00+08:00

In traditional applications, the web application talks directly to the database. It has rights to do anything, relying on application rules to control access. If an attacker compromises it, then they can do anything, e.g. grab all the data or create a funds transfer transaction.

When security is critical, e.g. in health care and financial services applications, there are benefits to separating the application that interacts with users from the back end data using a well defined API.

In health care applications, it's common to have users with different roles looking at the same information (patient, family member, nurse, doctor, admin). Those rules can be very complex, and bugs may leak information. The data that a user can access depends on their roles and relationships. A family member can view a patient's medical information once they have been authorized. A doctor in a clinic can view today's appointments and active cases. A specialist can view the cases that they have been referred.

A banking customer can view their own account and make transactions, and they may give third parties access to data. Inside the bank, we need to control staff access to data based on their roles, e.g. only staff handling KYC should have access to scans of IDs.

Using an API lets us clearly define operations and the permissions needed to execute them. We tie the operations to a user, and the API server ensures that they have rights to access data. This clean interface provides a central place for access control and audit trail. It is easier to understand and to test. There is a single API to access the data shared between web front end, mobile API and other services.

Sounds great, but doesn't it have a lot of overhead? Not with GraphQL and Elixir. We originally started using GraphQL for mobile APIs, then realized that it was also great for security.

API design using REST and GraphQL

It is popular to build APIs using REST, modeling our systems in terms of "resources," e.g. users, accounts, medical cases, transactions. We then express all actions in terms of create, read, update, and delete operations on those resources.

While conceptually simple, REST can be quite "chatty," requiring a lot of requests to build complex pages. We might make one request to get a list of patients, then one per patient to get their details. We make another request for open cases associated with each patient, then another to get the case details.

Things that would be joins in a relational database end up being multiple web requests. If the requests are all local, the performance is not too bad, but it certainly adds up. In a mobile context, round trips can be very slow, and complex pages can take seconds to load.

REST apps are supposed to use HTML-style links in messages, allowing the application to navigate between resources. Many "REST" applications don't follow the principles, however, they are just ad-hoc blobs of poorly defined JSON delivered over HTTP.

In REST, there is no standard way to specify search parameters, filtering or subsets of a resource's fields. We might want to restrict sensitive fields based on user role. An app listing cases may end up getting the body of each case and throwing it away, only to fetch it again when the user opens the case. Developers have to write custom code to handle and validate parameters. We end up with multiple versions of APIs which differ only in the number of fields.

In order to avoid making multiple requests to the back end, mobile app developers may use "view APIs," e.g. a /home-page API which gets all the data for the home page in one shot. This results in an explosion of API functions and versions as we add pages and data fields to the front end.

GraphQL was invented at Facebook to solve this problem. The client sends a query identifying the objects it wants, as well as fields in associated objects. The GraphQL API server returns all the data in a single request. It can authenticate the user and check their access permissions, filtering the result set to ensure that they only see what they should.

For example, here is a query for an article summary list:

{
  article {
    title
    published_at
    author {
      name
    }
  }
}

Standard schemas define a data model, allowing the framework to handle validation without hand coding. It handles filtering, selection and paging in a standard way. It's not necessary to force complex actions into the REST resource model, as GraphQL supports named operations with well defined parameters.

It also has a standard mechanism to publish real-time event messages between parts of the system, using the same schemas to define the structure. A client can select cases and display them, then subscribe to see new cases as they are created.

Access control

Every request has a user context associated with it, represented by an access token.

On the web, when a user logs into the system, they pass their username / password to the front end, which calls the API to authenticate the user. The back end verifies the information and returns the token. The front end stores the token in the user's session and uses it on subsequent requests.

Mobile applications work the same way, calling the same API and storing the token on the device while the session is active.

Rich front end apps running in the browser can talk directly to the GraphQL server, bypassing the front end web server entirely while sharing the login session.

If an attacker compromises the front end machine, then all they can do is execute operations as currently active users. They can only see a small subset of the data, and they lose access when the sessions expire.

Elixir for the win

We use the Absinthe GraphQL server, written in the Elixir programming language. It handles GraphQL queries along with our own custom application logic, combining traditional web development and GraphQL services on the same platform (Phoenix).

Modern stateful-web applications use Web Sockets or HTTP/2, making the user interface more interactive and powerful. Phoenix Channels let us combine web, mobile and other data sources like IoT using the same system. The Erlang platform can easily handle the load, while staying manageable and secure.

Integration

The GraphQL server provides a common interface to multiple back end servers. We can even make a single query resolve each field to a different back end server, combining the results into one response.

When interfacing with a REST back end, we can take advantage of the Repo application pattern used by Elixir's Ecto db library, but talking HTTP. That fits into the standard Phoenix structure, allowing easy filtering of queries via input parameters. For example:

q = from(i in GitHub.Issue,
         select: {i.title, i.comments},
         where: i.repo == "elixir-ecto/ecto" and
                i.state == "open" and
                "Kind:Feature" in i.labels,
         order_by: [desc: :comments])
Repo.all(q)

[{"Introducing Ecto.Multi", 60},
{"Support map update syntax", 14},
{"Create test db from development schema", 9},
{"Provide integration tests with ownership with Hound", 0}]

Incrementally migrating a legacy app to Phoenix

2017-12-25T00:00:00+08:00

Over the years we have done lots of projects where we migrated an application from one platform to another. We might do this to solve performance issues or to switch to a better technology stack. This can be a challenge when you have a big app that is in production, and you need to do it incrementally.

It depends, of course, on the specific technologies and application functionality, but the process is overall quite similar:

Analyze your existing system and prioritize the work

Before starting, analyze the logs from your existing system. That will give you information about the parts which are having failures and performance problems. You may be surprised about what kind of traffic you are getting, e.g. bots or broken clients.

You can add the $request_time variable to your Nginx log config to get information about how long it took to handle each request, identifying bottlenecks. Look at the slow query logs in your database to see problematic queries, you will need to deal with them in the new system as well.

There may be performance problems caused by bad HTTP/HTML practices on the legacy system, e.g. lack of JS/CSS asset consolidation. Use the network analysis tool in Chrome to see what assets the page is loading. Spending a bit of time making sure that static assets are served from Nginx or a CDN can help with performance during the transition with low risk.

Implementing production monitoring. That will help you stay on top of errors while your are transitioning and make sure that things are working properly. It will show you 404 errors from broken links, etc, as you roll out the new system.

The result is a better understanding of how your legacy system works and a list of priorities for things to fix.

Put the app behind a reverse proxy

Put both apps behind a proxy such as Nginx and use routes to direct traffic to one app or the other, allowing the two apps co-exist. You might have the API on api.example.com, or you might direct certain URL prefixes to Phoenix while keeping the rest in the legacy app. A good example is splitting off the API requests on /api for performance while keeping the user registration and/or admin pages in the old app.

Write the new or replacement features in Elixir/Phoenix

If the app is REST based, then you can generate standard REST controllers/routes. You can also take advantage of plugs to implement common logic like authentication, avoiding repetition.

Create Ecto database schemas for your legacy tables. After you switch to Phoenix, you will want to have database migrations, but during the transition period it's probably enough to just have a db snapshot that you can talk to.

Write tests

Tests give you the confidence to make changes and know that things are working properly.

If you are making an exact replacement, then you have an opportunity to verify that the old code and new code behaves the same. This is particularly easy for API endpoints, because there is no UI to get in the way. Just collect inputs and expected outputs from your legacy system and turn them into ExUnit test cases. A logging or caching HTTP proxy can help with this.

Share authentication between the apps

This allows users log in on one system and access pages on the other.

For an API, it's generally straightforward, e.g. we just validate an API key, though it might involve something like OAuth2. In any case, the new system doesn't have to interact much with the legacy system.

For a web app, it can mean getting into the guts of the session mechanism on your legacy system. Most commonly, that involves getting a session id from a cookie and looking it up in a db, Redis or Memcached. Then you may need to parse the data stored in the session, e.g. to extract the user id and use it to authenticate the user.

This can get ugly if the legacy system is using a language-specific serialization format for its convenience, e.g. PHP serialization. We wrote a parser for the PHP serialization format in Erlang years ago... if there is interest, I will share it.

If you control the source system, then you may be able to modify it to make your life easier. For example, you could switch to a standard format for session data like JSON. If the legacy system is using a proprietary session store, you can likely switch it to use a database (e.g. MySQL or Memcached), allowing both systems to talk to it.

Another option is to set a new cookie on login just for interop, e.g. after the user logs in, write a JWT which has the user id. Then the Elixir side can grab the data easily using a library like Joken or PlugJwt.

Things may be more urgent if you find that the legacy application was not using secure password practices, e.g. it is storing passwords in clear text. In that case, it might make sense to first migrate the login process to Phoenix and fix the security issues, while creating sessions which are compatible with the legacy system. As you transition, you may need to expire sessions, forcing people to log in again. This lets you e.g. update from MD5 hashes to bcrypt, capturing their password, generating the new hash and updating the database.

In an enterprise environment, where you have a lot of systems that may need to interop for a long time, then you can use a single-sign-on system like SAML.

Convert the page layout and navigation

For web apps, you may want to have some pages in the new app and some in the legacy app, allowing users to seamlessly work between both apps. The only thing the user will notice is that the Phoenix pages are 10x faster :-).

To do that you need to take the page layout and convert it to Phoenix format, including compatible navigation links.

That may be straightforward, but probably you want to update the style on the new system, and it doesn't make sense to spend too much time on the old design. It can take a surprisingly large amount of time to incrementally update the graphical design on an existing site. It's faster and more predictable to implement a new template and graphical design, so we need a transition plan that spends the minimum amount of work that will be thrown away when we update the design later.

For one legacy site written in Symfony, the original templates were a mess, so we just did "File | Save Page As" in the browser to create the HTML template in all of its horrible glory. We ran it through tidy to make it valid XHTML, then roughly cut it into templates with header, body and footer.

There are Elixir implementations of the template syntax of many different systems. For example you could use Django templates to convert templates, integrating with Phoenix using PhoenixDtl. Long term you are probably better off just converting templates to EEx, though.

Abuse and cryptocurrency business models

2017-12-23T00:00:00+08:00

When I design systems, one of my favorite things is looking at "abuse cases" which define how they behave when confronted by bad actors.

I am a big fan of cryptocurrencies. They give us an opportunity to design systems which enforce and incentivize behaviors, e.g. removing risk and rewarding good behavior. It's important to recognize that these are new, unregulated markets, though, and some of the protections from bad actors that we expect aren't there unless we build them in.

A replacement for credit cards?

If we think of Bitcoin as simply being a decentralized replacement for credit cards, then we may forget the consumer protections which are built into the current credit card system.

If I am a merchant selling products from my online store, criminals can buy my products with a stolen credit card. When we charge the card, it looks good, and we ship the products. A month later the card holder gets their bill and disputes the transaction. We have to refund their money and take the loss, or we get in trouble with the credit card company and lose our ability to make sales.

From the merchant's perspective, one of the nice things about Bitcoin is that transactions are final. If there is fraud, where is my incentive to refund the money? This goes against consumer expectations and consumer protection laws.

There is a very interesting opportunity to rate vendors and consumers to deal with this situation, but it's not part of Bitcoin.

A lottery?

In most lotteries, the players are ok with the payouts not matching the inputs. Governments take advantage of this to fund various things, and, as my economist friend says, "lotteries are a way for the government to convert post-tax money into pre-tax money".

The winners are happy with this situation, as they get a windfall. Some people say that a lottery is a tax on people that can't do math, but even the losers gain something they value, the hope of winning.

Lotteries tend to be very heavily regulated due to abuse. In a legal lottery, there can be a complete accounting of the money going in and going out. It's actually a perfect application for a blockchain system.

Years ago there was no government lottery in Taiwan. Chinese people like to gamble, though, so the mob offered an underground lottery based on the Hong Kong government lottery. They sold tickets to Taiwanese customers, then when the Hong Kong lottery announced its winners, it would use their numbers and pay out.

There was no accountability in the first place for what percentage got paid out, and there was an interesting twist: if you won, then they would negotiate with you about how much you would actually receive. Getting out your winnings was a lot harder than putting in your money, but the people who won were still happy.

A gambling platform?

A company once contacted us for a quote on a website for online sports betting. We designed a system and gave them an estimate to build it. They said, however, that they had found an open source package that would do the job, written in PHP. We had a look at the code and found a lot of issues with accuracy, error handling, etc. I told them, but much to my surprise, they were unconcerned.

In a standard gambling system, the odds are a mathematical reflection of how people bet, e.g. 2:1 odds means that twice as many people are betting for one team to win compared to the other. The house makes their money by taking a percentage of the bids / payouts. In sports, though, it's common for people to bet for their home team whether or not they actually think it will win. So some gambling site operators take people's money but don't change the published odds. If people are being emotional, then they will bet on a losing team and the operator doesn't have to pay out. In this case, the accuracy of the system is less important to the operator.

There is an interesting opportunity for perfectly accountable gambling systems based on the blockchain.

A stock exchange?

Right now, I am wondering whether the current cryptocurrency exchanges look more like the gambling systems or lotteries described above than their regulated equivalents. What does a bad actor exchange look like? How could you tell?

ABC device support and explicit release criteria

2017-07-14T00:00:00+08:00

One of the most important decisions we make in product development is when to make a release. From a business perspective, it's better to release early and often, with a "minimum viable product".

It's also important to define explicit technical quality criteria, or we will waste a lot of resources and miss our target dates. We need to focus development and testing resources to provide the best possible balance between quality, time to market and implementation effort.

As part of this, we define an explicit set of target devices and put them in three classes, A, B and C. On a web project, our target devices are web browsers, and on a mobile project they are mobile devices.

Class A devices are continuously tested, e.g. on every ticket. If a feature doesn't work on a Class A device, we don't release.

Class B devices are supported in a degraded mode. They need to work, but may have less functionality. For example, they may be be too old to support advanced graphical effects, but all of the core functionality works. We test Class B devices as part of the release cycle, e.g. once every two weeks. We might allow a release if there is a minor problem on Class B devices, but block the release it if there is a serious problem.

Class C devices are explicitly out of the support window. They may be too old, unusual, or too new. If it works, we are happy, but we won't put in extra effort to support them.

Web example

On web projects, for Class A, we might support IE 10, latest Chrome and latest Firefox. For Class B, we might support Safari on Mac and IE 8. Safari only represents about 5% of total browsers, but is the default browser for Mac. Similarly, IE 8 is the last version of IE that runs on Windows XP, so if we have XP users, then we need to test on it. For Class C, we might say that we do not support IE 6 or Microsoft Edge browsers.

On some projects, mobile browsers are Class C, and other projects they are Class A. For example, we might support Mobile Safari on iPad even if we don't support Safari on Mac, e.g. the "lying in bed shopping for clothes" users sometimes represent 40% of sales for women's e-commerce sites. If we are making a native mobile app for a project, then full support for mobile web users would be excessive, we would detect them and display a page pointing them to the app store.

Mobile example

Apple users typically upgrade immediately to the latest iOS version that will run on their phone, so we normally only support the latest iOS version as Class A. Android users typically upgrade the OS when they get a new phone, every two years. So an OS release starts with about 5% of the market, holds there for about six months, then at about 18 months we see a very rapid shift (see the latest Android Play stats). So the latest Android OS version normally gets Class B support. The top 70% OS versions by usage get Class A support. Older OS versions get Class B support, and the bottom 5% gets Class C.

Android tablets are typically not a primary platform, so they are Class B or C, but if we were making a restaurant Point of Sale system, they would be Class A.

Apple hardware is very consistent, so we can realistically support all the modern iPhones. There is an incredible variety of Android hardware, however, with different screen sizes, screen resolutions and hardware capabilities (e.g. CPU, memory, graphics acceleration). We need to aggressively prioritize the hardware devices that represent the most popular and representative phones used by customers in the target market.

Developers normally use Google Nexus phones, as Google themselves does the Android OS port and releases regular OS updates. So that platform is automatically tested on every release. But it's common for something to work on the developer's phone and not work on an older, low end phone.

For startups, we normally support a flagship Samsung device and a few other popular top 10 phones. After that, we rely on partners and crash reporting software to identify problems.

We have produced apps with tens of millions of downloads. At those levels we effectively need to support every device, but we are dealing with all kinds of broken hardware and we need to prioritize testing efforts.

Quality criteria and risk management

We need to release and iterate in order to move forward with the business. Having explicit release criteria allows us to hit release dates effectively, getting to market and avoiding wasting resources and losing credibility with partners.

It is always possible to improve a product, but having regular releases is more important than having perfect releases. As we get to a final release, we need to objectively evaluate which defects should block the release and which we can fix later (or never).

We typically do that by classifying defects by severity vs impact.

In general we want Class A devices representing the majority of our users to have no significant defects (nothing High or Medium severity). A severe defect that affects 5% of users but has a workaround might be acceptable. A low severity bug that affects all users might be acceptable, e.g. a cosmetic or performance problem in certain situations.

We also need to use statistical techniques to identify how many unidentified defects may be in the product. For example, if we have done final release testing for one week and we have seen no High severity defects, one Medium severity defect and 10 Low severity defects, we might say that we can release if we have no Medium severity defects at the end of the next week and a maximum of 10 Low severity defects on Class A platforms and one Medium defect on Class B.

At a certain point we reach diminishing returns on testing and we need to present the product to the full set of devices and use cases to see the issues.

Some applications have particular high quality requirements, e.g. health care or financial. A "small" bug dealing with money has a much bigger impact than an ugly button. Problems with health care applications can literally cost lives or expose us to lawsuits from leaking protected health information.

In these cases we need more than quality assurance processes, we need processes to mitigate risk, detect problems and respond to issues. Monitoring becomes very important to identify issues in production and quickly deploy fixes. For embedded systems, we need to be able to upgrade devices remotely. It does us no good to fix a bug in our office and not be able fix devices deployed on customer sites.

Psychologically, it's difficult to put something out in the world that is not perfect, but we need to do it. If we fail to have explicit release criteria, we may delay the product launch until everything is perfect. The effect may be that we lose credibility with partners and investors. We make the release process so hard that we lose the ability to iterate and learn from our customers, and, ironically, produce a worse product.

Anti-pattern: graphical design driven development

2017-07-14T00:00:00+08:00

Everyone wants to have a beautiful graphical design for their product. The problem comes when graphical design becomes more important than usability and affects the efficiency of the development process.

There is an anti-pattern we call "graphical design driven development." The way it goes is that the client starts by creating a graphical design for their product in Photoshop. The designer focuses on making something beautiful. Because Photoshop can do anything, they make everything custom, e.g. custom fonts, custom buttons, drop shadows, custom controls. They tweak the margins below the headings, add hairlines, make it really shine. They add special user avatars, images and content, so each page in the mockup looks great.

They make a page flow diagram, and give it to the developers to estimate. Developers don't really see the graphical details, they just count buttons and think of the logic, database and communications protocols. They create an estimate, the client approves and they get started. They implement all the custom things on the page exactly as the designer drew it, going through a couple of iterations to get it pixel perfect. It looks great on iOS, but is really hard to get the design perfect on Android because of the different screen sizes / resolutions and dynamic layouts.

It ends up taking more time than everyone expected, but finally it's ready, and we give it to beta users to start using the app with real content.

We find that the content is too big to fit on the page, or too small and looks lonesome. Normal users don't bother with avatars, so we have a line of "tombstones" with the default user avatar. We find that some features are hard to use. It takes too many clicks to do common things. Some pages need to be split up, others combined. That custom control we implemented doesn't get used, or needs to be modified.

We need a new design for the new pages. So the graphical designer comes in again, and they spend a week or so making new beautiful designs, while the developers wait. The developers implement all the custom details again, and we have a round or two of tweaks. Or perhaps the graphical designer has been working on new designs while the developers were doing the initial development. Everyone is excited about the new things in the pipeline, and the investors are asking when it will be done. We keep telling them that it will be there soon, but we are losing credibility with every delay. How hard could it be to just implement a few pages of buttons?

We spent all our budget getting the app done, but at least now it's finally released and beautiful. We can see some things that are not optimal, but changing is so painful, we don't want to do it.

There is a better way.

Fundamentally, the most important thing for your product is whether it helps your users achieve their goals. Feeling is important, and graphical design is a big part of that, but we need a process that delivers usability first. It may sound like I am hating on the designers, but the good ones understand this. Having "concept" graphical designs can help us with fund raising presentations and overall UI approach, but they can't drive the design.

We start with a user-focused process that defines user personas and goals, then user stories. Once we have a good base, we start prototyping the application.

One of the best ways is to start with Keynotopia. They provide a set of reasonably-priced templates which you can use inside of Keynote or PowerPoint to create your user interface. Their library of standard mobile buttons and controls let you make realistic user flows which switch pages by clicking on buttons or links. Entrepreneurs and product managers can create the initial prototype without a designer, and we can go back and forth quickly to iterate on the design. We don't have to wait on graphical design or use specialized software like Photoshop or Illustrator.

When we are happy with the prototype, we start implementing it as a real mobile app. We first create a skeleton app which has the pages but minimal real logic. We use Interface Builder to create the UI by dragging standard controls onto the pages and connecting them. We use mocked up static data and do as little custom UI work as we can, focusing on how the app works dynamically to handle user tasks, not what it looks like. If we find we need to make changes to the pages, we can do it with minimal rework. We can show the investors the prototype and they can try it out themselves. It's clear that it's a work in progress, but it's functional and doesn't crash. They see progress in the initial UI prototype and in the first version of the app. We know we have gotten it right when they say things like "I could use this today, I don't care what it looks like."

Next step, the graphical designer comes in and creates a beautiful design based on the actual app. While they are doing that, the developers implement the application logic and communication with the back end.

This way we do the prototyping and iteration work at the beginning when change is cheaper, using presentation software instead of code. We create a development road map which is well defined and predictable, allowing us to hit our delivery dates. During development we create small feature tickets which can be completed incrementally instead of a monolithic and ill defined "graphical spec" which is always "almost done." We avoid thrashing the dev team with changes, they can sit down and execute the bulk of the app all at once. The process is more efficient, faster and saves money. And it ends up being a better app, beautiful and usable.

Development effort of Android vs iOS

2017-07-14T00:00:00+08:00

We often need to estimate development projects which have both iOS and Android. It's tempting to say that Android will be the same, but what we have found is that Android takes more effort.

The rule of thumb in Silicon Valley is that it takes two to three times the work for an app with an equivalent level of polish as a "world class" iOS app. That may be true for some applications, but is generally excessive when you are first starting.

We generally see about 30 to 50 percent more time required to build the initial Android app compared to iOS. Then the effort increases again when we need to support a wider range of devices or for applications which are more dependent on the hardware, e.g. using video or Bluetooth, or have complex user interfaces.

There are a number of reasons for this, primarily coming down to quality of developer tools, platform fragmentation and testing effort:

1. The development tools for Android are not as productive as for iOS

Apple provides the very mature Xcode integrated development environment for iOS, and everything works well out of the box. Tools like Interface Builder make it easy to lay out UIs graphically, the debugger and profiler are easier to use, and the simulator compiles the app to run natively and quickly on the dev machine.

Android developers use Android Studio or general purpose IDEs like Eclipse. They have to set up their environment from multiple pieces and plugins. Build performance is much slower than on iOS (minutes instead of seconds), making iteration slow. The app runs ARM CPU code in an emulator, which is slower than the actual hardware.

2. Java is a lower-level language than Objective-C or Swift

Objective-C and the new Swift programming language are higher level, making coding more efficient. Android apps typically have 40% more code than iOS apps.

Apple has also made some fundamental design decisions which give a better customer experience. A common programming mistake in both platforms is a null pointer exception. On iOS, this results in an empty result object. On Android, it results in a crash, which gives a poor user experience and perception of quality. For the developer, it wastes time as they need to restart the app and go back to where they were working whenever a crash occurs.

3. Android APIs are lower level and less complete than for iOS

Android often requires a 3rd party library where iOS has functions in the standard framework. This makes it harder to use multiple libraries together and requires the developer to spend time finding and evaluating library options.

High level widgets and consistent UI make prototyping easier on iOS. For example, the CoreData API on iOS makes it easy for developers to work with data "objects" and store them without considering the low level details. Android developers need to use raw SQL with databases or other 3rd party frameworks.

Custom screen animations are easier to implement in iOS, and have hardware acceleration making them faster and more energy efficient.

4. The iPhone has a limited number of display aspect ratios and resolutions

For iPhone, it is reasonable to design for specific screen sizes, making "pixel perfect" UI mockups in Photoshop and implementing them exactly in the app. Testing on a few devices covers all the important resolutions, and it is practical to test on all the iOS devices in the market.

Android has many different screens, effectively every variation possible is on the market. This analysis and visualization of Android device fragmentation is good, though a bit old. Here are up to date stats for Android from the Google Play Store.

There was a joke was that Samsung didn't know what the right screen aspect ratio was, so they made every variation and counted what people actually bought.

Because device capabilities vary widely, developers have to work in "logical" device-independent pixels and translate into device pixels. Text may have different thickness at different screen resolutions, causing differences in appearance, margins and line breaks.

Developers need to use dynamic layouts which change the relationship between elements to accommodate different screen aspect ratios. Spacing between elements is fundamentally variable. Instead of using the drag-and-drop Interface Builder like in iOS, developers generally need to code user interfaces in XML text files.

Sometimes these dynamic layouts result in poor results. A tall and narrow display may have too much space between vertical elements and unsightly line breaks, and an especially wide display may have blank spaces on the sides of the screen. It is possible to create special optimized layouts for different screens, but that requires extra work and can only cover a subset of the market.

5. Android hardware is highly variable and software support is buggy

iOS hardware is powerful, at the top end of the market. It is reasonable for iOS developers to only target newer devices, as users upgrade regularly. Apple aggressively stops supporting older hardware with newer OS releases, pushing people to upgrade.

Android has some extremely low end devices, often of poor quality and unbalanced resources, e.g. low end tablets run phone chips combined with big displays and may not have enough RAM to handle the screen.

On iOS, each app gets the hardware to itself when it's running. On Android, apps may be sharing resources with other apps running in the background. The app is also limited by the OS from using all the resources that are available.

The iPhone uses the GPU to accelerate the user interface, and has APIs to support animations. This gives better performance for scrolling or fancy UI effects with no additional work for the developer. The iPhone has advanced features like fingerprint recognition or advanced Bluetooth, and they generally work without issues.

The normal process for Android hardware support is as follows: Google releases a new Android version. Chip manufacturers like Qualcomm or Allwinner add drivers to support the special features of their chips, e.g. audio / video codecs. This takes a few weeks or months, depending on their relationship with Google and their access to pre-release versions of Android. Mobile device manufacturers choose chips from the manufacturer and get a reference kernel and hardware design. They may use it as is, or may customize it to differentiate themselves. They may have more or less software engineering ability in house, and may not even have complete datasheets for all the features of the chips they use.

The result is that there may be bugs in hardware support which cause strange results and crashes when using features like video or Bluetooth. Mass market apps need to work with all of the devices in the market, so they can't assume advanced hardware features. Developers need to design for the lowest common denominator, provide fallbacks and do more performance optimization, which takes more development time. Carrier and OEM preload agreements can force Class A support for marginal hardware.

In applications which have millions of users, we can expect to see crashes due to broken hardware, e.g. bad RAM. Debugging these problems can be extremely difficult, because there is not fundamentally anything wrong with the software.

6. Android has many OS variations

Most iOS users are running the latest operating system release within a week of the release. More than 95% of users with up-to-date hardware run it immediately. Overall more than 80% of users run the latest release.

Android users rely in their device manufacturer and carrier to provide OS updates, and most do not. They expect users to buy new devices and consider the new OS to be an incentive to buy.

The result is that less than five percent of Android users are running the latest Android OS release months later. About 30% run the previous release family, 30% are two releases back, 30% are three back, and less than 5% run very old releases. New OS penetration comes as people get new phones every two years. In developing markets, significant percentages of customers run used phones until they die or run low end hardware that can't handle the latest OS.

Manufacturers such as Samsung differentiate themselves by customizing the user interface. This takes time and may result in unique platform bugs or incompatibilities. Some manufacturers modify how the OS works in fundamental ways, e.g. automatically saving a copy of every photo the user takes in a gallery. This causes problems for health-related apps which need to maintain privacy of patient data.

7. Apple is more secure than Android

Apple differentiates itself with security and privacy compared to Google. Apple sells phones and Google sells advertising. Outside of niche devices like the Pixel, Google does not control the hardware. It makes money from services, not hardware, so it wants the most users possible no matter what they are. Apple supports features like finger print recognition and hardware security modules.

Google does not police the app store for quality the way that Apple does, and carrier- or 3rd-party app stores are common. Users may install pirate apps or side-load unsigned apps. The OS is more likely to be "jailbroken". Users may be less sophisticated and install malware, or they may customize everything up to the point of installing custom firmware.

iOS apps are restricted in what they can do. Android apps can do almost anything, including running in the background and capturing keystrokes, listening to the microphone, reading the user's location and sending everything to the net.

Android devices often have external storage and support plugging in USB devices. iOS devices require permission from Apple to connect hardware.

The result is that creating high security apps on Android is much harder, as they need to operate in a hostile environment.

8. Testing effort

Creating a high quality app is already more difficult, then we need to test on a wide range of devices.

Applications with many users must test on tens or hundreds of devices. Developers must debug problems on devices that they don't have access to, so they rely on issue reports from users or software which records crash dumps and sends cryptic reports.

This effort can significantly increase the cost of developing an application as the user base increases. It's necessary to make explicit decisions about which devices will be tested and define release criteria and triage issues based on ther impact on the user base.

Links

How to port an iOS app to Android

This consulting company found an average of 30% more time for Android on their projects when making the same apps on both platforms, with a lot of variation

Discussion of development issues:

An example of user personas

2017-07-12T00:00:00+08:00

When we create products, it's important that they help specific users with their issues, not generic users. It's easy to create a list of features, all of which sound good, but don't provide a compelling solution to a specific problem for a specific user. Without that, people won't buy your product. If you make one user happy, then you can make a sale, and then expand out from there.

User personas (or personae) are a way of getting into the mind of your users and figuring out what they need. We define the users and then describe their background in what may initially seem like excessive detail. But by defining this background, we can get at their motivations and the emotions associated with their decisions.

There is a lot of overlap between user personas for product definition and user personas for marketing. Once you have defined your users, you know how to talk to them, where they hang out, and what messages will attract them to your product.

An example

Say we are working to define a product for the accounting needs of small businesses, similar to QuickBooks or Xero. We have identified a few personas to help us understand people's needs, skills and motivations.

Bob

Bob is the manager of a small contracting company with four employees that mainly does kitchen renovations. He is 45 years old, and has been running his own business for about 15 years.

He is ok with a PC. He mainly uses Excel to keep track of expenses and manage the books for his company. He doesn't spend much time in the office, though, as he makes his money working at customer sites.

He understands the business side of invoices, and has the scars to prove it. He has a big stack of paper on his desk. He has unnecessary cash flow problems because he pays for materials before getting reimbursed by his customers, but his records are disorganized and he doesn't know what people owe him or what they have paid. He doesn't have any formal training in accounting.

He hates to do admin stuff, as it takes time away from being with his family.

He was recently fined by the IRS for failing to get his taxes done on time. This is motivating him to "get organized" and get a "real accounting system" in place.

He decided to hire his niece, Alice, part time to help organize things. He is a bit embarrassed to let her see how bad things are.

Alice

Alice is a 17 year old senior in high school. She is a "digital native", who has been using computers all her life, and uses her phone for hours a day. She doesn't know much about business, and certainly not accounting. She had a class in high school in Microsoft Office.

Her mother told her she needs to help Bob. She doesn't really know what she is supposed to do, but having a bit of money is nice. She wants to show that she is independent and can get the job done so her mom will stop treating her like a child.

Nancy

Nancy is Bob's bookkeeper. She is 40 years old. She used to work as an accountant at an auto parts wholesaler, then quit to have kids. She started doing the books for small businesses on the side about five years ago. She charges $100/month, fixed price. She has 20 customers now. The money is OK, but she wishes she didn't have to deal with so much chaos. She doesn't like having to chase people to give her what she needs to do her job, it takes up as much of her time as the actual accounting.

Bob has the real pain points: he needs something to help him get organized and improve the way his business works without spending too much time or money on it. Ideally he would be able to see reports about who owes him money, easily create invoices and figure out if he is making money on different projects. He needs to be able to keep track of expenses paid for by him and by his team. He recently got a store credit card at Home Depot so he can track expenses, but he still needs to match them to the job. Having the ability to take pictures of receipts from his phone to get them into the system would be nice, sometimes he loses receipts and it causes him a lot of trouble.

Alice is responsible for tracking expenses and payments. She gets the bank statements and matches money received to outstanding invoices and money paid to other expenses like the truck, the rent and electricity. Some bills are paid from the bank, others via check, others via credit card. She needs to enter paper receipts and checks received.

She doesn't really know what these words mean, though. She needs the software to walk her through the process. Bob will be there sometimes, but she needs to be able to do things by herself if possible. She will be working after school and on weekends when Bob may not be around. If she could do it from home instead of his office, that would be even better.

Nancy mainly wants to get her job done as quickly as possible, since she doesn't make any more money by having it take longer. If she didn't have to visit Bob or have to call him on the phone, so much the better. She needs to have all the data in the system, then generate the reports for the IRS. She doesn't need hand holding from an accounting perspective. She doesn't want to know who Bob's customers are, they are just names on accounts to her.

We could define some other people in this scenario, e.g. members of Bob's team, his customers, his suppliers. We could also think about how the software could help Nancy's business run better and get more business herself. For example, she would like to have something to keep track of her customers and help her follow up on what they should give her and other important deadlines.

We can define users from some other small businesses, e.g. a restaurant with part time staff, a retail operation that has lots of transactions, inventory accounting, etc.

Ultimately, the most interesting insight may be that Bob doesn't actually want accounting software, he wants his accounting to be done. This is a trend that I call "Software As A Service plus Service". We could provide a software platform plus the people to run it for our customers. This is particularly useful for SMEs, where a key problem they have is that they don't need (and can't afford) a full time person to handle a task like accounting, unlike a bigger company which may have a whole accounting department.

An example of user stories

2017-07-12T00:00:00+08:00

User stories are the "Director of Operations" view of the world. They describe step by step how the business works, and what the software needs to do to support it. They should be done after we have defined the user personas, to make sure we are starting from users and their goals.

An example

Let's say we are making a food delivery service.

A customer hears about the delivery service by seeing a flier on the counter when they are going to their favorite Mexican restaurant. They take the flier, and the next weekend it's raining, so they decide to try the take-out service.

They download the mobile app using the QR code on the flier and search for the restaurant's profile. They choose the food they want and make their order. We will assume cash on delivery at this point, though we could also use credit cards or some other mobile payment method.

The mobile app sends the order to the service website via the API, then the server notifies the restaurant that there is an order.

The cashier at the restaurant has the restaurant side of the mobile app in her pocket. It buzzes and she takes her phone out and confirms the order. Accepting the order schedules the driver to pick it up.

The cashier writes the order down on their standard paper order sheet and gives it to the kitchen. She adds the order number from the delivery system as well, connecting their order and the delivery service order. We could also integrate with their POS and kitchen management system, but that's out of scope right now.

The driver comes in with a list of orders on his phone. He checks off each one using the order number on the paper and leaves with the food in his messenger bag.

The driver's phone app gives him a suggested sequence of delivery locations. He goes to them one by one, giving them the food and taking the money. He checks off each delivery as he does it. The system knows where he is from the phone's GPS and can tell the customer when he is getting close to their house. The app can give directions to the driver using Google navigation.

What can go wrong?

There are various things that can go wrong at each step. These problems and how we deal with them are the most important thing to think about, they can make or break the service.

For example, the restaurant might be too busy to deliver when the customer wants. Or they might not acknowledge at all. Or they might make a mistake with the order. Or the driver might pick up the wrong bag.

We have some standard expected time frames, e.g. delivery within 45 minutes. So if the restaurant can meet that, then the cashier can simply acknowledge the order. If they are busy and can't do it in the standard time, then she might propose another time, e.g. one hour, or reject the order entirely. Then we would need to notify the customer to see if the time is acceptable or give them an opportunity to change to a different restaurant.

If we get the order wrong, the customer might accept it if we give it to them for free along with a coupon for another order. But if the delivery guy trips on the stairs and drops the food, then we have a bigger problem: a hungry customer and no food. We might need to expedite an order from the restaurant to take care of them. We might need a customer service phone number that they can call, as we can't expect delivery people to have good customer support skills. We also have a branding problem: the problems are generally going to be the restaurant's fault, but the service may get blamed for it.

We are relying on mobile phones a lot, but we need to make sure that we can survive if they fail. We would like to be able to start working with paper in the first implementation stage, then implement the mobile apps. We have a source of human error if we write on order forms, so a printer in the restaurant would be nice. But hand writing is still the backup. If the driver drops his phone and breaks it, he could maybe work from a print of the order stapled to the bag which includes the customer info.

We need to consider how having the cashier take a phone order will make customers standing in line feel about having to wait. This emotional aspect is an important part of user personas.

Next steps

The next step is to figure out what we need to get started running this business, and how software can help. If you can do it with Excel or paper in the beginning, then you can start running a "prototype" of the business and validating your assumptions. Other parts like a nice looking website might be necessary from a marketing perspective.

After user stories are done, we prioritize "paths" through the application, and write detailed technical use cases, estimate them and implement them.

Presentation on Elixir performance

2017-06-25T00:00:00+08:00

Here are the slides for the presentation on performance tuning Elixir I gave to the local Elixir user's group.

It's only a minimum viable product if it hurts

2017-04-08T00:00:00+08:00

One definition of a startup is "a company in search of a repeatable business model." If we have a clear idea of what the customer needs, what they are willing to buy, and have built the product that satisfies them, then we are no longer a startup, we are an established business.

Maybe we understand customers and the market well enough that we can sit in our cave, build the product, then release. But it's more likely that we will get it somewhat wrong the first time and need to change direction. The process of building a startup is iterating as quickly and efficiently as possible to get to product-market fit before we run out of money.

The term "Minimum Viable Product" means that we define the smallest "complete" product which can be launched and satisfy a customer. A lot of entrepreneurs get caught in the trap of thinking that they know what they are doing, or they pay lip service to MVP and don't do it. Instead they need to actually build the product step by step based on feedback from customers.

It can feel bad to make a MVP, because you show it to your friends or customers or investors and they say "is that all there is?". Or you get worried that the product will not be successful, and think "in order to be successful, a product needs to have more features than the competition", so you add more and more features to the initial release. But then you burn all your budget going in the wrong direction and have nothing left to make necessary changes or to market it.

One of the key things that drives the real MVP is marketing. You only get one chance to make a first impression, and it can cost a lot of money to do a proper marketing campaign. On the other hand, if you do a "soft launch" with your MVP, then you can get customers without having to be big and perfect. You just have to meet the needs of a set of specific customers, then you can build on in the next release. So it's a matter of setting expectations properly, then showing progress.

You may be better off reserving say 1/3 of your budget for customer development at the beginning and marketing and sales at the end, 1/3 for initial product development, then 1/3 for a second phase of development. This ensures that you actually get a product to market. It can also bring in revenue which funds the business (bootstrapping), reducing reliance on later funding rounds.

In the movie business, they have a concept of a "completion bond", a form of insurance. It makes sure that a movie can be finished when it has run out of money. If the film is not released, then it makes no money, and all the investors get nothing. The insurance company comes in with a hard-core penny-pinching producer who finishes the movie and gets it into theaters. We don't have this in startups, unfortunately.

If you really take this to heart, there is a lot that you can do at the beginning by focusing on reducing risk. For example, you can visit potential customers and interview them about their top ten problems. You may have a good idea, but are building a solution for problem #7 instead of #1, and all the budget is gone before the customer gets to #7. You can get them to sign a "letter of intent" that says the will buy the product if you build it. It is non-binding, but makes them take it seriously and talk with other people in their organization to get approval to sign something. That helps avoid the trap of customers being friendly and supportive to talk with you, but not ultimately being really serious about buying your product.

The key is to build relationships with target customers. If we can get the customers to feel ownership in the solution, then they can become powerful advocates, e.g. providing initial references and introductions.

There is a danger that if we pay too much attention to the needs of our initial customers that we don't build a general product, we become a consulting company building custom solutions. But at least that makes money, and is a better problem to have, and gives us more insights into customers which we can turn into products.

The Four Stages of Santa Claus for software developers

2016-09-08T00:00:00+08:00

There is a joke, "The Four Stages of Santa Claus":

You believe in Santa Claus
You do not believe in Santa Claus
You are Santa Claus
You look like Santa Claus

There is an equivalent for software developers when it comes to requirements documents and specs.

In the beginning of your career, you only see complete specs, and you act like Santa Claus brings the requirements in the middle of the night and puts them under the tree.
You realize that someone had to write the requirements, but it's not you. You complain that the specs are not clear or "the customer doesn't know what they are doing."
You realize that you need to write the specs and clarify the requirements, or the project is going to have problems.
You run a software consulting company and it gives you gray hairs :-)

In software development, we have to bridge the gap between the "requirements" side and the "solution" side. That means we have to figure out exactly what the goals of the system are and how we are going to satisfy them, in detail. It's everyone's responsibility to do this, from the product manager and project manager on the requirements side or from the tech lead and dev team on the solution side. There is no Santa Claus, unfortunately.

If you are not doing everything you can, then you are making extra work for other people. Your tech lead or project manager can do it, but they have plenty of other work to do. Sometimes it is a business person, and it causes real problems because you are expecting them to do things that they can't do. Programmers are good at logic, and sales people are good with people. It works a lot better when you understand the business requirements and write up a proposed solution for the business owner to approve.

This is why outsourcing to cheap programmers in the developing world often has problems. The client needs to be able to write the spec in incredible detail. The developers can't bridge the gap because they are too junior or don't have enough understanding of the business context.

Sometimes in projects there is a situation I call "someone needs to think hard and make some decisions." This can happen if we don't get complete requirements up front, or we are iterating on product / market fit. It's also something to watch out for in an agile process where we are implementing cycle by cycle: everyone is doing something that works for this iteration but we still need to pay attention to the big picture. It's easy to coast along without decisions being made, wasting time and causing bad user experiences which need to be reworked.

For example, say we have a SaaS product, so we need a registration and purchase process. The developer says, "tell me what the registration process is." The designer says, "give me the screens and I will make it look awesome." The entrepreneur says "I don't know, I am a business guy".

The best way I have found is to start by understanding / defining your target users with user personas, then determining high level user goals / pain points and their ideal user experience. Then we define user stories which show how the product will help people achieve their goals. Next we go through how the service will work step by step, identifying any problems that may occur and how we will deal with them. I call this the "Director of Operations" view. It's not technical, it's still on the "requirements side", but it's sweating all the details. With this preparation, we can architect a technical solution which meets these requirements. If we skip directly to the technical side, then there is a danger that we will build the wrong thing.

Presentation on Elixir and embedded programming

2016-07-25T00:00:00+08:00

Here are the slides for the presentation on Elixir and embedded programming I gave to the Elixir user's group.

90 percent immutable

2016-06-10T00:00:00+08:00

After a fair amount of debugging, I got an app running in an AWS Auto Scaling Group (ASG), pulling its config on startup from S3 and code from Amazon CodeDeploy. On the way I found out some annoying parts of the cloud initialization process in AWS.

The idea is that we can build a "generic" AMI which has the application dependencies, then when the ASG starts up an instance, it will get the latest code and configuration.

Initially, I was planning to use the instance tags to keep bootstrapping configuration information, e.g. whether it's running in staging or production environment. This would let the instance get the config from the right S3 bucket, contact the right RDS instance, etc.

I wrote some systemd init scripts, one of which reads the EC2 instance metadata and tags and writes the data to files on the disk in JSON format and as a shell include file. Another script syncs the data from S3 to the local disk, and the third starts up the application. Getting the dependencies set up properly on these scripts ended up being tedious and difficult to get running reliably.

The fundamental problem is that it takes some time for the metadata to be available after the machine starts. And, critically, instance tags are not available until after CodeDeploy runs.

As part of the deployment process, CodeDeploy takes an instance out of the auto-scaling group, upgrades it, tests if it's healthy, then puts it back. When CodeDeploy launches a fresh instance, it puts it in Waiting state, then loads the code, then enables it. But instance tags are not available in Waiting state, they are only available when the instance is ready. So you can call boto and read the tags, but they won’t be there, the list will be empty. And you can’t wait for them, because they won’t be there until you start successfully.

The next problem is that it takes some time for the basic metadata to be available. So the startup script may run, make a HTTP request to the metadata service and not get the data, so it needs to sleep and retry.

What I ended up doing is putting the parameters in base64 JSON in the user_data field, which is available, though you may have to wait a while to get it. And I set up hard dependencies between the startup scripts. So the metadata service runs first, then the S3 sync script (which needs the metadata to know which bucket to read from), then the application startup. Yay systemd.

And to make startup faster, I added manual calls in each script to call the previous script to get its dependencies if they are not there.

Making this even more fun to debug is that it works fine in a stand alone instance, but has problems in the auto-scaling group. And when the instance fails in the ASG, the app doesn’t start up so it fails the health check, so it gets shut down and another instance runs, over and over again. So when you are debugging, you ssh in and then the instance is abruptly terminated, and you wait for the next instance to be started so you can try again. And each debug cycle takes 20 minutes as you build an AMI and deploy it. Sigh.

After all that, I am tempted to use the same lifecycle hook that CodeDeploy uses to drive Ansible i.e. an instance starts up, and it pushes a message to SNS/SQS. A python script sits on the ops server waiting for it to happen when it gets the message, it runs an Ansible playbook on the instance to configure it and deploy the code.

See the lifecycle hooks docs for details.

ANSIBLE ALL THE THINGS!

This is a case where the Chef "pull" model might be more convenient, though in general I like Ansible a lot. I find that Ansible is better at creating the instances in the first place. I like the fact that Ansible doesn't need an agent and is basically just a list of canned tasks. The list of tasks is comprehensive, and it's easy enough to define your own. Ansible is also natural for the systems admins on the team.

So here we are at something like 90% "immutable infrastructure". I could go all the way and burn the deps, env-specific config and code into an AMI and launch it from the ASG.

Packer makes it quite easy to build AMIs, and I am kinda tempted at this point, but it's still slow enough to be annoying.

Adding the runtime configuration would mean putting "secrets" into the AMI. I didn’t find a really satisfactory way of passing the vault key into Ansible. By not satisfactory, I mean it worked, but was a bit hackish. i.e. you somehow get the password into an env var, which gets passed into the Ansible script which means it visible in your terminal. Or if you are paranoid, you can use temp files which you hopefully delete. So choose your poison.

Here is an example Packer file:

{
    "variables":{
        "pass": "{{env `ANSIBLE_VAULT_PASS`}}"
    },
    "builders": [
        {
            "type": "amazon-ebs",
            "region": "us-east-1",
            "source_ami": "ami-123",
            "instance_type": "t2.micro",
            "ssh_username": "centos",
            "ami_name": "foo app {{timestamp}}",
            "vpc_id": "vpc-123",
            "subnet_id": "subnet-123",
            "associate_public_ip_address": "true",
            "ami_virtualization_type": "hvm",
            "communicator": "ssh",
            "ssh_pty": "true",
            "launch_block_device_mappings": [{
                "device_name": "/dev/sda1",
                "volume_size": 8,
                "volume_type": "gp2",
                "delete_on_termination": true
            }]
        }
    ],
    "provisioners": [
        {
            "type": "shell",
            "inline": [
                "sudo yum install -y epel-release",
                "sudo yum install -y ansible"
            ]
        },
        {
            "type": "ansible-local",
            "playbook_file": "../ansible/foo-app.yml",
            "playbook_dir": "../ansible",
            "inventory_groups": "app,tag_env_staging",
            "command": "echo '{{user `pass`}}' | ansible-playbook",
            "extra_arguments": "--vault-password-file=/bin/cat --tags setup"
        }
    ]
}

CodeDeploy

I am pretty happy with CodeDeploy so far. By standardizing the deployment process across apps, our “follow the sun” sysadmin team in Europe, Asia and Latin America can roll back to a previous successful release without needing to know much about the app. So if something goes bump in the night, someone will be able to deal with it during their day without having to get the developers out of bed.

UPDATE

We are now using Terraform to provision instances, with Ansible to set them up. We build an AMI for each environment (staging, production) using Packer, with the config it needs baked in, then deploy the app using CodeDeploy. This avoids the problems discussed here.

Presentation on thinking functionally in Elixir

2016-05-25T00:00:00+08:00

Here are the slides for the presentation on thinking functionally in Elixir I gave to the local Elixir user's group.

Presentation on Erlang and functional programming

2016-05-16T00:00:00+08:00

Here are the slides for the presentation on Erlang and functional programming I gave to the local Elixir user's group.

Cogini

Building and Testing Elixir Containers with GitHub Actions

Breaking up the monolith: building, testing, and deploying microservices

Show me the code!

Testing is critical

Mocks and fake data

Static code analysis tools

Testing in containers

Example CI build and test process on GitHub Actions

Security scanning

Health checks

Developer experience

Jobs vs Events

Kafka architecture

Permanent failures

Transient failures

Job frameworks

Leveraging both

Access to data

Events

Event sourcing

Data ownership

Kubernetes Health Checks for Elixir Apps

Kubernetes health checks

Dependencies

Running OS commands

Higher-level checks

Presentation on thinking functionally in Elixir 2020

Choosing a Linux distribution

Short answer: Use Ubuntu LTS

Long answer

Recommendations

The future

Deploying an Elixir app to Digital Ocean with mix_deploy

Overall approach

Create the server

Configure ssh to talk to your server

Create a deploy user on the web server

Configure ssh access to the deploy user.

Check out the app source

Install build dependencies

Create the database

Create app databases and users

Configuration

Building

Prepare runtime configuration

Build

Initialize local system

Deploy the app

Check that it works

Configure the server to listen on port 80

SSL

How to prepare your Phoenix app for deployment

Generate Phoenix project

Configure releases

Tune the OS for performance

Add runtime config files

Add migrator module (optional)

Set up ASDF

Install mix_deploy and mix_systemd

Copy build and utility scripts into your repo

Configure for running in a release

Deploying complex apps to AWS with Terraform, Ansible, and Packer

The solution

Architecture

Multiple AWS accounts

Sharing resources between accounts

Structure

Meeting legacy apps half way

Deploying with CodeDeploy and Capistrano

Encryption

Show me the code!

Minimal EC2 + RDS

CloudFront for assets

CodePipeline for CI/CD

Auto Scaling Group and Load Balancer

Worker ASG

Multiple front end apps

Shared S3 buckets

Static website

Configure ssh access to the `deploy` user.