This PR adds the @long_callback decorator the was developed in Dash Labs.

Draft documentation below

TODO:

• Integration tests
• long_callback docstring
• Merge Add support for adding "extra components" to an app's layout #1700 and resolve conflicts

Overview

This decorator is designed to make it easier to create callback functions that take a long time to run, without locking up the Dash app or timing out.

@long_callback is designed to support multiple backend executors. Two backends are currently implemented:

• A diskcache backend that runs callback logic in a separate process and stores the results to disk using the diskcache library. This is the easiest backend to use for local development.
• A Celery backend that runs callback logic in a celery worker and returns results to the Dash app through a Celery broker like RabbitMQ or Redis.

The @long_callback decorator requires a long callback manager instance. This manager instance may be provided to the dash.Dash app constructor as the long_callback_manager keyword argument. Or, it may be provided as the manager argument to the @app.long_callback decorator itself.

The @app.long_callback decorator supports the same arguments as the normal @app.callback decorator, but also includes support for several additional optional arguments that will be discussed below: manager, running, cancel, progress, and progress_default.

Dependencies

The examples below use the discache manager, which requires the diskcache, multiprocess, and psutil libraries

$ pip install diskcache multiprocess psutil

Example 1: Simple background callback

Here is a simple example of using the @long_callback decorator to register a callback function that updates an html.P element with the number of times that a button has been clicked. The callback uses time.sleep to simulate a long-running operation.

import time
import dash
import dash_html_components as html
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output

## Diskcache
import diskcache
cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(cache)

app = dash.Dash(__name__)

app.layout = html.Div(
    [
        html.Div([html.P(id="paragraph_id", children=["Button not clicked"])]),
        html.Button(id="button_id", children="Run Job!"),
    ]
)

@app.long_callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    manager=long_callback_manager,
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Example 2: Disable button while callback is running

In the previous example, there is no visual indication that the long callback was running. It is also possible to click the "Run Job!" button multiple times before the original job has the chance to complete. This example addresses these shortcomings by disabling the button while the callback is running, and re-enabling it when the callback completes.

This is accomplished using the running argument to @long_callback. This argument accepts a list of 3-element tuples. The first element of each tuple should be an Output dependency object referencing a property of a component in the app layout. The second elements is the value that the property should be set to while the callback is running, and the third element is the value the property should be set to when the callback completes.

This example uses running to set the disabled property of the button to True while the callback is running, and False when it completes. In this example, the long callback manager is provided to the dash.Dash app constructor instead of the @app.long_callback decorator.

import time
import dash
import dash_html_components as html
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output

## Diskcache
import diskcache

cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(cache)

app = dash.Dash(__name__, long_callback_manager=long_callback_manager)

app.layout = html.Div(
    [
        html.Div([html.P(id="paragraph_id", children=["Button not clicked"])]),
        html.Button(id="button_id", children="Run Job!"),
    ]
)

@app.long_callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    running=[
        (Output("button_id", "disabled"), True, False),
    ],
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Example 3: Cancelable callback

This example builds on the previous example, adding support for canceling a long-running callback using the cancel argument to the @long_callback decorator. The cancel argument should be set to a list of Input dependency objects that reference a property of a component in the app's layout. When the value of this property changes while a callback is running, the callback is canceled. Note that the value of the property is not significant, any change in value will result in the cancellation of the running job (if any).

import time
import dash
import dash_html_components as html
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output

## Diskcache
import diskcache

cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(cache)

app = dash.Dash(__name__, long_callback_manager=long_callback_manager)

app.layout = html.Div(
    [
        html.Div([html.P(id="paragraph_id", children=["Button not clicked"])]),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)

@app.long_callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
    ],
    cancel=[Input("cancel_button_id", "n_clicks")],
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Example 4: Progress bar

This example uses the progress argument to the @long_callback decorator to update a progress bar while the callback is running. The progress argument should be set to an Output dependency grouping that references properties of components in the app's layout.

When a dependency grouping is assigned to the progress argument of @long_callback, the decorated function will be called with a new special argument as the first argument to the function. This special argument, named set_progress in the example below, is a function handle that the decorated function should call in order to provide updates to the app on its current progress. The set_progress function accepts a single argument, which correspond to the grouping of properties specified in the Output dependency grouping passed to the progress argument of @long_callback.

import time
import dash
import dash_html_components as html
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output

## Diskcache
import diskcache
cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(cache)

app = dash.Dash(__name__, long_callback_manager=long_callback_manager)

app.layout = html.Div(
    [
        html.Div(
            [
                html.P(id="paragraph_id", children=["Button not clicked"]),
                html.Progress(id="progress_bar"),
            ]
        ),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)

@app.long_callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
        (
            Output("paragraph_id", "style"),
            {"visibility": "hidden"},
            {"visibility": "visible"},
        ),
        (
            Output("progress_bar", "style"),
            {"visibility": "visible"},
            {"visibility": "hidden"},
        ),
    ],
    cancel=[Input("cancel_button_id", "n_clicks")],
    progress=[Output("progress_bar", "value"), Output("progress_bar", "max")],
)
def callback(set_progress, n_clicks):
    total = 10
    for i in range(total):
        time.sleep(0.5)
        set_progress((str(i + 1), str(total)))
    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Example 5: Progress bar chart graph

The progress argument to the @long_callback decorator can be used to update arbitrary component properties. This example creates and updates a plotly bar graph to display the current calculation status. This example also uses the progress_default argument to long_callback to specify a grouping of values that should be assigned to the components specified by the progress argument when the callback is not in progress. If progress_default is not provided, all the dependency properties specified in progress will be set to None when the callback is not running. In this case, progress_default is set to a figure with a zero width bar.

import time
import dash
import dash_html_components as html
import dash_core_components as dcc
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output
import plotly.graph_objects as go

## Diskcache
import diskcache
cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(cache)

def make_progress_graph(progress, total):
    progress_graph = (
        go.Figure(data=[go.Bar(x=[progress])])
        .update_xaxes(range=[0, total])
        .update_yaxes(
            showticklabels=False,
        )
        .update_layout(height=100, margin=dict(t=20, b=40))
    )
    return progress_graph


app = dash.Dash(__name__, long_callback_manager=long_callback_manager)

app.layout = html.Div(
    [
        html.Div(
            [
                html.P(id="paragraph_id", children=["Button not clicked"]),
                dcc.Graph(id="progress_bar_graph", figure=make_progress_graph(0, 10)),
            ]
        ),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)

@app.long_callback(
    output=Output("paragraph_id", "children"),
    inputs=Input("button_id", "n_clicks"),
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
        (
            Output("paragraph_id", "style"),
            {"visibility": "hidden"},
            {"visibility": "visible"},
        ),
        (
            Output("progress_bar_graph", "style"),
            {"visibility": "visible"},
            {"visibility": "hidden"},
        ),
    ],
    cancel=[Input("cancel_button_id", "n_clicks")],
    progress=Output("progress_bar_graph", "figure"),
    progress_default=make_progress_graph(0, 10),
    interval=1000,
)
def callback(set_progress, n_clicks):
    total = 10
    for i in range(total):
        time.sleep(0.5)
        set_progress(make_progress_graph(i, 10))

    return [f"Clicked {n_clicks} times"]


if __name__ == "__main__":
    app.run_server(debug=True)

Caching results with long_callback

The long_callback decorator can optionally memoize callback function results through caching, and it provides a flexible API for configuring when cached results may be reused.

Note: The current caching configuration API is fairly low-level, and in the future it might be useful to provide preconfigured caching profiles.

How it works

Here is a high-level description of how caching works in long_callback. Conceptually, you can imagine a dictionary is associated with each decorated callback function. Each time the decorated function is called, the input arguments to the function (and potentially other information about the environment) are hashed to generate a key. The long_callback decorator then checks the dictionary to see if there is already a value stored using this key. If so, the decorated function is not called, and the cached result is returned. If not, the function is called and the result is stored in the dictionary using the associated key.

The built-in functools.lru_cache decorator uses a Python dict just like this. The situation is slightly more complicated with Dash for two reasons:

1. We might want the cache to persist across server restarts.
2. When an app is served using multiple processes (e.g. multiple gunicorn workers on a single server, or multiple servers behind a load balancer), we might want to shared cached values across all of these processes.

For these reasons, a simple Python dict is not a suitable storage container for caching Dash callbacks. Instead, long_callback uses the current diskcache or Celery callback manager to store cached results.

Caching flexibility requirements

To support caching in a variety of development and production use cases, long_callback may be configured by one or more zero-argument functions, where the return values of these functions are combined with the function input arguments when generating the cache key. Several common use-cases will be described below.

Enabling caching

Caching is enabled by providing one or more zero-argument functions to the cache_by argument of long_callback. These functions are called each time the status of a long_callback function is checked, and their return values are hashed as part of the cache key.

Here is an example using the diskcache callback manager. In this example, the cache_by argument is set to a lambda function that returns a fixed UUID that is randomly generated during app initialization. The implication of this cache_by function is that the cache is shared across all invocations of the callback across all user sessions that are handled by a single server instance. Each time a server process is restarted, the cache is cleared an a new UUID is generated.

import time
from uuid import uuid4
import dash
import dash_html_components as html
from dash.long_callback import DiskcacheLongCallbackManager
from dash.dependencies import Input, Output

## Diskcache
import diskcache
launch_uid = uuid4()
cache = diskcache.Cache("./cache")
long_callback_manager = DiskcacheLongCallbackManager(
    cache, cache_by=[lambda: launch_uid], expire=60,
)

app = dash.Dash(__name__, long_callback_manager=long_callback_manager)
app.layout = html.Div(
    [
        html.Div([html.P(id="paragraph_id", children=["Button not clicked"])]),
        html.Button(id="button_id", children="Run Job!"),
        html.Button(id="cancel_button_id", children="Cancel Running Job!"),
    ]
)


@app.long_callback(
    output=(Output("paragraph_id", "children"), Output("button_id", "n_clicks")),
    inputs=Input("button_id", "n_clicks"),
    running=[
        (Output("button_id", "disabled"), True, False),
        (Output("cancel_button_id", "disabled"), False, True),
    ],
    cancel=[Input("cancel_button_id", "n_clicks")],
)
def callback(n_clicks):
    time.sleep(2.0)
    return [f"Clicked {n_clicks} times"], (n_clicks or 0) % 4


if __name__ == "__main__":
    app.run_server(debug=True)

Here you can see that it takes a few seconds to run the callback function, but the cached results are used after n_clicks cycles back around to 0. By interacting with the app in a separate tab, you can see that the cache results are shared across user sessions.

cache_by function workflows

Various cache_by functions can be used to accomplish a variety of caching policies. Here are a few examples:

• A cache_by function could return the file modification time of a dataset to automatically invalidate the cache when an input dataset changes.
• In a Heroku or Dash Enterprise deployment setting, a cache_by function could return the git hash of the app, making it possible to persist the cache across redeploys, but invalidate it when the app's source changes.
• In a Dash Enterprise setting, the cache_by function could return user meta-data to prevent cached values from being shared across authenticated users.

Celery configuration

Here is an example of configuring the CeleryLongCallbackManager for use in @long_callback. This callback manager uses Celery as the execution backend rather than a background process with diskcache.

from dash.long_callback import CeleryLongCallbackManager
from celery import Celery

celery_app = Celery(
    __name__, broker="redis://localhost:6379/0", backend="redis://localhost:6379/1"
)
long_callback_manager = CeleryLongCallbackManager(celery_app)

See the Celery documentation for more information on configuring a Celery app instance.