Skip to content
This repository was archived by the owner on Mar 9, 2026. It is now read-only.

docs for new MINID trim strategy #1488

Merged
oranagra merged 3 commits into from
Jan 8, 2021
Merged

docs for new MINID trim strategy #1488

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
8e207d1
docs for new MINID trim strategy
guybe7 Jan 6, 2021
35dce7d
Edits
itamarhaber Jan 6, 2021
921d5a3
spelling
oranagra Jan 7, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
82 changes: 48 additions & 34 deletions commands.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4671,43 +4671,49 @@
},
"XADD": {
"summary": "Appends a new entry to a stream",
"complexity": "O(1)",
"complexity": "O(1) when adding a new entry, O(N) when trimming where N being the number of entires evicted.",
"arguments": [
{
"name": "key",
"type": "key"
},
{
"name": "maxlen",
"command": "NOMKSTREAM",
"optional": true
},
{
"name": "trim",
"type": "block",
"optional": true,
"block": [
{
"command": "MAXLEN"
"name": "strategy",
"type": "enum",
"enum": [
"MAXLEN",
"MINID"
]
},
{
"name": "operator",
"type": "block",
"block": [
{
"type": "enum",
"enum": [
"=",
"~"
],
"optional": true
},
{
"name": "length",
"type": "integer"
}
]
"type": "enum",
"enum": [
"=",
"~"
],
"optional": true
},
{
"name": "threshold",
"type": "string"
},
{
"command": "LIMIT",
"name": "count",
"type": "integer",
"optional": true
}
],
"optional": true
},
{
"command": "NOMKSTREAM",
"optional": true
]
},
{
"type": "enum",
Expand Down Expand Up @@ -4740,17 +4746,19 @@
"type": "key"
},
{
"name": "strategy",
"type": "enum",
"enum": [
"MAXLEN"
]
},
{
"name": "trim",
"type": "block",
"name": "operator",
"block": [
{
"name": "strategy",
"type": "enum",
"enum": [
"MAXLEN",
"MINID"
]
},
{
"name": "operator",
"type": "enum",
"enum": [
"=",
Expand All @@ -4759,8 +4767,14 @@
"optional": true
},
{
"name": "length",
"type": "integer"
"name": "threshold",
"type": "string"
},
{
"command": "LIMIT",
"name": "count",
"type": "integer",
"optional": true
}
]
}
Expand Down
27 changes: 8 additions & 19 deletions commands/xadd.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,26 +48,15 @@ IDs to match the one of this other system.

## Capped streams

It is possible to limit the size of the stream to a maximum number of
elements using the **MAXLEN** option. By default, or when used with the `=`
argument, the **MAXLEN** option performs an exact trimming. That means that the
trimmed stream's length will be exactly the minimum between its original length
and the specified maximum length.

Trimming with **MAXLEN** can be expensive compared to just adding entries with
`XADD`: streams are represented by macro nodes into a radix tree, in order to
be very memory efficient. Altering the single macro node, consisting of a few
tens of elements, is not optimal. So it is possible to give the command in the
following special form:
`XADD` incorporates the same semantics as the `XTRIM` command - refer to its documentation page for more information.
This allows adding new entries and keeping the stream's size in check with a single call to `XADD`, effectively capping the stream with an arbitrary threshold.
Although exact trimming is possible and is the default, due to the internal representation of steams it is more efficient to add an entry and trim stream with `XADD` using **almost exact** trimming (the `~` argument).

XADD mystream MAXLEN ~ 1000 * ... entry fields here ...
For example, calling `XADD` in the following form:

The `~` argument between the **MAXLEN** option and the actual count means that
the user is not really requesting that the stream length is exactly 1000 items,
but instead it could be a few tens of entries more, but never less than 1000
items. When this option modifier is used, the trimming is performed only when
Redis is able to remove a whole macro node. This makes it much more efficient,
and it is usually what you want.
XADD mystream MAXLEN ~ 1000 * ... entry fields here ...

Will add a new entry but will also evict old entries so that the stream will contain only 1000 entries, or at most a few tens more.

## Additional information about streams

Expand All @@ -87,7 +76,7 @@ key doesn't exist.

@history

* `>= 6.2`: Added the `NOMKSTREAM` option.
* `>= 6.2`: Added the `NOMKSTREAM` option, `MINID` trimming strategy and the `LIMIT` option.

@examples

Expand Down
59 changes: 39 additions & 20 deletions commands/xtrim.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,39 +1,58 @@
`XTRIM` trims the stream to a given number of items, evicting older items
(items with lower IDs) if needed. The command is conceived to accept multiple
trimming strategies, however currently only a single one is implemented,
which is `MAXLEN`, and works exactly as the `MAXLEN` option in `XADD`.
`XTRIM` trims the stream by evicting older entries (entries with lower IDs) if needed.

For example the following command will trim the stream to exactly
the latest 1000 items:
Trimming the stream can be done using one of these strategies:

* `MAXLEN`: Evicts entries as long as the stream's length exceeds the specified `threshold`, where `threshold` is a positive integer.
* `MINID`: Evicts entries with IDs lower than `threshold`, where `threshold` is a stream ID.

For example, this will trim the stream to exactly the latest 1000 items:

```
XTRIM mystream MAXLEN 1000
```

By default, or when provided with the optional `=` argument, the command
performs exact trimming. That means that the trimmed stream's length will be
exactly the minimum between its original length and the specified maximum
length.
Whereas in this example, all entries that have an ID lower than 649085820-0 will be evicted:

```
XTRIM mystream MINID 649085820
```

By default, or when provided with the optional `=` argument, the command performs exact trimming.

Depending on the strategy, exact trimming means:

It is possible to give the command in the following special form in
order to make it more efficient:
* `MAXLEN`: the trimmed stream's length will be exactly the minimum between its original length and the specified `threshold`.
* `MINID`: the oldest ID in the stream will be exactly the minimum between its original oldest ID and the specified `threshold`.

Nearly exact trimming
---

Because exact trimming may require additional effort from the Redis server, the optional `~` argument can be provided to make it more efficient.

For example:

```
XTRIM mystream MAXLEN ~ 1000
```

The `~` argument between the **MAXLEN** option and the actual count means that
the user is not really requesting that the stream length is exactly 1000 items,
but instead it could be a few tens of entries more, but never less than 1000
items. When this option modifier is used, the trimming is performed only when
Redis is able to remove a whole macro node. This makes it much more efficient,
and it is usually what you want.
The `~` argument between the `MAXLEN` strategy and the `threshold` means that the user is requesting to trim the stream so its length is **at least** the `threshold`, but possibly slightly more.
In this case, Redis will stop trimming early when performance can be gained (for example, when a whole macro node in the data structure can't be removed).
This makes trimming much more efficient, and it is usually what you want, although after trimming, the stream may have few tens of additional entries over the `threshold`.

Another way to control the amount of work done by the command when using the `~`, is the `LIMIT` clause.
When used, it specifies the maximal `count` of entries that will be evicted.
When `LIMIT` and `count` aren't specified, the default value of 100 * the number of entries in a macro node will be implicitly used as the `count`.
Specifying the value 0 as `count` disables the limiting mechanism entirely.

@return

@integer-reply, specifically:
@integer-reply: The number of entries deleted from the stream.

@history

* `>= 6.2`: Added the `MINID` trimming strategy and the `LIMIT` option.

The command returns the number of entries deleted from the stream.
@examples

```cli
XADD mystream * field1 A field2 B field3 C field4 D
Expand Down
4 changes: 2 additions & 2 deletions topics/streams-intro.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -655,9 +655,9 @@ Or, as for the **XADD** option:
> XTRIM mystream MAXLEN ~ 10
```

However, **XTRIM** is designed to accept different trimming strategies, even if only **MAXLEN** is currently implemented.
However, **XTRIM** is designed to accept different trimming strategies. Another trimming strategy is **MINID**, that evicts entries with IDs lower than the one specified.

As **XTRIM** is an explicit command, the user is expected to know about the possible shortcomings of different trimming strategies. As such, it's possible that trimming by time will be implemented at a later time.
As **XTRIM** is an explicit command, the user is expected to know about the possible shortcomings of different trimming strategies.

Another useful eviction strategy that may be added to **XTRIM** in the future, is to remove by a range of IDs to ease use of **XRANGE** and **XTRIM** to move data from Redis to other storage systems if needed.

Expand Down