Redis Function Libraries

This PR implements Redis Functions Libraries as describe on: #9906.

Libraries purpose is to provide a better code sharing between functions by allowing to create multiple functions in a single command. Functions that were created together can safely share code between each other without worrying about compatibility issues and versioning.

Creating a new library is done using 'FUNCTION LOAD' command (full API is described below)

This PR introduces a new struct called libraryInfo, libraryInfo holds information about a library:

• name - name of the library
• engine - engine used to create the library
• code - library code
• description - library description
• functions - the functions exposed by the library

When Redis gets the FUNCTION LOAD command it creates a new empty libraryInfo. Redis passes the CODE to the relevant engine alongside the empty libraryInfo. As a result, the engine will create one or more functions by calling 'libraryCreateFunction'. The new funcion will be added to the newly created libraryInfo. So far Everything is happening locally on the libraryInfo so it is easy to abort the operation (in case of an error) by simply freeing the libraryInfo. After the library info is fully constructed we start the joining phase by which we will join the new library to the other libraries currently exist on Redis. The joining phase make sure there is no function collision and add the library to the librariesCtx (renamed from functionCtx). LibrariesCtx is used all around the code in the exact same way as functionCtx was used (with respect to RDB loading, replicatio, ...). The only difference is that apart from function dictionary (maps function name to functionInfo object), the librariesCtx contains also a libraries dictionary that maps library name to libraryInfo object.

New API

FUNCTION LOAD

FUNCTION LOAD <ENGINE> <LIBRARY NAME> [REPLACE] [DESCRIPTION <DESCRIPTION>] <CODE>
Create a new library with the given parameters:

• ENGINE - REPLACE Engine name to use to create the library.
• LIBRARY NAME - The new library name.
• REPLACE - If the library already exists, replace it.
• DESCRIPTION - Library description.
• CODE - Library code.

Return "OK" on success, or error on the following cases:

• Library name already taken and REPLACE was not used
• Name collision with another existing library (even if replace was uses)
• Library registration failed by the engine (usually compilation error)

Changed API

FUNCTION LIST

FUNCTION LIST [LIBRARYNAME <LIBRARY NAME PATTERN>] [WITHCODE]
Command was modified to also allow getting libraries code (so FUNCTION INFO command is no longer needed and removed). In addition the command gets an option argument, LIBRARYNAME allows you to only get libraries that match the given LIBRARYNAME pattern. By default, it returns all libraries.

INFO MEMORY

Added number of libraries to INFO MEMORY

Commands flags

DENYOOM flag was set on FUNCTION LOAD and FUNCTION RESTORE. We consider those commands as commands that add new data to the dateset (functions are data) and so we want to disallows to run those commands on OOM.

Removed API

• FUNCTION CREATE - Decided on Redis Functions - Libraries #9906
• FUNCTION INFO - Decided on Functions on Redis Cluster #9899

Lua engine changes

When the Lua engine gets the code given on FUNCTION LOAD command, it immediately runs it, we call this run the loading run. Loading run is not a usual script run, it is not possible to invoke any Redis command from within the load run. Instead there is a new API provided by library object. The new API's:

• redis.log - behave the same as redis.log
• redis.register_function - register a new function to the library

The loading run purpose is to register functions using the new redis.register_function API. Any attempt to use any other API will result in an error. In addition, the load run is has a time limit of 500ms, error is raise on timeout and the entire operation is aborted.

redis.register_function

redis.register_function(<function_name>, <callback>, [<description>])
This new API allows users to register a new function that will be linked to the newly created library. This API can only be called during the load run (see definition above). Any attempt to use it outside of the load run will result in an error. The parameters pass to the API are:

• function_name - Function name (must be a Lua string)
• callback - Lua function object that will be called when the function is invokes using fcall/fcall_ro
• description - Function description, optional (must be a Lua string).

Example

The following example creates a library called lib with 2 functions, f1 and f1, returns 1 and 2 respectively:

local function f1(keys, args)
    return 1
end

local function f2(keys, args)
    return 2
end

redis.register_function('f1', f1)
redis.register_function('f2', f2)

Notice: Unlike eval, functions inside a library get the KEYS and ARGV as arguments to the functions and not as global.

Technical Details

On the load run we only want the user to be able to call a white list on API's. This way, in the future, if new API's will be added, the new API's will not be available to the load run unless specifically added to this white list. We put the while list on the library object and make sure the library object is only available to the load run by using lua_setfenv API. This API allows us to set the globals of a function (and all the function it creates). Before starting the load run we create a new fresh Lua table (call it g) that only contains the library API (we make sure to set global protection on this table just like the general global protection already exists today), then we use lua_setfenv to set g as the global table of the load run. After the load run finished we update g metatable and set __index and __newindex functions to be _G (Lua default globals), we also pop out the library object as we do not need it anymore. This way, any function that was created on the load run (and will be invoke using fcall) will see the default globals as it expected to see them and will not have the library API anymore.

An important outcome of this new approach is that now we can achieve a distinct global table for each library (it is not yet like that but it is very easy to achieve it now). In the future we can decide to remove global protection because global on different libraries will not collide or we can chose to give different API to different libraries base on some configuration or input.

Notice that this technique was meant to prevent errors and was not meant to prevent malicious user from exploit it. For example, the load run can still save the library object on some local variable and then using in fcall context. To prevent such a malicious use, the C code also make sure it is running in the right context and if not raise an error.

• Implement FUNCTION LIST pattern matching
• Count libraries memory overhead
• Functions flags, decide if we want it and which flags we want - delayed to followup PR and will be discussed on Redis Functions Flags #10025