Phase 2 from #27004.

This is currently blocked on #59960 and #63358

Note to self: add UI docs for Spaces (sharing).

1. Overview

The “Sharing saved objects” feature (#27004) changes how saved objects are stored in Elasticsearch. Particularly, objects that exist in different spaces will no longer be able to have the same ID; each object of a given type must have a globally unique ID.

In Phase 2, we will add support for converting existing "single-namespace" saved object types into new "multi-namespace" types. This entails two core pieces of work:

• Migrations: Existing Kibana migration logic only applies consumer-defined "migration transforms" to each object when Kibana is upgraded. This will be augmented to apply platform-defined "conversion transforms" and "reference transforms" to change existing object fields (id, references), add new object fields (namespaces, originId), and create new Alias objects.
• Aliases: When we change object IDs, we don’t want to break URLs and integrations that may rely on thes. For this reason, we intend to create an Alias for each object that is converted, so it can still be accessed in a particular space using its old ID.

Note: the originId field is used during Copy and Transform operations, but it is not a substitute for Aliases.

2. Migrations

2.1. Usage

When registering an existing object type, consumers will need to change the namespaceType field and add a new convertToMultiNamespaceTypeVersion field. This new field will inform the Kibana platform that objects of this type need to be converted when Kibana is upgraded to the specified version.

core.savedObjects.registerType({
  name: 'doodad',
  hidden: false,
  namespaceType: 'single',
  mappings: { ... },
  migrations: { ... },
});

core.savedObjects.registerType({
  name: 'doodad',
  hidden: false,
  namespaceType: 'multiple',
  mappings: { ... },
  migrations: { ... },
  convertToMultiNamespaceTypeVersion: '8.0.0',
});

2.2. Technical Details

The DocumentMigrator currently applies consumer-defined "migration transforms" to each object with an outdated migrationVersion field at two times: 1. When Kibana is upgraded, and 2. When creating a new object. During upgrade only, we will apply two new types of transforms:

"Conversion transforms"

During the upgrade process, for each object that has an outdated migrationVersion field, if its type has a convertToMultiNamespaceTypeVersion defined, we apply a "conversion transform" to it.

The conversion process follows this algorithm:

1. Remove the namespace field and add the corresponding namespaces field.
2. If the object does not exist in the default namespace:
   1. Deterministically regenerate the id field (using uuidv5)
   2. Add the originId field, equivalent to the old ID
   3. Create a Legacy URL Alias for this object using the old ID and the new ID

"Reference transforms"

The SavedObject type will be updated to define a new coreMigrationVersion field. Whenever an object is created or Kibana is upgraded, this value is set to match the current Kibana version.

During the upgrade process, for each object that has an outdated coreMigrationVersion field, its references are enumerated, and we apply one or more "reference transforms" to it. These transforms are defined by other object types.

The reference transform is only applied to objects that do not exist in the default namespace. It deterministically regenerates the reference ID for the affected types.

Examples

Kibana 8.0 defines three object types:

{
  name: 'foo',
  namespaceType: 'multiple',
  convertToMultiNamespaceTypeVersion: '8.0.0',
},
{
  name: 'bar',
  namespaceType: 'multiple',
  migrations: {
    '8.0.0': (doc) => { ... },
  },
  convertToMultiNamespaceTypeVersion: '8.0.0',
},
{
  name: 'baz',
  namespaceType: 'single',
  migrations: {
    '7.9.0': (doc) => { ... },
    '7.10.0': (doc) => { ... },
    '8.0.0': (doc) => { ... },
  },
}

{
  type: 'bar',
  id: '111',
  migrationVersion: {},
  references: [{ type: 'foo', id: '222' }],
  namespace: 'somespace',
}

{
  type: 'bar',
  id: deterministicNewId('somespace:bar:111'),
  migrationVersion: { 'bar': '8.0.0' },
  references: [{ type: 'foo', id: deterministicNewId('somespace:foo:222') }],
  referencesMigrationVersion: '8.0.0',
  namespaces: ['somespace'],
  originId: '111',
}

{
  type: 'baz',
  id: '333',
  migrationVersion: { 'baz': '7.9.0' },
  references: [{ type: 'foo', id: '222' }, { type: 'bar', id: '111' }],
  namespace: 'somespace',
}

{
  type: 'baz',
  id: '333',
  migrationVersion: { 'baz': '8.0.0' },
  references: [
    { type: 'foo', id: deterministicNewId('somespace:foo:222') },
    { type: 'bar', id: deterministicNewId('somespace:bar:111') }
  ],
  referenceMigrationVersion: '8.0.0'
  namespace: 'somespace',
}

3. Aliases

3.1. Usage

A new API will be exposed to retrieve objects use aliases. Instead of SavedObjectsClient.get, consumers should use SavedObjectsClient.resolve. For example, if a consumer calls client.resolve('bar', '111'), it will attempt to find an object with the ID or a matching alias with that ID.

If an object and alias exist for the same namespace+type+id combination, a new "alias conflict" error will occur. That must be resolved by the end user -- either the object must be deleted, or the alias must be disabled/deleted. Some diagrams that describe this scenario are below.

Since "alias conflict" errors can occur, at a minimum users need to be able to directly read and delete object aliases. A user interface will also be needed to facilitate this. We can add two pages:

• Stack management / Saved objects / Aliases: New tab in the existing "Saved objects" page. Renders a table that lists all aliases with the following information: type, ID, target title, last resolved, and error.
  • The ID is a link that takes you to another page for that particular alias.
  • The target title is a link that takes you to another page for that particular object (defined by getTitle and getInAppUrl when that object type is registered).
  • The last resolved column displays when this alias was last resolved (when the target object was last accessed through this alias).
  • The error column displays any "not found" or "alias conflict" error for this alias.
• Stack management / Saved objects / Aliases / <Alias ID>: Renders a page that shows the same information for that row in the aforementioned table. This will be needed so we can deep link to this page from other Kibana apps when an "alias conflict" error is encountered.

3.2. Technical Details

How aliases will be stored

Aliases will be stored as separate saved objects.

{
  type: 'dashboard',
  id: 'cf3b7012-4abf-4d17-9874-8b55bf7fc844',
  attributes: { ... },
  namespaces: ['somespace']
}
{
  type: 'alias',
  id: 'somespace:dashboard:old-dashboard-id',
  attributes: {
    targetType: 'dashboard',
    targetId: 'cf3b7012-4abf-4d17-9874-8b55bf7fc844',
    lastResolved: '2020-10-22T20:09:50.587Z'
  },
  namespaces: ['somespace']
}

We also considered storing aliases in an array inside of each existing object, but decided on this approach instead. Benefits of this approach:

• Less effort to implement
• Better performance to retrieve a specific alias by its ID (using get instead of search)
• No changes to “root fields” for saved objects
• Alias raw ID ensures that there will never be multiple aliases with the same target namespace+type+ID
• If we ever want to resolve aliases in bulk operations, this will be much easier to work with

How aliases will be resolved

The action of "resolving" an alias attempts to find the specified object (namespace + type + ID). There are four possible outcomes when resolving an alias:

Aliases will only be resolved by using the new SavedObjectsClient.resolve API.

We also considered automatically resolving aliases in all SOC APIs, but decided on this approach instead. Benefits of this approach:

• Much less effort to implement
• Less effort for consumers that will not be interested in resolving aliases
• If an “alias conflict” exists, it has a less severe impact on the user experience

How users can interact with aliases

TBD Decision: end users can only read and delete aliases.

Time permitting: users will also be able to disable aliases. This may come in handy if they're not sure whether the alias is being used or not.