Implements functions to support other teams usage of the Entity Unique Identifier EUID.

These are all naive implementations doing sequential checks. The current implementation allows teams to start using already. We might still tweak their logic to have less checks (e.g. have only 1 host.name check instead of 2 in the host definition).

import { euid } from '@kbn/entity-store/common';

euid.getIdFromObject('host', doc);
euid.getEuidPainlessEvaluation('user');

Note 1: If for some reason those functions are needed in the front end, we might need to move to commons, but that will also require refactoring definitions to be under common. Possible, but let's do only if it's necessary. Done!

Note 2: We need to know the entity type to know what euid logic to follow. I'm not sure if there are cases where we don't know the entity type and we just want to find any id (the same document can return all ids). Then this logic will need to be handled on the consumer side (imo).

Added Functions

DSL getEuidDslFilterBasedOnDocument(entityType: EntityType, doc: any)

Based the entity id and on any document you have in memory (it can't be flattened, if that's a use case, we need to implement it) it returns a query DSL to be filtered on elasticsearch. For a full list of examples, see x-pack/solutions/security/plugins/entity_store/server/domain/euid/dsl.test.ts.

const result = getEuidDslFilterBasedOnDocument('host', {
  host: { name: 'to-be-ignored', entity: { id: 'host-entity-1' } },
});
  
expect(result).toEqual({
  bool: {
    filter: [{ term: { 'host.entity.id': 'host-entity-1' } }],
  },
});

const result = getEuidDslFilterBasedOnDocument('user', { user: { id: 'user-id-42' } });

expect(result).toEqual({
  bool: {
    filter: [{ term: { 'user.id': 'user-id-42' } }],
    must_not: [
      { exists: { field: 'user.entity.id' } },
      { exists: { field: 'user.name' } },
      { exists: { field: 'host.entity.id' } },
      { exists: { field: 'host.id' } },
      { exists: { field: 'host.name' } },
    ],
  },
});

ESQL getEuidEsqlFilterBasedOnDocument(entityType: EntityType, doc: any)

ESQL counterpart of the DSL filter. Same concept of not being available for flattened objects apply.

const result = getEuidEsqlFilterBasedOnDocument('host', {
  host: { name: 'to-be-ignored', entity: { id: 'host-entity-1' } },
});

expect(result).toBe('((host.entity.id == "host-entity-1"))');

const result = getEuidEsqlFilterBasedOnDocument('user', { user: { id: 'user-id-42' } });

expect(result).toBe(
        '((user.id == "user-id-42") AND (user.entity.id IS NULL OR user.entity.id == "") AND (user.name IS NULL OR user.name == "") AND (host.entity.id IS NULL OR host.entity.id == "") AND (host.id IS NULL OR host.id == "") AND (host.name IS NULL OR host.name == ""))'
      );

ESQL getEuidEsqlDocumentsContainsIdFilter(entityType: entityType)

Generates ESQL filter to make sure that we only have EUID Eligible documents in the context of the query. This makes for aids better performance before generating ids ESQL for every document available in the query. It's used in the main Entity Store Query.

Example of host

(host.entity.id IS NOT NULL AND host.entity.id != "") OR (host.id IS NOT NULL AND host.id != "") OR (host.name IS NOT NULL AND host.name != "") OR (host.hostname IS NOT NULL AND host.hostname != "")

ESQL getEuidEsqlEvaluation(entityType: EntityType)

Generates a ESQL EVAL expression which calculates the euid. It doesn't assign to any field so in your code you need to do it. E.g.

const query = `
  FROM logs-*
  | EVAL my_desired_id ${getEuidEsqlEvaluation('host')}
`;

It's used in the main Entity Store Query.

const result = getEuidEsqlEvaluation('host');

const expected = `CONCAT("host:", CASE((host.entity.id IS NOT NULL AND host.entity.id != ""), host.entity.id,
                    (host.id IS NOT NULL AND host.id != ""), host.id,
                    (host.name IS NOT NULL AND host.name != "" AND host.domain IS NOT NULL AND host.domain != ""), CONCAT(host.name, ".", host.domain),
                    (host.hostname IS NOT NULL AND host.hostname != "" AND host.domain IS NOT NULL AND host.domain != ""), CONCAT(host.hostname, ".", host.domain),
                    (host.name IS NOT NULL AND host.name != ""), host.name,
                    (host.hostname IS NOT NULL AND host.hostname != ""), host.hostname, NULL))`;
expect(normalize(result)).toBe(normalize(expected));

Typescript getEuidFromObject(entityType: EntityType, doc: any)

Based the entity id and on any document you have in memory (it can't be flattened, if that's a use case, we need to implement it) it returns an EUID. E.g.

expect(getEuidFromObject('host', { host: { name: 'myserver', domain: 'example.com' } })).toBe(
  'host:myserver.example.com'
);

Painless getEuidPainlessEvaluation(entityType: EntityType)

Generates a painless script for the EUID generation. It's based on the existence of Map doc and it returns null. It's hard to assume what will be available in the painless context or if we must return a value or emit a value, depending on where you use this script (runtime, aggregation, transform). What is nice is that a Map will be available in some shape and form.

For example, to use in a runtime field (tested with scout), you can wrap the generation around a function and emit the value:

function toRuntimeFieldEmitScript(entity: EntityType): string {
  return `String euid_eval(def doc) { ${getEuidPainlessEvaluation(entityType)} } 
    def result = euid_eval(doc); 
    if (result != null) {
       emit(result);
    }`;
}

Now looking at it, maybe we could be the ones providing it these context painless specific scripts. But the beauty of the current implementation is that users could use how they want, where they want, without depending on us to implement something new.

if (
  doc.containsKey("host.entity.id") &&
  doc["host.entity.id"].size() > 0 &&
  doc["host.entity.id"].value != null &&
  doc["host.entity.id"].value != ""
) {
  return "host:" + doc["host.entity.id"].value;
}
if (
  doc.containsKey("host.id") &&
  doc["host.id"].size() > 0 &&
  doc["host.id"].value != null &&
  doc["host.id"].value != ""
) {
  return "host:" + doc["host.id"].value;
}
if (
  doc.containsKey("host.name") &&
  doc["host.name"].size() > 0 &&
  doc["host.name"].value != null &&
  doc["host.name"].value != "" &&
  doc.containsKey("host.domain") &&
  doc["host.domain"].size() > 0 &&
  doc["host.domain"].value != null &&
  doc["host.domain"].value != ""
) {
  return "host:" + doc["host.name"].value + "." + doc["host.domain"].value;
}
if (
  doc.containsKey("host.hostname") &&
  doc["host.hostname"].size() > 0 &&
  doc["host.hostname"].value != null &&
  doc["host.hostname"].value != "" &&
  doc.containsKey("host.domain") &&
  doc["host.domain"].size() > 0 &&
  doc["host.domain"].value != null &&
  doc["host.domain"].value != ""
) {
  return "host:" + doc["host.hostname"].value + "." + doc["host.domain"].value;
}
if (
  doc.containsKey("host.name") &&
  doc["host.name"].size() > 0 &&
  doc["host.name"].value != null &&
  doc["host.name"].value != ""
) {
  return "host:" + doc["host.name"].value;
}
if (
  doc.containsKey("host.hostname") &&
  doc["host.hostname"].size() > 0 &&
  doc["host.hostname"].value != null &&
  doc["host.hostname"].value != ""
) {
  return "host:" + doc["host.hostname"].value;
}
return null;