So, I decided to make knowledge base entries (first-class) citizens of the kibana spec instead of having them live in their own, dedicated root folder.

The reasoning is that:

1. KB entries will be "managed" by Kibana
2. KB entries will only be used via Kibana (at least for now)

So I felt like it made more sense to have it done that way.

I was thinking that this was just data, and it could maybe be reused for other features, but your reasoning makes sense. If it is managed and used only by Kibana, let's keep it under kibana.

at least for now

Do you foresee other uses of this data?

I was thinking that this was just data, and it could maybe be reused for other features

Yeah I was initially thinking that too, but the KB entry installation will do more than just creating the index and indexing the docs - it will also create an entry in the knowledge base registry to register the source, so I felt like it would be potentially complicated to convert that to some abstract generic concept...

Do you foresee other uses of this data?

We might want at some point to expose this data more directly to the end user, and make is less of an implementation detail. But we don't have any use case for this right now, so I think we should just ignore it for now and eventually revisit later.

I decided to KISS and keep the index's mapping as a plain json file, first because I don't think adding validation on the index's mapping format or having it as a yaml file would bring much value (AFAIK we're not doing it for index templates, mappings are just defined as object type with allowed extra props), and I also did not use the fields.yml feature because we don't need, or even want, any kind of discoverability on this type of content.

Please tell me if you think this doesn't make sense.

I understand that having to convert to the fields.yml format can be some additional effort, but it has some advantages.

For example, Fleet already has the code to generate mappings from this format, and over that it can more easily add opinionated settings or mappings, for example settings that could be defined by users, or fields known to be everywhere.

Also, elastic-package has code to perform validation of documents based on the mappings defined in the package. This could be leveraged for example to check that the provided content comply with the defined mappings.

As you mention, in principle these features are not wanted at least by now. But just in case and if it is only for consistency with other features, I would prefer to use the fields.yml format.

So another part of the reasoning was that each knowledge base entry can, technically, have different mappings. I'm not sure how that would play with that fields.yml feature, given that from my understanding, this is only a top level section in the package?

Or should we use the same format but with a different pattern, like having one fields.yml file per knowledge base entry folder (that would then replace the index-mappings.json file)?

We reuse the fields definitions on different places. The more direct way to add it here would be to have a fields directory here, something like this:

        - description: Folder containing field definitions to be used as the mappings for the index template
          type: folder
          name: fields
          required: false
          $ref: "../../data_stream/fields/spec.yml"

As suggested by @jsoriano, we're allowing an arbitrary number of file following the fixed content.json and
dynamic content.{num}.json naming format. Each file have a single documents property containing the source of the documents we will be indexing into the KB's index.

I did not add any validation on the file's content format, both because I wasn't sure we can add validation on JSON documents, and also because we wouldn't have much to validation (only thing we could do it make sure the only top level property present is documents and that it's an array).

But if there is a way to perform validation on JSON docs with the spec validator, I can absolutely add it.

As suggested by @jsoriano, we're allowing an arbitrary number of file following the fixed content.json and
dynamic content.{num}.json naming format. Each file have a single documents property containing the source of the documents we will be indexing into the KB's index.

👍

In other cases where we allow multiple files we allow to use letters too, so it can give meaningful names when this is used for organizational purposes, and we use to separate with hyphens.

I did not add any validation on the file's content format, both because I wasn't sure we can add validation on JSON documents, and also because we wouldn't have much to validation (only thing we could do it make sure the only top level property present is documents and that it's an array).

But if there is a way to perform validation on JSON docs with the spec validator, I can absolutely add it.

In principle we can add validation to JSON documents if we have a schema, but not sure if it is worth for this case.

The spec checks that the JSON is valid for files with contentMediaType: "application/json".

In other cases where we allow multiple files we allow to use letters too, so it can give meaningful names when this is used for organizational purposes, and we use to separate with hyphens.

Sounds good, I will follow the pattern for consistency. I'll just wait for the other question about the format (json vs ndjson) to be addressed to perform both changes at the same time if required.

Example file of the format specified in spec/integration/kibana/kb_entry/manifest.spec.yml.

Most of this manifest is meta that will be used by the KB implementation:

• name and description are meta that will be surfaced in the KB management page. name will be used at the identifier.

• retrieval.syntactic_fields and retrieval.semantic_fields are the list of fields that will be used for retrieving documents from this KB entry. Later we could find way to be smarter and more automated about field detection, but I wanted to start with something both easy and concrete, so the KB entry contains those meta.

• index.name is self explanatory - the name of the index that will be created with the specified mappings from the index-mapping.json file

• index.system is just some future-proofing by exposing the info about the index being Kibana-managed or not (probably don't need more details on this specific PR)

• retrieval.syntactic_fields and retrieval.semantic_fields are the list of fields that will be used for retrieving documents from this KB entry. Later we could find way to be smarter and more automated about field detection, but I wanted to start with something both easy and concrete, so the KB entry contains those meta.

How are these going to be different between different knowledge bases?

• index.name is self explanatory - the name of the index that will be created with the specified mappings from the index-mapping.json file

The index name, or anything that could conflict with other resources, should be managed by Fleet. It could be built from the package and entry names.

How are these going to be different between different knowledge bases?

By having those defined in the manifest, and then registered during installation. The search/retrieval code will then forge its request to target the specified fields.

The index name, or anything that could conflict with other resources, should be managed by Fleet

I guess that makes sense

I was thinking that this was just data, and it could maybe be reused for other features, but your reasoning makes sense. If it is managed and used only by Kibana, let's keep it under kibana.

at least for now

Do you foresee other uses of this data?

No need to abbreviate, kb may be actually confusing, it could be "kibana" around this context. Would it make sense to call this directly "knowledge_base"?

Or I would also prefer "knowledge_base_entry" rather than its abbreviated form.

That's totally fair. knowledge_base_entry sounds good, I will adapt accordingly

I understand that having to convert to the fields.yml format can be some additional effort, but it has some advantages.

For example, Fleet already has the code to generate mappings from this format, and over that it can more easily add opinionated settings or mappings, for example settings that could be defined by users, or fields known to be everywhere.

Also, elastic-package has code to perform validation of documents based on the mappings defined in the package. This could be leveraged for example to check that the provided content comply with the defined mappings.

As you mention, in principle these features are not wanted at least by now. But just in case and if it is only for consistency with other features, I would prefer to use the fields.yml format.

As suggested by @jsoriano, we're allowing an arbitrary number of file following the fixed content.json and
dynamic content.{num}.json naming format. Each file have a single documents property containing the source of the documents we will be indexing into the KB's index.

👍

In other cases where we allow multiple files we allow to use letters too, so it can give meaningful names when this is used for organizational purposes, and we use to separate with hyphens.

I did not add any validation on the file's content format, both because I wasn't sure we can add validation on JSON documents, and also because we wouldn't have much to validation (only thing we could do it make sure the only top level property present is documents and that it's an array).

But if there is a way to perform validation on JSON docs with the spec validator, I can absolutely add it.

In principle we can add validation to JSON documents if we have a schema, but not sure if it is worth for this case.

The spec checks that the JSON is valid for files with contentMediaType: "application/json".

I wonder if this file could be just a ndjson file.

Is the documents entry needed? Do we expect other entries in this file?

How are these files generated? Do we expect manual editions on these files?

I'm being (probably overly) cautious with NDJSON, as we've learned the hard way with Kibana import/export that making the format evolve can be a pain (very difficult to add global metadata in addition to the entries). But for Kibana import/export, we're only generating a single file, we don't have a full folder structure like we do there, so the limitations of the format were more impactful.

So no, I don't foresee us needing to add anything to those files, so if you prefer using ndjson, I'm fine with it

Not a strong opinion, but I think I'd prefer using ndjson here, yes. If there is some case were we need to add metadata in the future we can use the manifest file, or some other file.

Is this setting going to be the same in all knowledge bases?

Not necessarily, I mean given the mappings are defined per KB entry, I assume we would be fine letting this be the source of truth for the dynamicallocation too (given it's technically a part of the mapping, not a setting).

But having this "hardcoded" and non-configurable by the package would be fine too ihmo, if we want to switch to the fields.yml approach

But having this "hardcoded" and non-configurable by the package would be fine too ihmo, if we want to switch to the fields.yml approach

Yes, if we use the fields.yml approach this could be a setting managed by Fleet, hardcoded by now to strict. If needed for some future case we could add a setting to the knowledge base manifest to control the value.

• retrieval.syntactic_fields and retrieval.semantic_fields are the list of fields that will be used for retrieving documents from this KB entry. Later we could find way to be smarter and more automated about field detection, but I wanted to start with something both easy and concrete, so the KB entry contains those meta.

How are these going to be different between different knowledge bases?

• index.name is self explanatory - the name of the index that will be created with the specified mappings from the index-mapping.json file

The index name, or anything that could conflict with other resources, should be managed by Fleet. It could be built from the package and entry names.

This should be added only to content packages by now.

Is name needed? Fleet could use the package and knowledge base directory names to build a unique one.

I was planning on autogenerating an "id" based on the package name and KB directory, but the manifest's name is more of a meta data field for a user-readable "name", which will be displayed in the KB registry's UI. Does that make sense?

Ah ok, makes sense. Maybe then we could call it label or title? But as you prefer, name in that context sounds also good.

I think you're right, title is probably more explicit and avoid mixing the concepts. Will update

Please move this file under spec/content by now.

Thinking about the issue with the assumptions in the kibana directory, and if we have to move the knowledge bases out of it, maybe we can revisit the idea of making this part of a more generic feature for data.

It could be done by:

• Moving knowledge base from ./kibana/knowledge_base_entry/foo to ./index_data/foo.
• Adding a type field in the manifest indicating what this data is about.

Thinking on other features that might provide data in the future, we would probably follow the same structure in all cases: the actual content, the field mappings, and some manifest with metadata or additional options. The type would instruct Fleet about different steps done during installation for each kind of data.

👍, reworked in in cdb4098

This is the part I'm the less sure about. Are we fine with all subtype's fields "leaking" at top level, or would we rather have dedicated groups per subtype, e.g.

knowledge_base_entry:
   type: object
   properties: 
       [...]

We can revisit when we add a second type, it LGTM by now.

We can revisit when we add a second type, it LGTM by now.

@mrodm we are adding a new "index_data" directory at the root level, for collections of documents that should be installed on package installation. By now the only kind of collection we support would be "knowledge bases" for AI assistants. Other collections we could add in the future could be sample data for example.

Do you foresee any problem with this approach or this new directory at the root level?

As there is no change in the package manifest, I'd say it would not be needed to add any new field into the index for the Elastic Package Registry. If that is right, it would be ok as it is.

Tested also to add another dummy type under index_data folder. Looks that is possible (see other comment).

Yeah, we would only need a change in the registry if this were going to be used for discovery of packages. Let's see how the support in Kibana progresses.

Tested adding another type of index_data:

##
## Describes the specification for a Kibana knowledge base entry's manifest.yml file
##
spec:
  type: object
  additionalProperties: false
  properties:
    type:
      type: string
      description: >
        Type of the index data.
      enum:
        - knowledge_base_entry
        - foo
    description:
      description: A description of what the index data provides
      type: string

    # Conditional properties.
    title: true
    index: true
    retrieval: true
    some: true
  allOf:
    - if:
        properties:
          type:
            const: knowledge_base_entry
        required:
          - type
      then:
        properties:
          title:
            description: The title of the knowledge base entry as used by the knowledge base
            type: string
          index:
            type: object
            additionalProperties: false
            properties:
              system:
                description: Specify whether the index should be system-managed or not
                type: boolean
            required:
              - system
          retrieval:
            type: object
            additionalProperties: false
            properties:
              syntactic_fields:
                description: List of fields that should be used for syntactic search during retrieval.
                type: array
                items:
                  type: string
              semantic_fields:
                description: List of fields that should be used for semantic search during retrieval.
                type: array
                items:
                  type: string
            required:
              - syntactic_fields
              - semantic_fields
        required:
          - index
          - retrieval
      else:
        not:
          required:
            - some
            - index
            - retrieval
    - if:
        properties:
          type:
            const: foo 
        required:
          - type
      then:
        properties:
          title:
            description: The title of the knowledge base entry as used by the knowledge base
            type: string
          some:
            description: Some variable
            type: string
        required:
          - title
          - some
      else:
        not:
          required:
            - some
            - index
            - retrieval
  required:
    - type

What I checked is that it requires:

• to define as true all the properties at the top where the "Conditional properties" comment is.
• to define as not required all those conditional properties in the else clause.

Thanks for checking!

As there is no change in the package manifest, I'd say it would not be needed to add any new field into the index for the Elastic Package Registry. If that is right, it would be ok as it is.

Tested also to add another dummy type under index_data folder. Looks that is possible (see other comment).