Copyright (C) 2020-2023 The Open Library Foundation

This software is distributed under the terms of the Apache License, Version 2.0. See the file "LICENSE" for more information.

• Introduction
• Compiling
• Running it
• Docker
• Multi-language search support
  • Adding new languages via REST
  • Defining initial languages via ENV variable
• Deploying the module
  • Configuring Elasticsearch
    • Configuring on-premise Elasticsearch instance
    • Recommended production set-up
  • Environment variables
  • Configuring spring-boot using JAVA_OPTIONS
  • Configuring connection to elasticsearch
  • Tenant attributes
• Data Indexing
  • Recreating Elasticsearch index
  • Updating index settings
  • Monitoring reindex process
• Indexing of Instance Records
• API
  • CQL support
    • CQL query operators
    • CQL query modifiers
  • Search API
    • Searching and filtering
      • Matching all records
      • Matching undefined or empty values
      • Search by all field values
      • Instance search options
      • Holdings search options
      • Item search options
      • Authority search options
      • Contributors search options
    • Search Facets
      • Instance facets
      • Holdings facets
      • Item facets
      • Authority facets
      • Contributors facets
    • Sorting results
      • Instance sort options
      • Authority sort options
  • Browse API
  • Resource IDs streaming API
    • Create Job
    • Retrieve ids
• Additional Information
  • Issue tracker
  • API Documentation
  • Code analysis
  • Download and configuration
  • Development tips

This module provides a search functionality for instance and authorities via REST API. It uses The Contextual Query Language as a formal language to query records using filters, boolean conditions, etc.

mod-search works with either Elasticsearch and OpenSearch, however, many variables still have Elasticsearch in their name for backwards compatibility.

mvn install

See that it says "BUILD SUCCESS" near the end.

By default, the integration tests run against an OpenSearch server. To run them against an Elasticsearch server use

SEARCH_ENGINE_DOCKERFILE="docker/elasticsearch8/Dockerfile" mvn install

or run GitHub Action elasticsearch.yml.

The GitHub Action automatically runs whenever a commit is pushed to the master branch.

The recommended way to run the module locally is using Docker Compose. This provides a complete development environment with all dependencies.

# Build the module JAR
mvn clean package -DskipTests

# Start all services (infrastructure + module)
docker compose -f docker/app-docker-compose.yml up -d

# View logs
docker compose -f docker/app-docker-compose.yml logs -f mod-search

For detailed Docker Compose documentation, see docker/README.md.

For local development, you can run the application directly from IntelliJ IDEA with the dev profile. Spring Boot will automatically start the required infrastructure services (PostgreSQL, Kafka, OpenSearch) using Docker Compose.

1. Open the project in IntelliJ IDEA
2. Run the main application class with the dev profile active
3. Spring Boot will automatically start infrastructure containers from docker/infra-docker-compose.yml

Run the module locally on the default listening port (8081) with infrastructure services:

# Start infrastructure services
docker compose -f docker/infra-docker-compose.yml up -d

# Run the module
KAFKA_HOST=localhost KAFKA_PORT=29092 \
DB_HOST=localhost DB_PORT=5432 DB_DATABASE=okapi_modules DB_USERNAME=folio_admin DB_PASSWORD=folio_admin \
ELASTICSEARCH_URL=http://localhost:9200 \
java -Dserver.port=8081 -jar target/mod-search-*.jar

Build the docker container with:

mvn clean package -DskipTests
docker build -t mod-search .

The project includes a comprehensive Docker Compose setup with two main configurations:

1. Infrastructure only (infra-docker-compose.yml) - PostgreSQL, OpenSearch, Kafka, and supporting services
2. Full stack (app-docker-compose.yml) - Infrastructure + mod-search module with scalable instances

# Run infrastructure only (for local development)
docker compose -f docker/infra-docker-compose.yml up -d

# Run full stack with 2 module instances
docker compose -f docker/app-docker-compose.yml up -d

# Scale module instances
docker compose -f docker/app-docker-compose.yml up -d --scale mod-search=3

See docker/README.md for detailed documentation on:

• Configuration options via .env file
• Multiple development workflows
• Service descriptions and ports
• Troubleshooting guide
• Best practices

Each tenant is allowed to pick up to 5 languages from pre-installed list for multi-language indexes (e.g. title, contributors, etc.). This can be done via following API (languageAnalyzer field is optional): POST /search/config/languages

{
  "code": "eng",
  "languageAnalyzer": "english"
}

The code here is an ISO-639-2/B three-letter code. Here is the list of pre-installed languages analyzers:

• ara
• ger
• eng
• spa
• fre
• heb
• ita
• jpn
• kor (default analyzer: seunjeon_analyzer, alternative for k8s and on-premise deployment - nori)
• rus
• swe
• chi

It is allowed to add new languages via rest endpoint /search/config/languages. Please note, when you add a new language, a whole reindex is required in order to apply new configuration.

It is possible to define initial languages via INITIAL_LANGUAGES env variable. These languages will be added on tenant init and applied to index. Example usage: INITIAL_LANGUAGES=eng,fre,kor,chi,spa. If the variable is not defined, only eng code is added.

It is required to install some required plugins for your search engine instance, here is the list:

• analysis-icu
• analysis-kuromoji
• analysis-smartcn
• analysis-nori
• analysis-phonetic

You can find sample Dockerfile in [docker/opensearch/Dockerfile] or install plugins manually:

${ES_HOME}/bin/opensearch-plugin install --batch \
  analysis-icu \
  analysis-kuromoji \
  analysis-smartcn \
  analysis-nori \
  analysis-phonetic

See also Install OpenSearch/Docker Image.

There is an alternative Elasticsearch image from Bitnami - bitnami/elasticsearch, that does not require extending dockerfile but has an env variable ELASTICSEARCH_PLUGINS to specify plugins to install.

For production installations it is strongly recommended enabling security for instance and set-up user/password. The user must have at least following permissions:

• Create/delete/update index and mappings for it.
• Create/update/delete documents in index.

The data nodes OpenSearch configuration completely depends on the data. If there are 7 mln of instances the configuration with 2 nodes with 8Gb RAM and 500 Gb disk (AWS m5.large) works well. The nodes were both master and data node. We performed performance tests for this configuration, and it showed good results. We would recommend to performing additional performance testing (try to reindex and search with different configurations) with different type of nodes, and see what configuration is sufficient for what data volume.

Also, for fault tolerance OpenSearch requires dedicated master nodes (not to have quorum problem which is called split brain) with less powerful configuration ( see High availability and Cross-cluster replication).

The module uses system user to communicate with other modules from Kafka consumers. For production deployments you MUST specify the password for this system user via env variable: SYSTEM_USER_PASSWORD=<password>.

Spring boot properties can be overridden using the specified environment variables, if it is not it can be done using one of the following approaches (see also the documentation Spring Boot Externalized Configuration):

1. Using the environment variable SPRING_APPLICATION_JSON ( example: SPRING_APPLICATION_JSON='{"foo":{"bar":"spam"}}')
2. Using the system variables within the JAVA_OPTIONS ( example: JAVA_OPTIONS=-Xmx400m -Dlogging.level.org.folio.search=debug)

In order to configure connection to OpenSearch or Elasticsearch you have to provide following env variables:

• ELASTICSEARCH_URL - URL to OpenSearch or Elasticsearch master node (e.g. http(s)://elasticsearch:9200);
• ELASTICSEARCH_USERNAME - username of the user to connect to OpenSearch or Elasticsearch;
• ELASTICSEARCH_PASSWORD - password for the user (see OpenSearch Security configuration or Secure the Elastic Stack for details).

It is possible to define specific tenant parameters during module's initialization for particular tenant.

Sometimes we need to recreate OpenSearch or Elasticsearch index, for example when a breaking change introduced to index structure (mapping). It can be fixed by running reindex request:

POST [OKAPI_URL]/search/index/inventory/reindex

x-okapi-tenant: [tenant]
x-okapi-token: [JWT_TOKEN]

{
  "recreateIndex": true,
  "resourceName": "authority",
  "indexSettings": {
    "numberOfShards": 2,
    "numberOfReplicas": 4,
    "refreshInterval": 1
  }
}

• resourceName parameter is required. Possible values: authority, location, linked-data-instance, linked-data-work, linked-data-hub. Please note that location reindex is synchronous. Refer to Indexing of Instance Records section for reindexing of instances
• recreateIndex parameter is optional and equal to false by default. If it is equal to true then mod-search will drop existing indices for tenant and resource, creating them again. Executing request with this parameter equal to true in query will erase all the tenant data in mod-search.
• indexSettings parameter is optional and defines the following Elasticsearch/Opensearch index settings:
  • numberOfShards - the number (between 1 and 100) of primary shards for the index
  • numberOfReplicas - the number of replicas (between 0 and 100) each primary shard has
  • refreshInterval - the refresh interval reflecting how often to make new changes to the index visible to search (seconds). -1 disables refresh.
• Please note that for linked-data-instance, linked-data-work and linked-data-hub resources the endpoint is used only for index recreation purpose and actual reindex operation is triggered through mod-linked-data.

Elasticsearch/Opensearch index settings can be updated using the following request:

PUT [OKAPI_URL]/search/index/settings

x-okapi-tenant: [tenant]
x-okapi-token: [JWT_TOKEN]

{
  "resourceName": "authority",
  "indexSettings": {
    "numberOfReplicas": 4,
    "refreshInterval": 60
  }
}

• resourceName parameter is required. Possible values: instance, authority, location, linked-data-instance, linked-data-work, linked-data-hub.
• indexSettings parameter is optional and defines the following Elasticsearch/Opensearch index settings:
  • numberOfReplicas - the number of replicas (between 0 and 100) each primary shard has
  • refreshInterval - the refresh interval reflecting how often to make new changes to the index visible to search (seconds). -1 disables refresh.

Index settings can also be specified during the running of reindex process which is described in Recreating Elasticsearch index and in Indexing of Instance Records sections.

There is no end-to-end monitoring implemented yet, however it is possible to monitor it partially. In order to check how many records published to Kafka topic use inventory API:

GET [OKAPI_URL]/authority-storage/reindex/[reindex job id]

reindex job id - id returned by /search/index/inventory/reindex endpoint.

In order to estimate total records that actually added to the index, you can send a "match all" search query and check totalRecords, e.g. GET /search/authorities?query=id="*". Alternatively you can query Elasticsearch directly, see ES search API.

The section below describes the data indexing for instance, subject, contribution and classification resource types. Since subject, contribution and classification are derived or child records of an Instance, in short, we are referring to indexing of these and instance as Indexing of Instances (or Instances Indexing) and resources as Instance resources.

This is an improved and recommended version of instances indexing than that which is achieved with POST [OKAPI_URL]/search/index/inventory/reindex and documented in Data Indexing section

The whole indexing procedure consist of two steps:

1. Merge - building the necessary data model and collection of data in accordance with the built model
2. Upload - querying the data and adding them into OpenSearch index

Merge step can be considered as Data Aggregation or Data Reloading. We need to run this step when there is a breaking change introduced to index structure (mapping). Normally, we run Instances indexing with this step rarely, as introducing the breaking changes does not happen that frequently. The step aggregates all instances data available for the tenant into specific data models and keeps them there. For Create/Update/Delete operations performed on specific Instances mod-search will update or sync the aggregated data of those specific Instances, so that the aggregated data is always actual and represents the latest state of an instance resources.

Upload step queries the aggregated data for the specific Instance resources type and then runs the indexing by populating the data into resource's OpenSearch index.

We can execute both Merge and Upload steps with so-called full reindex API:

POST /search/index/instance-records/reindex/full

x-okapi-tenant: [tenant]
x-okapi-token: [JWT_TOKEN]

{
  "indexSettings": {
    "numberOfShards": 2,
    "numberOfReplicas": 4,
    "refreshInterval": 1
  }
}

• indexSettings parameter is optional and defines the following Elasticsearch/Opensearch index settings:
  • numberOfShards - the number (between 1 and 100) of primary shards for the index
  • numberOfReplicas - the number of replicas (between 0 and 100) each primary shard has
  • refreshInterval - the refresh interval reflecting how often to make new changes to the index visible to search (seconds). -1 disables refresh.

It runs the async process which performs both steps described above.

If Instances Indexing was performed in the past, we have aggregated data of instances stored in our models, and changes we made do not require to perform mapping between the inventory Instance resources and OpenSearch indices then running the Merge Step in this and subsequent times is not necessary, and we can run only Upload step. In order to do this we can call the following async API:

POST /search/index/instance-records/reindex/upload

x-okapi-tenant: [tenant]
x-okapi-token: [JWT_TOKEN]

{
  "entityTypes": ["instance", "subject", "contributor", "classification", "call-number"]
}

We can call the above API as upload reindex API as it runs only Upload step. So, the assumption for running upload reindex is that Merge step was run before with full reindex API. If it is not the case or if our changes require to run the Merge step which means there are breaking changes on the mapping between the inventory Instance resources and OpenSearch indices then full reindex API.

We can specify one or many resource types in entityTypes array depending on our needs and if more than one resource type is given then upload step will run for each one of the specified resources.

In order to monitor the async processes initiated by the above two APIs we can execute the endpoint:

GET /search/index/instance-records/reindex/status

and it returns a response showing the status details for the latest run operations, merge and upload alike, which may look like the following:

[
  {
    "entityType": "holding",
    "status": "MERGE_COMPLETED",
    "totalMergeRanges": 5,
    "processedMergeRanges": 5,
    "totalUploadRanges": 0,
    "processedUploadRanges": 0,
    "startTimeMerge": "2024-04-01T00:38:34.15755006Z",
    "endTimeMerge": "2024-04-01T00:38:35.15755006Z"
  },
  {
    "entityType": "item",
    "status": "MERGE_COMPLETED",
    "totalMergeRanges": 3,
    "processedMergeRanges": 3,
    "totalUploadRanges": 0,
    "processedUploadRanges": 0,
    "startTimeMerge": "2024-04-01T00:37:34.15755006Z",
    "endTimeMerge": "2024-04-01T00:37:35.15755006Z"
  },
  {
    "entityType": "instance",
    "status": "UPLOAD_COMPLETED",
    "totalMergeRanges": 3,
    "processedMergeRanges": 3,
    "totalUploadRanges": 2,
    "processedUploadRanges": 2,
    "startTimeMerge": "2024-04-01T00:37:34.15755006Z",
    "endTimeMerge": "2024-04-01T00:37:35.15755006Z",
    "startTimeUpload": "2024-04-01T00:37:36.15755006Z",
    "endTimeUpload": "2024-04-01T00:37:37.15755006Z"
  },
  {
    "entityType": "subject",
    "status": "UPLOAD_COMPLETED",
    "totalMergeRanges": 0,
    "processedMergeRanges": 0,
    "totalUploadRanges": 2,
    "processedUploadRanges": 2,
    "startTimeUpload": "2024-04-01T01:37:36.15755006Z",
    "endTimeUpload": "2024-04-01T01:37:37.15755006Z"
  }
]

When entityType has value of item or holding then we have status details of Merge step and when entityType has value of subject, contributor or classification then we have status details of Upload step. Only for entity type of instance we can have statuses of both Merge and Upload steps.

status response field can have values of "MERGE_IN_PROGRESS", "MERGE_COMPLETED" or "MERGE_FAILED" for entity types representing Merge step and values of "UPLOAD_IN_PROGRESS", "UPLOAD_COMPLETED" or "UPLOAD_FAILED" for the entities of Upload step.

We use CQL query for search queries, see documentation for more details.

In mod-search there are two main types of searchable fields:

1. Full text capable fields (aka. multi-lang fields) - analyzed and preprocessed fields;
2. Term fields (keywords, bool, date fields, etc.) - non-analyzed fields.

Depending on field type, CQL operators will be handled in different ways or not supported at all. Here is table of supported operators.

CQL operators could have modifiers that change search behaviour

Consortium feature is defined automatically at runtime by calling /user-tenants endpoint. Consortium feature on module enable is defined by 'centralTenantId' tenant parameter. Example:

{
  "module_to": "mod-sample-1.3.1",
  "parameters": [
    {
      "key": "centralTenantId",
      "value": "centralTenant"
    },
    {
      "key": "loadReference",
      "value": true
    }
  ]
}

The main endpoint that provides search capabilities is GET /search/instances. It consumes following request parameters:

*Basic fields for instances search are following:

• id
• tenantId
• shared
• hrid
• discoverySuppress
• isBoundWith
• title
• contributors
• identifiers
• publication
• dates

*Basic fields for authorities search are following:

• id
• headingType
• authRefType
• headingRef
• sourceFileId
• naturalId

A search matching all records in the target index can be executed with a cql.allRecords=1 (CQL standard, the fastest option) or a id=* (slower option, check all documents in index) query. They can be used alone or as part of a more complex query. Examples:

• cql.allRecords=1 NOT contributors=Smith sortBy title/sort.ascending matches all records where contributors name does not contain Smith as a word.
• id=* NOT contributors=Smith sortBy title/sort.ascending matches all records where contributors name does not contain Smith as a word.

A relation does not match if the value on the left-hand side is undefined. A negation (using NOT) of a relation matches if the value on the left-hand side is not defined or if it is defined but doesn't match.

• name="" matches all records where name is defined.
• cql.allRecords=1 NOT name="" matches all records where name is not defined.
• name=="" matches all records where name is defined and empty.
• cql.allRecords=1 NOT name=="" matches all records where name is defined and not empty or where name is not defined.
• name="" NOT name=="" matches all records where name is defined and not empty.
• languages == "[]" for matching records where lang is defined and an empty array

Search by all feature is optional and disabled by default. However, it can be enabled for tenant using following HTTP request:

POST /search/config/features

{
  "feature": "search.all.fields",
  "enabled": true
}

Also, search by all fields can be enabled globally by passing to mod-search service following ENV variable:

SEARCH_BY_ALL_FIELDS_ENABLED=true

By default, indexing processors for fields cql.allInstance, cql.allItems, cql.allHoldings are disabled and does not produce any values, so the following search options will return an empty result.

Facets can be retrieved by using following API GET /{recordType}/facets. It consumes following request parameters:

The module supports 2 forms of query parameters for the facet parameter:

GET /instances/facets?query=title all book&facet=source:5&facet=discoverySuppress:2

or

GET /instances/facets?query=title all book&facet=source:5,discoverySuppress:2

The default sorting is by relevancy. The sortBy clause is used to define sorting, for example:

title all "semantic web" sortBy title/sort.descending - sort by title in descending order

In case where options are similar, secondary sort is used

Query parameters

The query operator works as it described in CQL Query operators section. Anchor will be included only if <= or >= are used in the query. Otherwise, the empty row will be added if highlightMatch is equal to true. For call-number browsing check the query syntax here

The process of retrieving ids has two steps:

• Create a job with a CQL query
• Retrieve ids when a job is completed

Send a POST request to create a Job POST /search/resources/jobs

{
  "query": "id=*",
  "entityType": "AUTHORITY"
}

It is possible to check job status by job ID.

GET /search/resources/jobs/{jobId}

Response

{
  "id": "36233d1c-e4c1-4403-a6ee-1d4b5d460dc5",
  "query": "id=*",
  "status": "COMPLETED",
  "entityType": "AUTHORITY",
  "createdDate": "2022-04-29 00:00"
}

When the job is COMPLETED, it is possible to retrieve ids by job id

GET /search/resources/jobs/{jobId}/ids

After retrieving ids, job should change status to "DEPRECATED". If there are no completed job with prepared ids, client can't receive ids by query.

Special API that provide consolidated access to records in consortium environment. Works only for central tenant.

See project MSEARCH at the FOLIO issue tracker.

This module's API documentation.

SonarQube analysis.

The built artifacts for this module are available. See configuration for repository access, and the Docker image

The development tips are described on the following page: Development tips