Querying

Foundatio.Repositories provides powerful querying capabilities through ISearchableRepository<T>. The query system is built on Foundatio.Parsers, which provides Lucene-style query parsing with support for filtering, sorting, and aggregations.

Query Parser (Foundatio.Parsers)

The query expressions used throughout this library are powered by Foundatio.Parsers, which translates human-readable query strings into Elasticsearch queries. Key features include:

• Lucene-style syntax - Familiar query syntax for developers
• Field aliasing - Map user-friendly names to actual field names
• Type coercion - Automatic type conversion for dates, numbers, etc.
• Validation - Query validation against index mappings
• Extensibility - Custom query visitors and field resolvers

Basic Queries

Find with Filter Expression

Use Lucene-style filter expressions:

// Simple field match
var results = await repository.FindAsync(q => q.FilterExpression("age:30"));

// Range queries
var results = await repository.FindAsync(q => q.FilterExpression("age:>=25"));
var results = await repository.FindAsync(q => q.FilterExpression("age:[25 TO 35]"));

// Multiple conditions (AND)
var results = await repository.FindAsync(q => q.FilterExpression("age:>=25 AND department:Engineering"));

// OR conditions
var results = await repository.FindAsync(q => q.FilterExpression("department:Engineering OR department:Sales"));

// NOT conditions
var results = await repository.FindAsync(q => q.FilterExpression("NOT status:inactive"));

// Wildcards
var results = await repository.FindAsync(q => q.FilterExpression("name:John*"));

// Exists check
var results = await repository.FindAsync(q => q.FilterExpression("_exists_:email"));

Prefer Strongly-Typed Queries

Filter expressions are convenient for dynamic or user-supplied queries, but for application code we recommend using Strongly-Typed Queries whenever possible. Strongly-typed queries provide compile-time field name checking, full IDE support (Find References, Rename/Refactor, Go to Definition), and runtime field-type validation that catches misuse like FieldEquals on text-only fields with a QueryValidationException.

Find with Search Expression

Full-text search across analyzed fields:

var results = await repository.FindAsync(q => q.SearchExpression("john developer"));

Default Search Fields

SearchExpression generates a multi-match query across a set of default search fields. Without explicit configuration, Elasticsearch uses its own index.query.default_field setting (typically *, which matches all top-level fields). You can control exactly which fields are searched by overriding ConfigureQueryParser on your Index<T> class:

public sealed class EmployeeIndex : Index<Employee>
{
    // ... constructor, ConfigureIndex, ConfigureIndexMapping ...

    protected override void ConfigureQueryParser(ElasticQueryParserConfiguration config)
    {
        base.ConfigureQueryParser(config);
        config.SetDefaultFields([
            nameof(Employee.Name).ToLowerInvariant(),
            nameof(Employee.EmailAddress).ToLowerInvariant()
        ]);
    }
}

With this configuration, SearchExpression("john") generates a multi-match query targeting only name and emailAddress rather than every field in the index.

Alternative: CopyTo with a Catch-All Field

Instead of multi-match, you can copy field values into a single analyzed field at index time using CopyTo:

public override TypeMappingDescriptor<Employee> ConfigureIndexMapping(
    TypeMappingDescriptor<Employee> map)
{
    return map
        .Dynamic(false)
        .Properties(p => p
            .SetupDefaults()
            .Text(f => f.Name("_all"))
            .Text(f => f.Name(e => e.Name).AddKeywordAndSortFields()
                .CopyTo(c => c.Field("_all")))
            .Keyword(f => f.Name(e => e.EmailAddress)
                .CopyTo(c => c.Field("_all")))
        );
}

Approach	Trade-offs
SetDefaultFields	No extra index storage; generates a multi-match query at search time. Keeps the index lean.
CopyTo catch-all	Increases index size (field values stored twice); single-field match at search time, which can be faster for high-throughput queries.

Both approaches can be combined. For nested field support in default search fields, see Default Fields with Nested Paths in the Nested Queries section.

Find One

Get a single matching document:

var hit = await repository.FindOneAsync(q => q.FieldEquals(e => e.Email, "john@example.com"));
var employee = hit?.Document;

Strongly-Typed Queries

Field Equals

// Single value
var results = await repository.FindAsync(q => q.FieldEquals(e => e.Department, "Engineering"));

// Multiple values (OR)
var results = await repository.FindAsync(q => q.FieldEquals(e => e.Status, "active", "pending"));

// Enum values
var results = await repository.FindAsync(q => q.FieldEquals(e => e.Type, EmployeeType.FullTime));

Field Conditions

FieldCondition supports equality, text matching, existence checks, and range comparisons:

// Equality check
var results = await repository.FindAsync(q => q
    .FieldCondition(e => e.Name, ComparisonOperator.Equals, "John Smith"));

// Contains (for text fields)
var results = await repository.FindAsync(q => q
    .FieldCondition(e => e.Name, ComparisonOperator.Contains, "John"));

Available operators: Equals, NotEquals, IsEmpty, HasValue, Contains, NotContains, GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual.

Range Operators

For one-sided comparisons and non-date types, use the range operator shorthands:

// Date comparison (generates DateRangeQuery)
var results = await repository.FindAsync(q => q
    .FieldLessThanOrEqual(e => e.SnoozeUntilUtc, DateTime.UtcNow));

// Numeric comparison (generates NumericRangeQuery)
var results = await repository.FindAsync(q => q
    .FieldGreaterThanOrEqual(e => e.Age, 18));

// Bounded range (two conditions ANDed)
var results = await repository.FindAsync(q => q
    .FieldGreaterThanOrEqual(e => e.Age, 18)
    .FieldLessThan(e => e.Age, 65));

// String/keyword comparison (generates TermRangeQuery on keyword field)
var results = await repository.FindAsync(q => q
    .FieldGreaterThanOrEqual(e => e.InstanceKey, "20240101"));

// Conditional range (only applied when condition is true)
var results = await repository.FindAsync(q => q
    .FieldGreaterThanIf(e => e.Age, minAge, minAge is not null));

String ranges require keyword fields: String range operators generate a TermRangeQuery and automatically resolve to the .keyword sub-field (like FieldEquals). If the field is an analyzed text field with no .keyword sub-field, a QueryValidationException is thrown at build time.

DateRange vs range operators: Use .DateRange(start, end, field) for bounded date windows (validates start < end, supports timezone). Use FieldGreaterThan/FieldLessThanOrEqual etc. for one-sided comparisons and non-date types.

Numeric precision note: long values use NEST's LongRangeQuery which preserves full precision. decimal values are converted to double for NEST's NumericRangeQuery, which may lose precision for values exceeding ~15-17 significant digits. If exact precision matters, prefer long fields with an explicit scaling factor.

Contains (Full-Text Token Matching)

FieldContains generates a MatchQuery on analyzed fields. It matches complete analyzed tokens, NOT substrings or wildcards:

// Matches documents where Name contains the token "eric"
var results = await repository.FindAsync(q => q
    .FieldContains(e => e.Name, "Eric"));

// Multiple tokens: all must be present (AND), order-independent
// Matches "Eric J. Smith" AND "Smith, Eric" but NOT "Eric"
var results = await repository.FindAsync(q => q
    .FieldContains(e => e.Name, "Eric Smith"));

// DOES NOT WORK: "Er" is not a complete token
var results = await repository.FindAsync(q => q
    .FieldContains(e => e.Name, "Er"));  // returns nothing

OR / AND / NOT Grouping

For complex boolean logic, use FieldOr, FieldAnd, and FieldNot:

// Simple OR: match either condition
var results = await repository.FindAsync(q => q.FieldOr(g => g
    .FieldEquals(f => f.IsPrivate, false)
    .FieldEquals(f => f.CompanyId, companyIds)
));

// Nested AND inside OR
var results = await repository.FindAsync(q => q.FieldOr(g => g
    .FieldEquals(f => f.Status, "RunNow")
    .FieldAnd(g2 => g2
        .FieldEquals(f => f.IsEnabled, true)
        .FieldLessThanOrEqual(f => f.NextRunDateUtc, DateTime.UtcNow)
    )
));

// NOT: exclude documents matching any condition (AND-NOT semantics)
var results = await repository.FindAsync(q => q.FieldNot(g => g
    .FieldEquals(f => f.BillingStatus, BillingStatus.Active)
    .FieldEquals(f => f.BillingStatus, BillingStatus.Trialing)
));

// Dynamic/conditional OR groups (builder API)
var group = FieldConditionGroup<Program>.Or();
group.FieldEquals(f => f.IsPrivate, false);
if (privateIds.Count > 0)
    group.FieldEquals(f => f.PrivateId, privateIds);
if (includeIds.Count > 0)
    group.FieldEquals(f => f.Id, includeIds);
var results = await repository.FindAsync(q => q.FieldOr(group));

FieldNot semantics: Multiple conditions inside FieldNot produce NOT A AND NOT B (exclude documents matching any clause). For NOT (A AND B), nest an explicit AND: FieldNot(g => g.FieldAnd(g2 => g2.FieldEquals(A).FieldEquals(B))).

Field-Type Validation

The query builder performs runtime validation and throws QueryValidationException for detectable misuse:

• FieldEquals/FieldNotEquals on text-only fields — throws if the field is an analyzed text field with no .keyword sub-field (TermQuery on analyzed fields almost never matches).
• FieldContains/FieldNotContains on keyword fields — throws because MatchQuery requires an analyzed field.
• FieldEquals on IsDeleted with ActiveOnly soft-delete mode — throws because it creates a contradictory filter.
• Range with null value — throws with guidance to use *If variants or FieldHasValue/FieldEmpty.
• Range with collection value — throws with guidance to use FieldEquals for multi-value matching.

Note: For wildcard/prefix matching on keyword fields, use FilterExpression("field:pattern*"). For phrase matching (adjacent words in order), use FilterExpression("field:\"quick brown\"").

Field Empty/Has Value

// Find documents where field is null or empty
var results = await repository.FindAsync(q => q.FieldEmpty(e => e.ManagerId));

// Find documents where field has a value
var results = await repository.FindAsync(q => q.FieldHasValue(e => e.ManagerId));

Date Range

.DateRange() adds an Elasticsearch filter clause that restricts documents by a date field value. It does not select which physical indexes are queried — it only filters documents within whatever indexes are targeted.

var results = await repository.FindAsync(q => q
    .DateRange(
        start: DateTime.UtcNow.AddDays(-30),
        end: DateTime.UtcNow,
        field: e => e.CreatedUtc));

Time-Series Indexes (DailyIndex / MonthlyIndex)

When querying a DailyIndex or MonthlyIndex, each partition (day or month) is a separate physical Elasticsearch index. Without specifying which indexes to target, the query runs against the umbrella alias and scans all partitions regardless of the date range filter.

To limit which physical indexes are queried, use .Index(start, end) alongside .DateRange():

var start = DateTime.UtcNow.AddDays(-7);
var end = DateTime.UtcNow;

var results = await repository.FindAsync(q => q
    .Index(start, end)                         // target only the relevant daily index partitions
    .DateRange(start, end, e => e.CreatedUtc)  // filter documents within those indexes
);

Note: .Index(start, end) and .DateRange() serve different purposes and must be set independently. DateRange without .Index() is still valid — it will filter documents correctly — but it queries all partitions, which is less efficient for large time-series datasets.

Large Range Fallback

To prevent generating an excessively long list of individual index names, index selection falls back to the alias (which covers all partitions) when the requested range is too broad:

Index type	Threshold	Behavior
DailyIndex	Range >= 3 months, or exceeds MaxIndexAge	Falls back to alias
MonthlyIndex	Range > 1 year, or exceeds MaxIndexAge	Falls back to alias

The query is still executed correctly in the fallback case — Elasticsearch simply searches all partitions — but there is no index pruning optimization. The .DateRange() filter still narrows the returned documents.

ID Queries

// Find by IDs
var results = await repository.FindAsync(q => q.Id("emp-1", "emp-2", "emp-3"));

// Exclude IDs
var results = await repository.FindAsync(q => q.ExcludedId("emp-1"));

Sorting

Sort Expression

// Single field ascending
var results = await repository.FindAsync(q => q.SortExpression("name"));

// Descending
var results = await repository.FindAsync(q => q.SortExpression("-createdUtc"));

// Multiple fields
var results = await repository.FindAsync(q => q.SortExpression("department -salary"));

Strongly-Typed Sort

var results = await repository.FindAsync(q => q
    .SortAscending(e => e.Name)
    .SortDescending(e => e.CreatedUtc));

Pagination

Basic Pagination

var results = await repository.FindAsync(
    q => q.FieldEquals(e => e.Department, "Engineering"),
    o => o.PageNumber(1).PageLimit(25));

Console.WriteLine($"Page {results.Page}, Total: {results.Total}, HasMore: {results.HasMore}");

Automatic Pagination

var results = await repository.FindAsync(query, o => o.PageLimit(100));

do
{
    foreach (var doc in results.Documents)
    {
        await ProcessAsync(doc);
    }
} while (await results.NextPageAsync());

Snapshot Paging (Scroll API)

For large result sets, use snapshot paging:

var results = await repository.FindAsync(
    query,
    o => o.SnapshotPaging().SnapshotPagingLifetime(TimeSpan.FromMinutes(5)));

do
{
    foreach (var doc in results.Documents)
    {
        await ProcessAsync(doc);
    }
} while (await results.NextPageAsync());

Search After Paging

More efficient for deep pagination:

var results = await repository.FindAsync(
    q => q.SortExpression("createdUtc"),
    o => o.SearchAfterPaging());

// For subsequent pages, use the token
var nextResults = await repository.FindAsync(
    q => q.SortExpression("createdUtc"),
    o => o.SearchAfterToken(results.GetSearchAfterToken()));

Aggregations

Aggregation Expression

var results = await repository.CountAsync(q => q
    .AggregationsExpression("terms:department terms:status"));

// Access aggregation results
var departmentAgg = results.Aggregations.Terms("terms_department");
foreach (var bucket in departmentAgg.Buckets)
{
    Console.WriteLine($"{bucket.Key}: {bucket.Total}");
}

Common Aggregations

// Terms aggregation
var results = await repository.CountAsync(q => q
    .AggregationsExpression("terms:department"));

// Date histogram
var results = await repository.CountAsync(q => q
    .AggregationsExpression("date:createdUtc"));

// Cardinality (unique count)
var results = await repository.CountAsync(q => q
    .AggregationsExpression("cardinality:userId"));

// Statistics
var results = await repository.CountAsync(q => q
    .AggregationsExpression("avg:salary min:salary max:salary"));

// Multiple aggregations
var results = await repository.CountAsync(q => q
    .AggregationsExpression("terms:department avg:salary cardinality:userId"));

Accessing Aggregation Results

var results = await repository.CountAsync(q => q
    .AggregationsExpression("terms:department avg:salary max:createdUtc"));

// Terms aggregation
var deptAgg = results.Aggregations.Terms("terms_department");
foreach (var bucket in deptAgg.Buckets)
{
        Console.WriteLine($"Department: {bucket.Key}, Count: {bucket.Total}");
}

// Value aggregations
var avgSalary = results.Aggregations.Average("avg_salary")?.Value;
var maxDate = results.Aggregations.Max<DateTime>("max_createdUtc")?.Value;

Console.WriteLine($"Average Salary: {avgSalary}");
Console.WriteLine($"Latest Created: {maxDate}");

Nested Field Aggregations

When aggregating on fields that are mapped as nested in Elasticsearch, the framework automatically wraps the aggregation in a nested aggregation context. Use the same parentObject.childField syntax you use for flat fields:

// Terms aggregation on a nested field
var results = await repository.CountAsync(q => q
    .AggregationsExpression("terms:peerReviews.rating"));

// Multiple aggregation types on nested fields
var results = await repository.CountAsync(q => q
    .AggregationsExpression("terms:peerReviews.reviewerEmployeeId min:peerReviews.rating max:peerReviews.rating"));

The framework detects nested fields via the index mapping and groups all nested aggregations under a single SingleBucketAggregate keyed by the nested path. Access results through that wrapper:

var results = await repository.CountAsync(q => q
    .AggregationsExpression("terms:peerReviews.rating min:peerReviews.rating max:peerReviews.rating"));

// All nested aggregations are grouped under a single-bucket aggregate
var nestedAgg = results.Aggregations["nested_peerReviews"] as SingleBucketAggregate;

var ratingTerms = nestedAgg.Aggregations.Terms<int>("terms_peerReviews.rating");
foreach (var bucket in ratingTerms.Buckets)
{
    Console.WriteLine($"Rating {bucket.Key}: {bucket.Total}");
}

var minRating = nestedAgg.Aggregations.Min("min_peerReviews.rating")?.Value;
var maxRating = nestedAgg.Aggregations.Max("max_peerReviews.rating")?.Value;

Include and exclude filtering works the same way as non-nested aggregations:

// Only include specific terms
var results = await repository.CountAsync(q => q
    .AggregationsExpression("terms:(peerReviews.reviewerEmployeeId @include:emp1 @include:emp2)"));

// Exclude specific terms
var results = await repository.CountAsync(q => q
    .AggregationsExpression("terms:(peerReviews.rating @exclude:1 @exclude:2)"));

Top Hits Aggregation

The tophits sub-aggregation returns the top matching documents within each bucket:

var results = await repository.CountAsync(q => q
    .AggregationsExpression("terms:(age tophits:_)"));

var bucket = results.Aggregations.Terms<int>("terms_age").Buckets.First();
var topHits = bucket.Aggregations.TopHits();
var employees = topHits.Documents<Employee>();

TopHitsAggregate Cannot Be Serialized

TopHitsAggregate holds ILazyDocument references that contain raw Elasticsearch document bytes and require an active serializer instance to materialize into typed objects. These references are lost during JSON serialization, which means:

• Caching: CountResult or FindResults containing TopHitsAggregate cannot be cached and restored via JSON serialization (Newtonsoft or System.Text.Json). The top hits data will be null after deserialization.
• Workaround: If you need to cache results that include top hits, materialize the documents into concrete types before caching, and cache those typed results separately.

Nested Queries

When querying fields inside nested objects, the framework automatically wraps filter expressions in the required Elasticsearch nested query. You do not need to manually construct nested queries -- just use dotted field paths.

Prerequisites

The field must be mapped as nested in your index configuration:

public override TypeMappingDescriptor<Employee> ConfigureIndexMapping(
    TypeMappingDescriptor<Employee> map)
{
    return map
        .Dynamic(false)
        .Properties(p => p
            .SetupDefaults()
            .Keyword(f => f.Name(e => e.Id))
            .Text(f => f.Name(e => e.Name).AddKeywordAndSortFields())
            .Nested<PeerReview>(f => f.Name(e => e.PeerReviews).Properties(p1 => p1
                .Keyword(f2 => f2.Name(p2 => p2.ReviewerEmployeeId))
                .Scalar(p3 => p3.Rating, f2 => f2.Name(p3 => p3.Rating))))
        );
}

Basic Nested Queries

Query nested fields with standard filter expressions using parentObject.childField syntax:

// Exact match on a nested field
var results = await repository.FindAsync(q => q
    .FilterExpression("peerReviews.rating:5"));

// Range query on a nested field
var results = await repository.FindAsync(q => q
    .FilterExpression("peerReviews.rating:[4 TO 5]"));

// Match on a nested keyword field
var results = await repository.FindAsync(q => q
    .FilterExpression("peerReviews.reviewerEmployeeId:bob_456"));

Combining Nested Conditions

Multiple conditions on the same nested path are combined into a single nested query with a bool clause:

// AND: both conditions must match the SAME nested document
var results = await repository.FindAsync(q => q
    .FilterExpression("peerReviews.rating:5 AND peerReviews.reviewerEmployeeId:bob_456"));

// OR: either condition can match across different nested documents
var results = await repository.FindAsync(q => q
    .FilterExpression("peerReviews.rating:>=4 OR peerReviews.reviewerEmployeeId:bob_456"));

Mixing Nested and Non-Nested Fields

Nested fields and regular fields can be used together. The framework only wraps the nested portions in a nested query:

// "name" is a root-level field, "peerReviews.rating" is nested
var results = await repository.FindAsync(q => q
    .FilterExpression("name:Alice peerReviews.rating:5"));

Negating Nested Conditions

// Exclude employees who have any peer review with rating 5
var results = await repository.FindAsync(q => q
    .FilterExpression("NOT peerReviews.rating:5"));

Default Fields with Nested Paths

When default search fields include nested field paths, the framework automatically wraps the corresponding portion of the multi-match query in a nested query. No additional configuration is needed beyond including the dotted path in SetDefaultFields:

protected override void ConfigureQueryParser(ElasticQueryParserConfiguration config)
{
    base.ConfigureQueryParser(config);
    config.SetDefaultFields([
        nameof(Employee.Id).ToLowerInvariant(),
        nameof(Employee.Name).ToLowerInvariant(),
        "peerReviews.reviewerEmployeeId"  // nested field
    ]);
}

With this configuration, a bare search term like bob_456 will match against id and name (root-level) as well as peerReviews.reviewerEmployeeId (nested). The nested portion is automatically detected via the index mapping:

// Searches id, name, AND the nested peerReviews.reviewerEmployeeId field
var results = await repository.FindAsync(q => q.SearchExpression("bob_456"));

Sorting on Nested Fields

Sort expressions on nested fields automatically include the required nested context:

// Sort descending by a nested numeric field
var results = await repository.FindAsync(q => q
    .SortExpression("-peerReviews.rating"));

Exists / Missing on Nested Fields

_exists_ and _missing_ queries on nested fields are automatically wrapped in a nested query:

// Find employees that have at least one peer review with a reviewerEmployeeId
var results = await repository.FindAsync(q => q
    .FilterExpression("_exists_:peerReviews.reviewerEmployeeId"));

Deeply Nested Types

Multi-level nesting (nested objects inside other nested objects) is supported. The framework resolves the correct nested path at each level:

// Given a mapping: parent (nested) -> child (nested inside parent)
// Query a deeply nested field
var results = await repository.FindAsync(q => q
    .FilterExpression("parent.child.field1:value"));

Known Limitations

Limitation	Details
TopHits round-tripping	TopHitsAggregate cannot survive JSON serialization. See the Top Hits Aggregation warning above.

Field Selection

Field selection controls which fields are returned from Elasticsearch via _source filtering. This reduces network payload and deserialization cost when you only need a subset of fields from a document.

Including Fields

Use .Include() to specify individual fields to return:

var results = await repository.FindAsync(
    query,
    o => o.Include(e => e.Id).Include(e => e.Name).Include(e => e.Email));

You can also pass multiple fields at once:

var results = await repository.FindAsync(
    query,
    o => o.Include(e => e.Id, e => e.Name, e => e.Email));

Excluding Fields

Use .Exclude() to omit specific fields while returning everything else:

var results = await repository.FindAsync(
    query,
    o => o.Exclude(e => e.LargeContent).Exclude(e => e.Attachments));

Field Mask Expressions

For complex field selections, use .IncludeMask() or .ExcludeMask() with a Google FieldMask-style expression. Nested fields are grouped with parentheses and comma-separated:

Expression	Expanded Fields
"id,name"	id, name
"address(street,city)"	address.street, address.city
"results(id,program(name,id))"	results.id, results.program.name, results.program.id

var results = await repository.FindAsync(
    query,
    o => o.IncludeMask("id,name,address(street,city,state)"));

Masks and individual .Include()/.Exclude() calls are additive -- they are merged into a single set at query time.

Query-Level vs Options-Level

Field includes and excludes can be set on both the query and the command options. Both sources are merged at execution time:

var results = await repository.FindAsync(
    q => q.Include(e => e.Name),
    o => o.Include(e => e.Email));
// Both Name and Email are included

This is useful when a repository method sets default options-level field restrictions while callers add query-level overrides.

Merge and Precedence Rules

At execution time, the repository merges all field selection settings:

1. All includes are merged: individual fields from .Include() and parsed fields from .IncludeMask() from both the query and command options combine into one include set.
2. All excludes are merged: same for .Exclude() and .ExcludeMask().
3. Includes win over excludes: if the same field appears in both includes and excludes, it is included (the exclude is dropped).
4. Automatic Id field: when any includes are specified on an entity that implements IIdentity, the Id field is automatically added to ensure the document identity is always available.

Default Excludes

Repositories can register fields to exclude by default by calling AddDefaultExclude() in the constructor:

public class EmployeeRepository : ElasticRepositoryBase<Employee>
{
    public EmployeeRepository(/* ... */)
    {
        AddDefaultExclude(e => e.InternalNotes);
        AddDefaultExclude(e => e.AuditLog);
    }
}

Default excludes are only applied when no explicit excludes are set on the query. As soon as the caller specifies any .Exclude() call, the defaults are skipped entirely. This prevents unexpected interactions between default and explicit excludes.

Required Fields

Repositories can register fields that must always be present when any caller-specified _source field restrictions are active. This is useful for:

• Multi-tenancy: Ensuring OrganizationId or ProjectId is always available for authorization checks
• Cache invalidation: Fields used as custom cache keys must be present to invalidate correctly
• Event handling: Fields needed by DocumentsChanged / DocumentsChanging handlers

Call AddRequiredField() in the repository constructor. Multiple fields can be registered in a single call:

public class StackRepository : ElasticRepositoryBase<Stack>
{
    public StackRepository(MyAppElasticConfiguration configuration)
        : base(configuration.Stacks)
    {
        AddRequiredField(s => s.OrganizationId, s => s.ProjectId);
    }
}

When Required Fields Are Injected

Required fields are only injected when field restrictions are active — i.e., when any includes, excludes, masks, or default excludes are present. When no restrictions exist at all, the full _source is returned and required fields have no effect.

When repository-internal AddDefaultExclude() registrations are the only excludes present, required field injection runs but is effectively a no-op — default excludes never overlap with required fields like Id or CreatedUtc, so no fields are added or removed. If a field is registered as both a required field and a default exclude, the required field takes precedence — the field will be removed from the exclude set.

How Required Fields Are Applied

The behavior depends on what the caller specified:

• Caller has includes (with or without excludes): Required fields are added to the include set. This ensures they appear in the narrowed result alongside the caller's selected fields.
• Caller has only excludes: Required fields are removed from the exclude set. This preserves the "return everything except X" semantics while ensuring required fields cannot be excluded. All other non-excluded fields remain present.

Precedence Rules

When the caller specifies both includes and excludes, and a required field appears in both sets, the include takes precedence — the field is returned. This mirrors how systems like OData and GraphQL handle mandatory fields in projections: identity and authorization fields are always present regardless of what the client requests.

Affected Operations

Required fields apply to all source-filtered operations: GetByIdAsync, GetByIdsAsync, FindAsync, PatchAllAsync, and BatchProcessAsync.

Impact on Minimal-Field Queries

When you use AddRequiredField, callers requesting minimal fields (e.g., .Include(e => e.Id) for a lightweight list view) will also receive the required fields. Factor this into your API design — required fields add a small payload overhead to every partial-document response but ensure correctness for authorization, caching, and event handling.

Custom Cache Key Fields

If your repository uses custom cache keys based on specific field values (e.g., caching by CompanyId), register those fields as required to ensure cache invalidation works correctly even when callers request partial documents:

public class EventRepository : ElasticRepositoryBase<Event>
{
    public EventRepository(MyAppElasticConfiguration configuration)
        : base(configuration.Events)
    {
        AddRequiredField(e => e.OrganizationId);
    }

    protected override async Task InvalidateCacheAsync(
        IReadOnlyCollection<ModifiedDocument<Event>> documents, ChangeType? changeType = null)
    {
        await base.InvalidateCacheAsync(documents, changeType);
        // OrganizationId is guaranteed present because it's a required field
        await Cache.RemoveAllAsync(documents.Select(d => $"org:{d.Value.OrganizationId}"));
    }
}

Caching Impact

When includes or excludes are active, the repository skips ID-based caching to avoid storing incomplete documents in the cache. This means:

• GetByIdAsync / GetByIdsAsync with field restrictions will always hit Elasticsearch directly.
• Queries with custom cache keys still function normally since they cache the complete filtered result as-is.

If performance is important and you frequently fetch partial documents, consider using a dedicated query with a custom cache key rather than relying on ID-based caching.

Count and Exists

Count with Query

var count = await repository.CountAsync(q => q.FieldEquals(e => e.Department, "Engineering"));
Console.WriteLine($"Engineering employees: {count.Total}");

Exists with Query

bool hasActiveEmployees = await repository.ExistsAsync(
    q => q.FieldEquals(e => e.Status, "active"));

Building Complex Queries

Combining Query Methods

var results = await repository.FindAsync(q => q
    .FieldEquals(e => e.Status, "active")
    .FieldEquals(e => e.Department, "Engineering")
    .DateRange(DateTime.UtcNow.AddYears(-1), DateTime.UtcNow, e => e.HireDate)
    .SortExpression("-salary")
    .AggregationsExpression("terms:title avg:salary"),
    o => o.PageLimit(50));

Reusable Query Objects

var query = new RepositoryQuery<Employee>()
    .FieldEquals(e => e.Department, "Engineering")
    .FieldCondition(e => e.Name, ComparisonOperator.Contains, "John");

var results = await repository.FindAsync(q => query);
var count = await repository.CountAsync(q => query);

Custom Query Extensions

Create domain-specific query methods:

public static class EmployeeQueryExtensions
{
    public static IRepositoryQuery<Employee> ActiveInDepartment(
        this IRepositoryQuery<Employee> query, string department)
    {
        return query
            .FieldEquals(e => e.Status, "active")
            .FieldEquals(e => e.Department, department);
    }

    public static IRepositoryQuery<Employee> HiredBetween(
        this IRepositoryQuery<Employee> query, DateTime start, DateTime end)
    {
        return query.DateRange(start, end, e => e.HireDate);
    }
}

// Usage
var results = await repository.FindAsync(q => q
    .ActiveInDepartment("Engineering")
    .HiredBetween(DateTime.UtcNow.AddYears(-2), DateTime.UtcNow));

Query Logging

Enable query logging for debugging:

var results = await repository.FindAsync(
    query,
    o => o.QueryLogLevel(Microsoft.Extensions.Logging.LogLevel.Debug));

Async Queries

For long-running queries:

// Start async query
var results = await repository.FindAsync(
    query,
    o => o.AsyncQuery(waitTime: TimeSpan.FromSeconds(5), ttl: TimeSpan.FromHours(1)));

if (results.Total == 0 && results.IsAsyncQueryRunning())
{
    // Query is still running, get the ID
    var queryId = results.GetAsyncQueryId();

    // Check later
    var laterResults = await repository.FindAsync(
        query,
        o => o.AsyncQueryId(queryId, waitTime: TimeSpan.FromSeconds(30)));
}

Next Steps

• Configuration - Query configuration options
• Caching - Cache query results
• Soft Deletes - Query soft-deleted documents
• Index Management - Query across multiple indexes