Skip to content

[meta] 7.6 release #4341

Closed
Closed
[meta] 7.6 release#4341
@russcam

Description

@russcam
russcam
opened on Feb 3, 2020

Preliminary release notes for 7.6.0

(Generated from asciidoc release notes on 2020-02-03)

Breaking changes

Mapping

  • Add a cluster setting to disallow loading fielddata on _id field #49166 (issues: #26472 #43599)

Breaking Java changes

Security

  • Support Client and RoleMapping in custom Realms
    #50534
    (issue:
    #48369)

Deprecations

Analysis

Authorization

  • Deprecating kibana_user and kibana_dashboard_only_user roles
    #46456

Distributed

  • Deprecate synced flush
    #50835
    (issue:
    #50776)

  • Deprecate indices without soft-deletes
    #50502

Features/Indices APIs

  • Emit warnings when index templates have multiple mappings
    #50982

Infra/Core

  • Deprecate the local parameter of /_cat/nodes
    #50499
    (issue:
    #50088) will be picked up by the generator

MULTIPLE AREA LABELS

  • Ensure we emit a warning when using the deprecated template field.
    #50831
    (issue:
    #49460)

  • Add a deprecation warning regarding allocation awareness in search
    request
    #48351
    (issue:
    #43453)

Reindex

Search

  • Update the signature of vector script functions.
    #48604

  • Deprecate the sparse_vector field type.
    #48315

New features

Aggregations

Analysis

Features/ILM+SLM

Features/Ingest

MULTIPLE AREA LABELS

  • Password Protected Keystore (Feature Branch)
    #49210

Machine Learning

Mapping

Search

Enhancements

Aggregations

  • Add reusable HistogramValue object
    #49799
    (issue:
    #49683)

  • Optimize composite aggregation based on index sorting
    #48399
    (issue:
    #48130)

Allocation

  • Auto-expand indices according to allocation filtering rules
    #48974

  • Do not cancel ongoing recovery for noop copy on broken node
    #48265
    (issue:
    #47974)

  • Quieter logging from the DiskThresholdMonitor
    #48115
    (issue:
    #48038)

  • Faster access to INITIALIZING/RELOCATING shards
    #47817
    (issues:
    #46941,
    #48579)

Analysis

Authentication

  • Add Debug/Trace logging for authentication
    #49575
    (issue:
    #49473)

Authorization

  • Increase Size and lower TTL on DLS BitSet Cache
    #50535
    (issues:
    #43669,
    #49260)

  • Add monitor_snapshot cluster privilege
    #50489
    (issue:
    #50210)

  • Remove reserved roles for code search
    #50068
    (issue:
    #49842)

  • [Code] Remove code_admin/code_user roles
    #48164

  • Resolve the role query and the number of docs lazily
    #48036

CCR

  • Improve error message when pausing index
    #48915

  • Use MultiFileTransfer in CCR remote recovery
    #44514
    (issue:
    #44468)

CRUD

  • print id detail when id is too long.
    #49433

  • Add preflight check to dynamic mapping updates
    #48817
    (issue:
    #35564)

Cluster Coordination

Distributed

  • Use retention lease in peer recovery of closed indices
    #48430
    (issue:
    #45136)

Engine

  • Do not force refresh when write indexing buffer
    #50769

  • Deleted docs disregarded for if_seq_no check
    #50526

  • Allow realtime get to read from translog
    #48843

  • Do not warm up searcher in engine constructor
    #48605
    (issue:
    #47186)

  • Add a new merge policy that interleaves old and new segments on
    force merge
    #48533
    (issue:
    #37043)

  • Refresh should not acquire readLock
    #48414
    (issue:
    #47186)

Features/ILM+SLM

Features/Ingest

Features/Java High Level REST Client

Features/Monitoring

  • Significantly Lower Monitoring HttpExport Memory Footprint
    #48854

  • Validate proxy base path at parse time
    #47912
    (issue:
    #47711)

  • Validate index name time format setting at parse time
    #47911
    (issue:
    #47711)

  • Validate monitoring header overrides at parse time
    #47848
    (issue:
    #47711)

  • Validate monitoring username at parse time
    #47821
    (issue:
    #47711)

  • Validate monitoring password at parse time
    #47740
    (issue:
    #47711)

Features/Watcher

  • Log attachment generation failures
    #50080

  • Don’t dump a stacktrace for invalid patterns when executing
    elasticse…
    #49744
    (issue:
    #49642)

Geo

Infra/Core

  • Add "did you mean" to ObjectParser
    #50938

  • Consistent case in CLI option descriptions
    #49635

  • Improve resiliency to formatting JSON in server
    #48553
    (issue:
    #48450)

  • Don’t close stderr under --quiet
    #47208
    (issue:
    #46900)

Infra/Packaging

  • Respect ES_PATH_CONF on package install
    #50158

  • Restrict support for CMS to pre-JDK 14
    #49123
    (issue:
    #46973)

  • Remove parsed JVM settings from general settings in Windows service
    daemon manager
    #49061
    (issue:
    #48796)

  • Package the JDK into jdk.app on macOS
    #48765

  • Add UBI-based Docker images
    #48710
    (issue:
    #48429)

Infra/Plugins

  • Report progress of multiple plugin installs
    #51001
    (issue:
    #50924)

  • Allow installing multiple plugins as a transaction
    #50924
    (issue:
    #50443)

Infra/Scripting

  • Scripting: ScriptFactory not required by compile
    #50344
    (issue:
    #49466)

  • Scripting: Cache script results if deterministic
    #50106
    (issue:
    #49466)

  • Scripting: Groundwork for caching script results
    #49895
    (issue:
    #49466)

  • Scripting: add available languages & contexts API
    #49652
    (issue:
    #49463) EXPERIMENTAL API

  • Scripting: fill in get contexts REST API
    #48319
    (issue:
    #47411) EXPERIMENTAL API

  • Scripting: get context names REST API
    #48026
    (issue:
    #47411) EXPERIMENTAL API

Infra/Settings

  • Add parameter to make sure that log of updating IndexSetting be more
    detailed
    #49969
    (issue:
    #49818)

  • Enable dependent settings values to be validated
    #49942

  • Do not reference values for filtered settings
    #48066

License

MULTIPLE AREA LABELS

Machine Learning

  • [ML] Add audit warning for 1000 categories found early in job
    #51146
    (issue:
    #50749)

  • [ML] Add num_top_feature_importance_values param to regression
    and classi…
    #50914

  • [ML] Implement force deleting a data frame analytics job
    #50553
    (issue:
    #48124) will be picked up by generator

  • [ML] Delete unused data frame analytics state
    #50243

  • Make each analysis report desired field mappings to be copied
    #50219
    (issue:
    #50119)

  • [ML] retry bulk indexing of state docs
    #50149
    (issue:
    #50143)

  • [ML] Persist/restore state for DFA classification
    #50040

  • [ML] Introduce randomize_seed setting for regression and
    classification
    #49990

  • Pass prediction_field_type to C++ analytics process
    #49861
    (issue:
    #49796)

  • [ML] Add optional source filtering during data frame reindexing
    #49690
    (issue:
    #49531)

  • [ML] Add default categorization analyzer definition to ML info
    #49545 PR Add categorization analyzer to defaults on ml info response #4408

  • [ML] Add graceful retry for anomaly detector result indexing
    failures
    #49508
    (issue:
    #45711)

  • Lower minimum model memory limit value from 1MB to 1kB.
    #49227
    (issue:
    #49168)

  • Throw an exception when memory usage estimation endpoint encounters
    empty data frame.
    #49143
    (issue:
    #49140)

  • Change format of MulticlassConfusionMatrix result to be more
    self-explanatory
    #48174
    (issue:
    #46735)

  • Make num_top_classes parameter’s default value equal to 2
    #48119
    (issue:
    #46735)

  • [ML] Improve model_memory_limit UX for data frame analytics jobs
    [ISSUE]
    #44699

Mapping

Network

  • Add certutil http command
    #49827

  • Do not load SSLService in plugin contructor
    #49667
    (issue:
    #44536)

  • Netty4: switch to composite cumulator
    #49478

  • Add the simple strategy to cluster settings
    #49414
    (issue:
    #49067)

  • Deprecate misconfigured SSL server config
    #49280
    (issue:
    #45892)

  • Improved diagnostics for TLS trust failures
    #48911

Percolator

  • Refactor percolator’s QueryAnalyzer to use QueryVisitors
    #49238
    (issue:
    #45639)

Ranking

  • Support search_type in Rank Evaluation API
    #48542
    (issue:
    #48503)

Recovery

  • Use peer recovery retention leases for indices without soft-deletes
    #50351
    (issues:
    #45136,
    #46959)

  • Recovery buffer size 16B smaller
    #50100

Reindex

  • Reindex sort deprecation warning take 2
    #49855
    (issue:
    #49458)

SQL

  • SQL: Add TRUNC alias for TRUNCATE
    #49571
    (issue:
    #41195)

  • SQL: binary communication implementation for drivers and the CLI
    #48261
    (issue:
    #47785)

Search

  • Add Validation for maxQueryTerms to be greater than 0 for
    MoreLikeThisQuery
    #49966
    (issue:
    #49927)

  • Optimize numeric sort on match_all queries
    #49717
    (issue:
    #48804)

  • Pre-sort shards based on the max/min value of the primary sort field
    #49092
    (issue:
    #49091)

  • Optimize sort on long field
    #48804

  • Search optimisation - add canMatch early aborts for queries on
    "_index" field
    #48681
    (issue:
    #48473)

  • #48475 Pure disjunctions should rewrite to a MatchNoneQueryBuilder
    #48557

  • Disable caching when queries are profiled
    #48195
    (issue:
    #33298)

  • BlendedTermQuery’s equals method should consider boosts
    #48193
    (issue:
    #48184)

  • Increase the number of vector dims to 2048
    #46895

Security

  • Make .async-search-* a restricted namespace
    #50294

  • Security should not reload files that haven’t changed
    #50207
    (issue:
    #50063)

Snapshot/Restore

Store

Transform

  • [Transform] Improve force stop robustness in case of an error
    #51072

  • [Transform] add actual timeout in message
    #50140

  • [Transform] automatic deletion of old checkpoints
    #49496

  • [Transform] improve error handling of script errors
    #48887
    (issue:
    #48467)

  • [ML][Transforms] add wait_for_checkpoint flag to stop
    #47935
    (issue:
    #45293)

Bug fixes

Aggregations

  • Fix format problem in composite of unmapped
    #50869
    (issue:
    #50600)

  • SingleBucket aggs need to reduce their bucket’s pipelines first
    #50103
    (issue:
    #50054)

  • Avoid precision loss in DocValueFormat.RAW#parseLong
    #49063
    (issue:
    #38692)

  • Fix ignoring missing values in min/max aggregations
    #48970
    (issue:
    #48905)

Allocation

  • Collect shard sizes for closed indices
    #50645
    (issue:
    #33888)

  • Auto-expand replicated closed indices
    #48973

  • Ignore dangling indices created in newer versions
    #48652
    (issue:
    #34264)

  • Handle negative free disk space in deciders
    #48392
    (issue:
    #48380)

Analysis

  • Throw Error on deprecated nGram and edgeNGram custom filters
    #50376
    (issue:
    #50360)

  • _analyze api does not correctly use normalizers when specified
    #48866
    (issue:
    #48650)

Audit

Authentication

  • Don’t fallback to anonymous for tokens/apikeys
    #51042
    (issue:
    #50171)

  • Populate User metadata with OpenIDConnect collections
    #50521
    (issue:
    #50250)

  • Always return 401 for not valid tokens
    #49736
    (issue:
    #38866)

  • Fix iterate-from-1 bug in smart realm order
    #49473

  • Remove unnecessary details logged for OIDC
    #48746

  • Add owner flag parameter to the rest spec
    #48500
    (issue:
    #48499) will be picked up by generator.

Authorization

  • Fix memory leak in DLS bitset cache
    #50635
    (issue:
    #49261)

  • Validate field permissions when creating a role
    #50212
    (issues:
    #46275,
    #48108)

  • Validate field permissions when creating a role
    #48108
    (issue:
    #46275)

CCR

  • CCR should auto-retry rejected execution exceptions
    #49213

CRUD

  • Block too many concurrent mapping updates
    #51038
    (issue:
    #50670)

  • Ensure meta and document field maps are never null in GetResult
    #50112
    (issue:
    #48215)

  • Replicate write actions before fsyncing them
    #49746

  • Do not mutate request on scripted upsert
    #49578
    (issue:
    #48670)

  • Fix Transport Stopped Exception
    #48930
    (issue:
    #42612)

  • Return consistent source in updates
    #48707

  • Close query cache on index service creation failure
    #48230
    (issue:
    #48186)

Cluster Coordination

  • Import replicated closed dangling indices
    #50649

  • Ignore metadata of deleted indices at start
    #48918

  • Make elasticsearch-node tools custom metadata-aware
    #48390

Discovery-Plugins

  • Make EC2 Discovery Cache Empty Seed Hosts List
    #50607
    (issue:
    #50550)

  • Make EC2 Discovery Plugin Retry Requests
    #50550
    (issue:
    #50462)

Distributed

Engine

  • Account trimAboveSeqNo in committed translog generation
    #50205
    (issue:
    #49970)

  • Greedily advance safe commit on new global checkpoint
    #48559
    (issue:
    #48532)

  • Do not ignore exception when trim unreferenced readers
    #48470

Features/Features

  • Fix X-Pack SchedulerEngine Shutdown
    #48951

Features/ILM+SLM

  • Fix SLM check for restore in progress
    #50868

  • Handle failure to retrieve ILM policy step better
    #49193
    (issue:
    #49128)

  • Don’t halt policy execution on policy trigger exception
    #49128

  • Re-read policy phase JSON when using ILM’s move-to-step API
    #48827

  • Don’t schedule SLM jobs when services have been stopped
    #48658
    (issue:
    #47749)

  • Ensure SLM stats does not block an in-place upgrade from 7.4
    #48367

  • Ensure SLM stats does not block an in-place upgrade from 7.4
    #48361

  • Add SLM support to xpack usage and info APIs
    #48096
    (issue:
    #43663)

  • Change policy_id to list type in slm.get_lifecycle
    #47766
    (issue:
    #47765) will be picked up by generator

Features/Ingest

Features/Java High Level REST Client

  • Support es7 node http publish_address format
    #49279
    (issue:
    #48950)

  • Add slices to delete and update by query in HLRC
    #48420

  • fix incorrect comparison
    #48208

  • Fix HLRC parsing of CancelTasks response
    #47017

Features/Java Low Level REST Client

  • Improve warning value extraction performance in Response
    #50208
    (issue:
    #24114)

Features/Monitoring

Geo

  • Guard against null geoBoundingBox
    #50506
    (issue:
    #50505)

  • Geo: Switch generated GeoJson type names to camel case (#50285)
    #50400
    (issue:
    #49568)

  • Geo: Switch generated WKT to upper case
    #50285
    (issue:
    #49568)

  • Fix typo when assigning null_value in GeoPointFieldMapper
    #49645

  • Fix handling of circles in legacy geo_shape queries
    #49410
    (issue:
    #49296)

  • GEO: intersects search for geo_shape return wrong result [ISSUE]
    #49017

  • Geo: improve handling of out of bounds points in linestrings
    #47939
    (issue:
    #43916)

Highlighting

  • Fix invalid break iterator highlighting on keyword field
    #49566

Infra/Core

  • Ignore virtual ethernet devices that disappear
    #51581
    (issue:
    #49914)

  • Guess root cause support unwrap
    #50525
    (issue:
    #50417)

  • Allow parsing timezone without fully provided time
    #50178
    (issue:
    #49351)

  • [Java.time] Retain prefixed date pattern in formatter
    #48703
    (issue:
    #48698)

  • Don’t drop user’s MaxDirectMemorySize flag on jdk8/windows
    #48657
    (issues:
    #44174,
    #48365)

  • Warn when MaxDirectMemorySize may be incorrect (Windows/JDK8 only
    issue)
    #48365
    (issue:
    #47384)

  • [Java.time] Calculate week of a year with ISO rules
    #48209
    (issues:
    #41670,
    #42588,
    #43275,
    #43652)

Infra/Logging

  • Slow log must use separate underlying logger for each index
    #47234
    (issue:
    #42432)

Infra/Packaging

  • Extend systemd timeout during startup
    #49784
    (issue:
    #49593)

Infra/REST API

  • Return 400 when handling invalid JSON
    #49552
    (issue:
    #49428)

  • Slash missed in indices.put_mapping url
    #49468

MULTIPLE AREA LABELS

Machine Learning

  • [ML] Validate classification dependent_variable cardinality is at
    lea… #51232

  • Do not copy mapping from dependent variable to prediction field in
    regression analysis
    #51227

  • Handle nested and aliased fields correctly when copying mapping.
    #50918
    (issue:
    #50787)

  • [ML] Fix off-by-one error in ml_classic tokenizer end offset
    #50655

  • [ML] Improve uniqueness of result document IDs
    #50644
    (issue:
    #50613)

  • [7.x] Synchronize processInStream.close() call
    #50581
    (issue:
    #49680)

  • Fix accuracy metric
    #50310
    (issue:
    #48759)

  • Fix race condition when stopping a data frame analytics jobs
    immediately after starting it
    #50276
    (issues:
    #49680,
    #50177)

  • Use query in cardinality check
    #49939

  • Make only a part of stop() method a critical section.
    #49756
    (issue:
    #49680)

  • Fix expired job results deletion audit message
    #49560
    (issue:
    #49549)

  • [ML] Apply source query on data frame analytics memory estimation
    #49517
    (issue:
    #49454)

  • Stop timing stats failure propagation
    #49495

  • [ML] Fix r_squared eval when variance is 0
    #49439

  • Blacklist a number of prediction field names.
    #49371
    (issue:
    #48808)

  • Make AnalyticsProcessManager class more robust
    #49282
    (issue:
    #49095)

  • [ML] Fixes for stop datafeed edge cases
    #49191
    (issues:
    #43670,
    #48931)

  • [ML] Avoid NPE when node load is calculated on job assignment
    #49186
    (issue:
    #49150)

  • Do not throw exceptions resulting from persisting datafeed timing
    stats.
    #49044
    (issue:
    #49032)

  • [ML] Deduplicate multi-fields for data frame analytics
    #48799
    (issues:
    #48756,
    #48770)

  • [ML] Prevent fetching multi-field from source
    #48770
    (issue:
    #48756)

Mapping

  • Ensure that field collapsing works with field aliases.
    #50722
    (issues:
    #32648,
    #50121)

  • Improve DateFieldMapper ignore_malformed handling
    #50090
    (issues:
    #46675,
    #50081)

  • Annotated text type should extend TextFieldType
    #49555
    (issue:
    #49289)

  • Ensure parameters are updated when merging flattened mappings.
    #48971
    (issue:
    #48907)

Network

  • Fix TransportMasterNodeAction not Retrying NodeClosedException
    #51325

Percolator

  • Correctly handle MSM for nested disjunctions
    #50669
    (issue:
    #50305)

  • Fix query analyzer logic for mixed conjunctions of terms and ranges
    #49803
    (issue:
    #49684)

Recovery

  • Check allocation id when failing shard on recovery
    #50656
    (issue:
    #50508)

  • Migrate peer recovery from translog to retention lease
    #49448
    (issue:
    #45136)

  • Ignore Lucene index in peer recovery if translog corrupted
    #49114

Reindex

SQL

Search

  • Fix upgrade of custom similarity
    #50851
    (issue:
    #50763)

  • Fix NPE bug inner_hits
    #50709
    (issue:
    #50539)

  • Collect results in a local list when notifying partial results
    #49828
    (issue:
    #49778)

  • Fixes a bug in interval filter serialization
    #49793
    (issue:
    #49519)

  • Correctly handle duplicates in unordered interval matching
    #49775

  • Correct rewritting of script_score query
    #48425
    (issue:
    #48081)

  • Do not throw errors on unknown types in SearchAfterBuilder
    #48147
    (issue:
    #48074)

Security

  • Always consume the body in has privileges
    #50298
    (issue:
    #50288)

Snapshot/Restore

  • Fix Overly Aggressive Request DeDuplication
    #51270
    (issue:
    #51253)

  • Guard Repository#getRepositoryData for exception throw
    #50970

  • Fix Index Deletion During Partial Snapshot Create
    #50234
    (issues:
    #50200,
    #50202)

  • Fix Index Deletion during Snapshot Finalization
    #50202
    (issues:
    #45689,
    #50200)

  • Fix RepoCleanup not Removed on Master-Failover
    #49217

  • Make FsBlobContainer Listing Resilient to Concurrent Modifications
    #49142
    (issue:
    #37581)

  • Fix SnapshotShardStatus Reporting for Failed Shard
    #48556
    (issue:
    #48526)

  • Cleanup Concurrent RepositoryData Loading
    #48329
    (issue:
    #48122)

Transform

  • [Transform] fail to start/put on missing pipeline
    #50701
    (issue:
    #50135)

  • Fix possible audit logging disappearance after rolling upgrade
    #49731
    (issue:
    #49730)

  • [Transform] do not fail checkpoint creation due to global
    checkpoint mismatch
    #48423
    (issue:
    #48379)

Upgrades

Engine

  • Upgrade to Lucene 8.4.0.
    #50518

Infra/Packaging

  • Upgrade the bundled JDK to JDK 13.0.2
    #51511

NOT CLASSIFIED

Aggregations

  • Revert "Begin moving date_histogram to offset rounding (backport of
    #50873) (#50978)"
    #51240

Authentication

  • Change password API authenticating with a bearer token [ISSUE]
    #48752

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions