Skip to content

Fix for ES7 : Specifying types is deprecated.#1997

Merged
abitmore merged 20 commits intodevelopfrom
es7_fix
Dec 22, 2021
Merged

Fix for ES7 : Specifying types is deprecated.#1997
abitmore merged 20 commits intodevelopfrom
es7_fix

Conversation

@oxarbitrage
Copy link
Member

@oxarbitrage oxarbitrage commented Sep 20, 2019 •
edited by abitmore
Loading

Note: this PR contains breaking changes.

With this PR, the node attempts to obtain the version of ES at startup, and if it fails (for example, due to lack of permissions), the ES server is assumed to be version 7 or higher.

Before this change, in ES, we use the path pattern bitshares-*/data/2.9.x, where data is the type. After this change, we stop specifying the type if ES is version 7 or higher, and the path pattern becomes bitshares-*/2.9.x for newly auto-created indexes (Note: bitshares-*/_doc/2.9.x is usable with ES7 but not ES8, see #2707 for more info). Of course, the existing data will not change by itself. The same applies to the objects-* indexes.

  • ES service maintainers need to manually delete all existing indexes from their ES databases and regenerate all data with the new code, or use a new prefix other than the default one (bitshares-), or install a new ES database. This may take days or weeks to complete. If the old data is not deleted, it uses around 500G of disk space.
    By the way, we need more disk space to store other new data due to Store some data in ES as objects instead of just as strings #2565 and etc.
  • Client libraries and applications need to query data through the new path pattern and, if applicable, a new prefix.

Those who are interested, please check #2240 (comment) for more info.

The original OP is below.

Reported in telegram by @HarukaMa . Elasticsearch plugin is not working in ES7 as _type is deprecated. However this needs to be there for ES6(and probably lower versions).

This pull send a query to the server, get the version string and send the type accordingly in both plugins(elasticsearch and es_objects)

@abitmore
Copy link
Member

abitmore commented Sep 20, 2019 •
edited
Loading

I'm not sure whether this PR fixed a problem, but my node is running ES7 fine with default config.ini.

My ES node was installed following this guide: https://www.elastic.co/guide/en/elasticsearch/reference/7.3/deb.html

The issue @HarukaMa reported was an error when elasticsearch-mode parameter is set to 2 (perhaps not related to this PR):

# Mode of operation: only_save(0), only_query(1), all(2) - Default: 0
# elasticsearch-mode =

@oxarbitrage
Copy link
Member Author

oxarbitrage commented Sep 20, 2019

He later reported the issue was happening for him also in only_save(0) iirc.

@oxarbitrage
Copy link
Member Author

oxarbitrage commented Sep 20, 2019

In conversations in the dev channel yesterday he fixed by removing the _type from the queries just as we are doing here except that we need to keep functionality for ES6. I am positive that it will not work with types removed on ES6.
Not sure why your ES7 is working without this change while his not. A different minor version ?

Will try ES7 myself.

oxarbitrage
oxarbitrage commented Sep 20, 2019
View reviewed changes
@oxarbitrage
Copy link
Member Author

oxarbitrage commented Sep 21, 2019

I found the details of the error @HarukaMa was getting:

Rejecting mapping update to [bitshares-2015-10] as the final mapping would have more than 1 type: [_doc, data]

Version string is: 7.3.1

@abitmore can you post your version? curl -XGET http://localhost:9200/ will do it.

@abitmore
Copy link
Member

abitmore commented Sep 22, 2019

My node is 7.3.0.

@HarukaMa
Copy link
Contributor

HarukaMa commented Sep 22, 2019 via email

@abitmore abitmore added this to the 4.1.0 - Feature Release milestone Sep 29, 2019
@oxarbitrage
Copy link
Member Author

oxarbitrage commented Oct 13, 2019

This by now is not needed. i downloaded and tested elasticsearch 7.4.0 (last stable version) without any issue. The pull can be handy in the future but not needed now.

@abitmore abitmore removed this from the 4.1.0 - Feature Release milestone Oct 13, 2019
@abitmore abitmore mentioned this pull request Aug 25, 2020
Closed
17 tasks
@abitmore abitmore reopened this Dec 18, 2021
@abitmore
Copy link
Member

abitmore commented Dec 18, 2021

This is still needed. In ES8 the warning will become an error.

@abitmore abitmore self-assigned this Dec 18, 2021
@abitmore abitmore added this to the 6.1.0 - Feature Release milestone Dec 18, 2021
@abitmore
Copy link
Member

abitmore commented Dec 18, 2021

Those who are interested, please check #2240 (comment) for more info.

@abitmore abitmore force-pushed the es7_fix branch 2 times, most recently from 8b43b64 to ae067f0 Compare December 20, 2021 17:00
@abitmore abitmore linked an issue Dec 21, 2021 that may be closed by this pull request
ES plugins: Specifying types in bulk requests is deprecated. #2240
Closed
17 tasks
@sonarqubecloud
Copy link

sonarqubecloud bot commented Dec 21, 2021

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 9 Code Smells

67.6% 67.6% Coverage
0.0% 0.0% Duplication

@abitmore abitmore changed the title Fix for ES7 Fix for ES7 : Specifying types is deprecated. Dec 22, 2021
@abitmore abitmore merged commit 577fc9e into develop Dec 22, 2021
@abitmore abitmore deleted the es7_fix branch December 22, 2021 17:04
This was referenced Dec 22, 2021
Open
Closed
Open
@abitmore abitmore mentioned this pull request Nov 3, 2022
Closed
@abitmore abitmore mentioned this pull request Jan 7, 2023
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@abitmore abitmore

Labels

None yet

Projects

None yet

Milestone

6.1.0 - Feature Release

Development

Successfully merging this pull request may close these issues.

ES plugins: Specifying types in bulk requests is deprecated.

3 participants

@oxarbitrage @abitmore @HarukaMa