Skip to content

[Connector API] Support cleaning up sync jobs when deleting a connector#107253

Merged
jedrazb merged 16 commits intoelastic:mainfrom
jedrazb:connector-api-support-deleting-corrsponding-sync-jobs
Apr 11, 2024
Merged

[Connector API] Support cleaning up sync jobs when deleting a connector#107253
jedrazb merged 16 commits intoelastic:mainfrom
jedrazb:connector-api-support-deleting-corrsponding-sync-jobs

Conversation

@jedrazb
Copy link
Copy Markdown
Contributor

@jedrazb jedrazb commented Apr 9, 2024

Changes

  • Support cleaning up associated sync jobs in sync jobs index when deleting a connector, this is controlled by delete_sync_jobs URL param for the delete request
  • This flag is set to true by default, example of how to opt out: DELETE _connector/{id}?delete_sync_jobs=false
  • Added unit tests and yaml tests
  • Since this uses delete_by_query under the hood I needed to link Reindex plugin for unit tests in build.gradle
  • Added small workaround to handle template mappings in unit tests, this allows to defined "simplified" mappings to test functions that rely on particular field type
    • We have several tests that AwaitsFix for the issue linked to missing template mappings - I will follow up in next PR to check if all of the tests are fixed now

@jedrazb jedrazb added >feature :EnterpriseSearch/Application Enterprise Search Team:Enterprise Search Meta label for Enterprise Search team v8.14.0 labels Apr 9, 2024
@jedrazb jedrazb requested a review from timgrein April 9, 2024 09:53
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Hi @jedrazb, I've created a changelog YAML for you.

testImplementation(testArtifact(project(xpackModule('core'))))
testImplementation project(":test:framework")
testImplementation(project(':modules:lang-mustache'))
testImplementation(project(':modules:reindex'))
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to be needed for ESSingleNodeTestCase to register Reindex plugin that exposes delete_by_query action

@jedrazb jedrazb marked this pull request as ready for review April 9, 2024 09:56
@jedrazb jedrazb requested a review from a team as a code owner April 9, 2024 09:56
@elasticsearchmachine
Copy link
Copy Markdown
Collaborator

Pinging @elastic/ent-search-eng (Team:Enterprise Search)

@jedrazb jedrazb requested a review from navarone-feekery April 9, 2024 11:19
Copy link
Copy Markdown
Contributor

@timgrein timgrein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor comments, LGTM

jedrazb and others added 8 commits April 10, 2024 17:17
…/application/connector/ConnectorIndexService.java

Co-authored-by: Tim Grein <tim@4greins.de>
…/application/connector/ConnectorIndexService.java

Co-authored-by: Tim Grein <tim@4greins.de>
…/application/connector/ConnectorIndexService.java

Co-authored-by: Tim Grein <tim@4greins.de>
…/application/connector/syncjob/ConnectorSyncJobIndexService.java

Co-authored-by: Tim Grein <tim@4greins.de>
…/application/connector/syncjob/ConnectorSyncJobIndexService.java

Co-authored-by: Tim Grein <tim@4greins.de>
…/application/connector/ConnectorTestUtils.java

Co-authored-by: Tim Grein <tim@4greins.de>
@jedrazb jedrazb merged commit 07aa9cd into elastic:main Apr 11, 2024
craigtaverner pushed a commit to craigtaverner/elasticsearch that referenced this pull request Apr 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

:EnterpriseSearch/Application Enterprise Search >feature Team:Enterprise Search Meta label for Enterprise Search team v8.14.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants