Disable checking for conflicts when copying saved objects#83575
Disable checking for conflicts when copying saved objects#83575jportner merged 29 commits intoelastic:masterfrom
Conversation
496f14a to
a72f52a
Compare
4e1e791 to
5f1d254
Compare
|
Heads up: due to another PR merging to master recently, there are a significant number of merge conflicts. Since it looks like nobody has started reviewing this yet, I'm going to rebase and force-push here soon to fix these cleanly. |
This is data that is recorded over time as Spaces APIs are used. It is collected with the rest of the Spaces usage data.
Made this change on the client side and the server side.
This is data that is recorded over time as Core APIs are used. It is collected with the rest of the Core usage data.
Made this change on the client side and the server side.
5f1d254 to
ecabf59
Compare
jportner
left a comment
There was a problem hiding this comment.
Re-post of author's notes for reviewers.
I am aware that there are some gaps in testing (we don't have integration tests and functional tests for every permutation with createNewCopies enabled). I opted to update the existing integration tests and functional tests to retain their original assertions. In my opinion, we should keep it this way until we opt to remove pseudo-copy and get rid of the createNewCopies option completely. When we do that, it'll make more sense to update all of the integration tests and functional tests.
| `createNewCopies`:: | ||
| (Optional, boolean) Creates new copies of saved objects, regenerates each object ID, and resets the origin. When used, potential conflict | ||
| errors are avoided. The default value is `true`. | ||
| + | ||
| NOTE: This cannot be used with the `overwrite` option. |
There was a problem hiding this comment.
Somehow I forgot to document this when I added this option 🙈 Added it now!
| const registerTypeMappings = (typeRegistry: SavedObjectTypeRegistry) => { | ||
| typeRegistry.registerType({ | ||
| name: CORE_TELEMETRY_TYPE, | ||
| hidden: true, | ||
| namespaceType: 'agnostic', | ||
| mappings: CoreTelemetryMappings, | ||
| }); | ||
| }; |
There was a problem hiding this comment.
We need to register this type to include its mappings in the .kibana index, but we set up this service before the SavedObjectsService. So I exposed a function here to allow the SavedObjectsService to register this type and retain separation of concerns.
| export interface CoreTelemetry { | ||
| apiCalls?: { | ||
| savedObjectsImport?: { |
There was a problem hiding this comment.
I thought the attributes for this saved object type should be structured in a flexible manner, so in the future we can:
- Record telemetry data for other API calls
- Record telemetry data for something besides API calls
| public async incrementSavedObjectsImport({ | ||
| createNewCopies, | ||
| overwrite, | ||
| }: IncrementSavedObjectsImportOptions) { | ||
| const coreTelemetry = await this.getTelemetryData(); |
There was a problem hiding this comment.
Each of these telemetry client functions has three steps:
- Retrieve current telemetry data (or an empty attributes object, if no current telemetry data exists)
- Update the attributes, incrementing whatever counter field(s) are necessary
- Persist telemetry data to Elasticsearch
There is a race condition if two users call the same API at the same time (either in a single Kibana server, or separate Kibana servers in a HA configuration). However, the impact is minimal, in that edge case we'll simply lose one of the increments. I think that's an acceptable risk.
The alternative is using the document version to ensure the saved object hasn't changed before we update it, and in that scenario, calculate new attributes so we can retry. Seems like overkill to me, though.
src/core/server/index.ts
Outdated
| CoreConfigUsageData, | ||
| CoreEnvironmentUsageData, | ||
| CoreServicesUsageData, | ||
| CoreTelemetry, |
There was a problem hiding this comment.
Since CoreUsageData extends CoreTelemetry now, we have to export CoreTelemetry here. However, the CoreTelemetryService is only intended to be used by Core code, so we don't have to export anything else at the src/core/server level.
Edit: see #83575 (comment), I renamed CoreTelemetry to CoreUsageStats.
src/core/server/server.ts
Outdated
| const coreTelemetrySetup = this.coreTelemetry.setup({ | ||
| savedObjectsStartPromise: this.savedObjectsStartPromise, | ||
| }); |
There was a problem hiding this comment.
Here is the setup/start process:
CoreTelemetryServiceCoreUsageStatsService setup (depends on SavedObjectsService start)- SavedObjectsService setup (depends on
CoreTelemetryServiceCoreUsageStatsService setup) - CoreUsageDataService setup (depends on
CoreTelemetryServiceCoreUsageStatsService setup) - SavedObjectsService start
- CoreUsageDataService start, immediately fetching usage data and querying for any persisted
telemetry datausage stats in the process
The CoreTelemetryService CoreUsageStatsService needs a SavedObjectsRepository so it can persist telemetry data to Elasticsearch. I spoke with @pgayvallet and we determined the best way to handle this messy circular dependency is to pass a promise to the CoreTelemetryService CoreUsageStatsService which resolves after the SavedObjectsService is started.
Edit: changed type names based on PR review feedback.
There was a problem hiding this comment.
nit: savedObjectsStartPromiseResolver is just a function, rather than an instance of a class, so I would make that clearer with the naming. e.g. resolveSavedObjectsStartPromise
| idSelected={overwrite ? overwriteEnabled.id : overwriteDisabled.id} | ||
| onChange={(id: string) => onChange({ overwrite: id === overwriteEnabled.id })} | ||
| disabled={createNewCopies} | ||
| disabled={createNewCopies && !isLegacyFile} |
There was a problem hiding this comment.
This fixes a bug that I didn't run into until I changed createNewCopies to be enabled by default on the client side.
There was a problem hiding this comment.
Looks great, thanks for all the pointers in your description. ❤️
Verified functionality and that telemetry data is sent through correctly.
Added one suggestion regarding the internal API but feel free to ignore if it's not possible (you've got more context) or if I'm overlooking something.
Appreciate this isn't really part of this PR but maybe we could improve the UI a bit while working on it:
- Now that "create new objects with random IDs" is selected by default I would make that the first option so that it's read first when users scan the form
- Labels should be the same font size for all fields / field sets ("Select spaces / file" labels are currently much smaller than "copy / import options") even though those are the main action
- "Select spaces" should be the first field as that's the main action the user is trying to achieve and the only one they really need to interact with in "copy to space" flyover. Ideally I would also hide the filter options input unless there are more than 5 spaces or so (I don't think the majority of users have that many spaces and it's distracting more than anything when there's only a few spaces in the list)
Import flyover
Copy to space flyover
src/core/server/server.ts
Outdated
| const coreTelemetrySetup = this.coreTelemetry.setup({ | ||
| savedObjectsStartPromise: this.savedObjectsStartPromise, | ||
| }); |
There was a problem hiding this comment.
nit: savedObjectsStartPromiseResolver is just a function, rather than an instance of a class, so I would make that clearer with the naming. e.g. resolveSavedObjectsStartPromise
| }, | ||
| }; | ||
| await this.updateTelemetryData(attributes); | ||
| } |
There was a problem hiding this comment.
I feel like this class is doing the exact same thing as the CoreTelemetryClient except that it's tailored to the Spaces use case.
Maybe instead of having many of these clients, each one implementing basically the same logic, we could just have one telemetry service in core that every plugin can use to add telemetry data.
As far as I can tell the logic is the following when we want to record usage data:
- Register saved object type mappings for each use case / plugin
- Retrieve saved object
- Increment count (a bit like a redux reducer)
- Update saved object
Something like this could encapsulate this logic:
// Plugin setup
const update = core.telemetryService.register(name, mappings);
// Route handler
await update((currentAttributes = defaultAttributes) => nextAttributes);e.g. for Spaces plugin:
// Plugin setup
const updateSpacesUsage = core.telemetryService.register('spaces-telemetry', mappings);
// Route handler
await updateSpacesUsage((current = CREATE_NEW_COPIES_DEFAULT) => ({
total: current.total + 1,
createNewCopies: incrementBooleanCount(current.createNewCopies, createNewCopies),
overwrite: incrementBooleanCount(current.overwrite, overwrite),
}))Just thinking out loud here, but this would make it a lot easier to add additional telemetry events. The only thing you'd need to implement and test as a plugin developer would be the reducer function.
There was a problem hiding this comment.
I like the idea and it sounds elegant, and I agree this is essentially duplicated code. However, I know we are trying hard to keep the core API surface as small as possible, and I'm not sure we want to add a whole service for something that we don't yet have additional use cases for. I'll defer to @elastic/kibana-core since they would be the owners of said service.
Thanks!
I'm glad you mentioned it. I thought about this but I was hesitant to move around the form layout as I thought that might bring more confusion for existing users. @mdefazio do you have any opinion either way?
Can't change the label size for EuiFormRow ("Select spaces" / "Select a file to import"). We could change the other sub-header labels to be smaller, but that might look weird? Again I'll ask @mdefazio to weigh in.
I have the same concern regarding changing the form layout. @mdefazio |
|
@afharo Thanks for the ping! @jportner Can you open a Telemetry Mapping update issue after this is merged, so we can reflect these changes in the Telemetry production cluster? |
Will do! |
|
@elasticmachine merge upstream |
| }, | ||
| }, | ||
| }; | ||
| await this.updateUsageStats(attributes); |
There was a problem hiding this comment.
getUsageStats() silently swallows any exceptions and returns an empty object. So if there's an exception in getUsageStats we will reset all counters to 0. If we can't get the existing stats it would be better not to do an update operation (thereby losing one increment instead of all previous counters).
This behaviour is present in the other methods too (and the Core service), just leaving one comment for all of them.
There was a problem hiding this comment.
Good point, sounds like this will be resolved when using incrementCounter per #83575 (comment)
x-pack/plugins/spaces/server/routes/api/external/copy_to_space.ts
Outdated
Show resolved
Hide resolved
pgayvallet
left a comment
There was a problem hiding this comment.
Only reviewed owned code
| export interface CoreUsageStats { | ||
| apiCalls?: { | ||
| savedObjectsImport?: { | ||
| total: number; | ||
| kibanaRequest: { | ||
| yes: number; | ||
| no: number; | ||
| }; |
There was a problem hiding this comment.
Question: is this yes/no format required (or here for consistency)? As total = yes + no, I feel like one of these fields may be redundant?
src/core/server/saved_objects/routes/integration_tests/export.test.ts
Outdated
Show resolved
Hide resolved
pgayvallet
left a comment
There was a problem hiding this comment.
LGTM for my part of the review. Thanks for the changes 😄
|
Somehow I can't comment on the thread...
Don't have a lot of experience with this, but I think this could make visualising the data a bit more challenging since Kibana would have to calculate one of the metrics from the other two numbers. It would be possible to use a scripted field to do this kind of math (or something in the telemetry service that changes the data on ingest) but I would err towards duplicating this data. |
The way these were declared in Typescript caused the generated telemetry schema JSON to be malformed.
rudolf
left a comment
There was a problem hiding this comment.
Added some nits, but otherwise 👍
src/core/server/core_usage_data/core_usage_stats_client.test.ts
Outdated
Show resolved
Hide resolved
src/core/server/core_usage_data/core_usage_stats_client.test.ts
Outdated
Show resolved
Hide resolved
src/core/server/core_usage_data/core_usage_stats_client.test.ts
Outdated
Show resolved
Hide resolved
Co-authored-by: Rudolf Meijering <skaapgif@gmail.com>
thomheymann
left a comment
There was a problem hiding this comment.
LGTM!
Nice one on the UX improvements! ❤️
💚 Build SucceededMetrics [docs]Async chunks
Distributable file count
Page load bundle
History
To update your PR or re-run it, just comment with: |


Resolves #81907 and #82324
Overview
This PR primarily makes two changes:
Import andCopy operations now don't check for conflicts by default (changedcreateNewCopiesoption to be enabled by default)createNewCopies.The changes are split into four commits: two for Import (includes changes to Core), and two for Copy (includes changes to Spaces plugin).
Core and Spaces each now have a telemetry service and telemetry client which are only intended to be used internally, not exposed to external consumers. Whenever an external API call is made, the route handler increments a counter in using the telemetry client -- this is persisted to a "telemetry data" saved object. Whenever the usage collector runs periodically, it queries this telemetry data in addition to its normal behavior which checks config/etc.
Testing:
yarn kbn clean && yarn kbn bootstraptsc -b --clean tsconfig.refs.json && yarn kbn bootstrap, this is required because of changes to the usage collector package.Docs preview: https://kibana_83575.docs-preview.app.elstc.co/diff
Dev Docs
When copying saved objects, conflict checking is now disabled by default. See the
createNewCopiesparameter in the Copy saved objects to space API documentation for more information.