[SR] Add API for retrieving snapshots.#34598
[SR] Add API for retrieving snapshots.#34598cjcenizal merged 6 commits intoelastic:feature/snapshotsfrom
Conversation
|
Pinging @elastic/es-ui |
💔 Build Failed |
- Add TS types and pass tslint. - Refactor for readability and add comments. - Update tests.
💚 Build Succeeded |
sebelga
left a comment
There was a problem hiding this comment.
Great work @cjcenizal ! I had a few suggestion for Typescript and also one to make the route handler a bit clearer IMO. Let me know what you think! 😊
| ], | ||
| }; | ||
|
|
||
| await expect( |
There was a problem hiding this comment.
I made this comment to Jen's PR, I read much faster this 2 lines for the same result (with the benefit that we can add a breakpoint on the second line and inspect the response if needed).
const response = await getAllHandler(mockRequest, callWithRequest, mockResponseToolkit);
expect(response).toEqual(expectedResponse);| const idToSnapshotMap: any = {}; | ||
| const errors: any = []; | ||
|
|
||
| arraysOfSnapshotsAndErrors.forEach(arrayOfSnapshotsOrError => { |
There was a problem hiding this comment.
arraysOfSnapshotsAndErrors, arrayOfSnapshotsOrError. Normally giving these names means there is a bit of complexity that might be avoided. What do you think of something simpler like this:
const repositoriesByName = await callWithRequest('snapshot.getRepository', {
repository: '_all',
});
const repositoryNames = Object.keys(repositoriesByName);
if (repositoryNames.length === 0) {
return { snapshots: [], errors: [] };
}
const errors: any = [];
const fetchSnapshotsForRepository = async (repositoryName: string) => {
try {
const snapshots = await callWithRequest('cat.snapshots', {
repository: repositoryName,
format: 'json',
});
// Decorate each snapshot with the repository with which it's associated.
return snapshots.map((snapshot: any) => ({
repository: repositoryName,
...snapshot,
}));
} catch (error) {
// These errors are commonly due to a misconfiguration in the repository or plugin errors,
// which can result in a variety of 400, 404, and 500 errors.
errors.push(error);
return null;
}
};
const repositoriesSnapshots = await Promise.all(repositoryNames.map(fetchSnapshotsForRepository));
// Multiple repositories can have identical configurations. This means that the same snapshot
// may be listed as belonging to multiple repositories. A map lets us dedupe the snapshots and
// aggregate the repositories that are associated with each one.
const idToSnapshotMap: Record<string, Snapshot> = repositoriesSnapshots
.filter(Boolean)
.reduce((idToSnapshot, snapshots: any[]) => {
// create an object to store each snapshot and the
// repositories that are associated with it.
snapshots.forEach(({ id, repository, ...rest }) => {
if (!idToSnapshot[id]) {
// Instantiate the snapshot object
idToSnapshot[id] = {
id,
// The cat API only returns a subset of the details returned by the get snapshot API.
summary: { ...rest },
repositories: [],
};
}
idToSnapshot[id].repositories.push(repository);
});
return snapshotsById;
}, {});
return {
snapshots: Object.values(idToSnapshotMap),
errors,
};| repositories: string[]; | ||
| } | ||
|
|
||
| export interface SnapshotSummary { |
There was a problem hiding this comment.
Can we use camelCase in all our client code? I see it much easier if we know in advance that everything is camelCase, regardless of what ES sends.
There was a problem hiding this comment.
Sorry I missed that this was in "common" and was used on the server 😊
|
|
||
| export interface SnapshotSummary { | ||
| status: string; | ||
| start_epoch: string; // e.g. '1554501400'. This and other numerical values are typed as strings. |
There was a problem hiding this comment.
With Typescript we can get the helful doc whenever using the interface. But... it has to be written like this:
export interface SnapshotSummary {
status: string;
/** This and other numerical values are typed as strings. e.g. '1554501400'. */
start_epoch: string;
/** e.g. '21:56:40' */
start_time: string;
end_epoch: string;
/** e.g. '21:56:45' */
end_time: string;
/** Includes unit, e.g. '4.7s' */
duration: string;
indices: string;
successful_shards: string;
failed_shards: string;
total_shards: string;
}…gathering logic. - Create SnapshotSummaryEs and SnapshotDetailsEs types. - Create deserializeSnapshotSummary and deserializeSnapshotDetails utilities. - Return payloads with camel-cased properties.
|
@sebelga Thanks for the great review. I've incorporated your suggestions. Could you take another look? |
💚 Build Succeeded |
| import { deserializeSnapshotDetails, deserializeSnapshotSummary } from '../../lib'; | ||
|
|
||
| import { Snapshot, SnapshotDetails } from '../../../common/types'; | ||
| interface IdToSnapshotMap { |
There was a problem hiding this comment.
My preference is a Record<K, T> for these simple interfaces (actually it removes the need to declare them).
(https://levelup.gitconnected.com/advanced-typescript-types-with-examples-1d144e4eda9e).
But this is just a preference, feel free to leave it.
| repositories: string[]; | ||
| } | ||
|
|
||
| export interface SnapshotSummary { |
There was a problem hiding this comment.
Sorry I missed that this was in "common" and was used on the server 😊
sebelga
left a comment
There was a problem hiding this comment.
LGTM! Thanks for making the changes.
💔 Build Failed |
jen-huang
left a comment
There was a problem hiding this comment.
LGTM! I had a comment regarding (de)serialization, but it's more of a discussion topic rather than anything blocking 🙂
| // may be listed as belonging to multiple repositories. A map lets us dedupe the snapshots and | ||
| // aggregate the repositories that are associated with each one. | ||
| const idToSnapshotMap: Record<string, Snapshot> = repositoriesSnapshots | ||
| .filter(Boolean) |
There was a problem hiding this comment.
Seb's suggestion -- I think it's a neat trick too!
There was a problem hiding this comment.
And if we know in advanced the possible keys, then it woulbe be
const someVar: Record<'AWS' | 'GCP' | 'AZURE', CloudProvider> = ...| */ | ||
|
|
||
| export { booleanizeSettings } from './booleanize_settings'; | ||
| export { deserializeSnapshotSummary, deserializeSnapshotDetails } from './snapshot_serialization'; |
There was a problem hiding this comment.
I left out serialization/deserialization for the repository APIs, partly because there are 7 different repository types, each with different settings, so would result in either complex logic or a pair of methods for each type! but I also found that I haven't gotten much value out of performing deserialization, just that keys are camel cased. how do you feel about it?
There was a problem hiding this comment.
I left out serialization/deserialization for the repository APIs, partly because there are 7 different repository types, each with different settings, so would result in either complex logic or a pair of methods for each type!
I'm not very familiar with the code but that description sounds pretty arduous! As a consumer I think I'd generally prefer consistency with our API responses but I am OK applying de/serialization on a case-by-case basis and skipping it for repositories for now. If consuming snake-cased strings doesn't impede writing and reading the code on the front-end then I don't see a problem with it.
I implemented it here at Seb's request because it was a trivial change to make. Also, as I write this I realize that the SnapshotSummary type excludes the id and repository properties that are on the SnapshotSummaryEs type because there is some data munging going on. So in this case, de/serialization does play a role other than just converting name patterns.
There was a problem hiding this comment.
I understand what you're saying Jen, but having a serialization has several benefits to me:
- we have a single place where we control what is sent to the client from a raw ES resource. This means that if at any moment we have a new requirement and need to do a computation or add a prop (a formatted date for example), we do it there and know that the client will have it (coming from a list of resource, a resource updated or a single resource GET request).
- consistency 😊 I love consistency and it feels strange to have different name cases across apps
- avoid overriding eslint rules each time we'd need to access a deconstructed prop
|
Retest |
💚 Build Succeeded |
* [SR] Snapshot and restore plugin boilerplate (#32276) * Initial plugin set up * Set up client shell * Add initial repository list routes * Fix merge issues and some typings * Decouple server from plugin.ts files, tighten up typings * Use exported constant for required license * Translate plugin name, more typings * Fix more types, move list components under /home * Remove unused var * Change scss prefix * Uncouple unmount logic from routing shim, and some other PR feedback * [SR] Repository list and details UI (#33367) * Initial pass at repositories list UI * Add detail panel for file system repositories, and a generic detail panel with json settings view * Add detail components for other types * Add detail panel footer, rename `useStateValue` to `useAppState` * Fix detail panel footer * Fix unused vars * PR feedback * PR feedback * [SR] Refactor proposal (#33690) * Move app dependencies to its own context provider * Add index.ts barrel file for common types * Move Enums to constants.ts file * Refactor function component using `React.FunctionComponent<Props>` * Refactor service folder structure * Fix type import * Move REPOSITORY_DOC_PATHS from common to public constants * Move AppCore and AppPlugins interfaces back to shim and re-export them from app types * [SR] Create and edit repositories UI (#34020) * Add routing and placeholder form * Fix typings * Set up edit repository route, and basic form UI * Add typings for wrapCustomError, and copy extractCausedByChain from CCR wrapEsError * Throw errors that are already boomified * Create and edit for basic repository types (fs, url, source) * Add repository verification UI to table and details * Create and edit for plugin repository types (hdfs, azure, s3, gcs) * Fix linting * Fix test * Fix test * Remove unused import * Fix duplicate i18n key * Fix details opening on cancel edit, remove unnecessary Fragments, definition file for some EUI components to x-pack, rename saveError * Remove breaks * Adjust add and edit repo routes so they don't conflict with list route * Add repo plugin and types doc links to form * Bootstrap documentation service * Bootstrap text service and replace RepositoryTypeName component with it * Bootstrap breadcrumb service and replace usages * Bootstrap httpService, remove chrome and http from app dependencies(!) * Add request creator and replace all instances of useRequest and sendRequest with it * Fix typo * Simplify update repository and update repository setting methods * Adjust copy * Lint * Remove unused var * Remove unused import * [SR] Add API for retrieving snapshots. (#34598) * [SR] Single and multiple repository delete (#34593) * Add single/multi repository delete API and UI * Address PR feedback * [SR] Add SnapshotTable and SnapshotDetails. (#34837) * Remove associations between multiple repositories with a single snapshot. * Retrieve complete snapshot details in getAllHandler. * Fix cleanup function bug in useRequest hook. * Fix bug in useRequest which prevented old data from being cleared when subsequent requests returned errors. * Add initialValue config option to useRequest. * Add formatDate service to text module. * [SR] Fix linting and add (de)serialization for repositories (#35031) * Fix eslint issues and add (de)serialization for repositories * Add comment about flattening settings * [SR] Surface repository errors and index failures more prominently (#35042) * Add links to repositories from Snapshot Table and Snapshot Details. - Rename services/breadcrumbs to services/navigation and add linkToRepository function. - Refactor home component to update active tab when URL was changed. * Add warning callout to let user know when their repositories contain errors. * Sort failures by shard and add test for snapshot serialization. * Sort failures and indices. * Add filter for filtering snapshots by their repository. * Surface states with humanized text, icons, and tooltips where necessary. * Fix pluralization of seconds. * Surface failures tab even if there are none. - Display a '-' for missing times and durations. - Create DataPlaceholder component. * [SR] Polish repositories UX (#35123) * Refactor repository detail panel to load repository based directly on route param. * Display repository detail panel while table is loading. * Make 'Edit repository' table action a link instead of a button. * Render disabled EuiSelect as a readonly EuiFieldText. * Prepend HDFS URI with hdfs:// protocol. * Present scheme options for Read-Only URL repository as a select. * [SR] Add client-side validation to repository form and link to snapshots from details (#35238) * Add client side repository form validation, extract `flatten` into common lib * Add snapshot count to repository details and link to snapshot list * Reset validation when changing repository type * Fix snapshot list filter deep linking for repository names with slashes and spaces * Fix imports * PR feedback * [SR] Design and copywriting fixes (#35591) * Split repository form into two steps; move `clean_settings.ts` to server * Default to snapshots tab, adjust snapshot empty prompt, add app description * Add minimum timeout to list view requests to avoid flicker, use EuiEmptyPrompt for loading screen, add doc link to settings step * Add information about snapshots to delete repository behavior, add doc link for source only toggle, add size notation help text * Add main doc link * Copywriting and i18n fixes, and add some common settings to third party repo types * Add fields to third party repo detail panel * More copywriting fixes * Use spinner for duration and end time if snapshotting is still in progress * Show all repository type options, mark missing plugins * Revert "Show all repository type options, mark missing plugins" This reverts commit e34ee47. * Fix space * [SR] Add permissions UI and Cloud-specific repository type UI branch (#35833) * Add missing permissions UI and cloud-specific repository type UI branch * Add ES UI as owners of /snapshot_restore directory * Add no repository types callout for Cloud edge case * Redirect invalid section param to repositories * Add warning empty prompt if all repositories have errrors * Replace repository cards with EuiCard * Add snapshot doc link to repository error empty prompt * Remove auto-verification from list and get routes, add separate verification route, add manual verification to repository detail panel * Update copy and remove obsolete test * Remove unused scss files * Final changes to repository cards
* [SR] Snapshot and restore plugin boilerplate (elastic#32276) * Initial plugin set up * Set up client shell * Add initial repository list routes * Fix merge issues and some typings * Decouple server from plugin.ts files, tighten up typings * Use exported constant for required license * Translate plugin name, more typings * Fix more types, move list components under /home * Remove unused var * Change scss prefix * Uncouple unmount logic from routing shim, and some other PR feedback * [SR] Repository list and details UI (elastic#33367) * Initial pass at repositories list UI * Add detail panel for file system repositories, and a generic detail panel with json settings view * Add detail components for other types * Add detail panel footer, rename `useStateValue` to `useAppState` * Fix detail panel footer * Fix unused vars * PR feedback * PR feedback * [SR] Refactor proposal (elastic#33690) * Move app dependencies to its own context provider * Add index.ts barrel file for common types * Move Enums to constants.ts file * Refactor function component using `React.FunctionComponent<Props>` * Refactor service folder structure * Fix type import * Move REPOSITORY_DOC_PATHS from common to public constants * Move AppCore and AppPlugins interfaces back to shim and re-export them from app types * [SR] Create and edit repositories UI (elastic#34020) * Add routing and placeholder form * Fix typings * Set up edit repository route, and basic form UI * Add typings for wrapCustomError, and copy extractCausedByChain from CCR wrapEsError * Throw errors that are already boomified * Create and edit for basic repository types (fs, url, source) * Add repository verification UI to table and details * Create and edit for plugin repository types (hdfs, azure, s3, gcs) * Fix linting * Fix test * Fix test * Remove unused import * Fix duplicate i18n key * Fix details opening on cancel edit, remove unnecessary Fragments, definition file for some EUI components to x-pack, rename saveError * Remove breaks * Adjust add and edit repo routes so they don't conflict with list route * Add repo plugin and types doc links to form * Bootstrap documentation service * Bootstrap text service and replace RepositoryTypeName component with it * Bootstrap breadcrumb service and replace usages * Bootstrap httpService, remove chrome and http from app dependencies(!) * Add request creator and replace all instances of useRequest and sendRequest with it * Fix typo * Simplify update repository and update repository setting methods * Adjust copy * Lint * Remove unused var * Remove unused import * [SR] Add API for retrieving snapshots. (elastic#34598) * [SR] Single and multiple repository delete (elastic#34593) * Add single/multi repository delete API and UI * Address PR feedback * [SR] Add SnapshotTable and SnapshotDetails. (elastic#34837) * Remove associations between multiple repositories with a single snapshot. * Retrieve complete snapshot details in getAllHandler. * Fix cleanup function bug in useRequest hook. * Fix bug in useRequest which prevented old data from being cleared when subsequent requests returned errors. * Add initialValue config option to useRequest. * Add formatDate service to text module. * [SR] Fix linting and add (de)serialization for repositories (elastic#35031) * Fix eslint issues and add (de)serialization for repositories * Add comment about flattening settings * [SR] Surface repository errors and index failures more prominently (elastic#35042) * Add links to repositories from Snapshot Table and Snapshot Details. - Rename services/breadcrumbs to services/navigation and add linkToRepository function. - Refactor home component to update active tab when URL was changed. * Add warning callout to let user know when their repositories contain errors. * Sort failures by shard and add test for snapshot serialization. * Sort failures and indices. * Add filter for filtering snapshots by their repository. * Surface states with humanized text, icons, and tooltips where necessary. * Fix pluralization of seconds. * Surface failures tab even if there are none. - Display a '-' for missing times and durations. - Create DataPlaceholder component. * [SR] Polish repositories UX (elastic#35123) * Refactor repository detail panel to load repository based directly on route param. * Display repository detail panel while table is loading. * Make 'Edit repository' table action a link instead of a button. * Render disabled EuiSelect as a readonly EuiFieldText. * Prepend HDFS URI with hdfs:// protocol. * Present scheme options for Read-Only URL repository as a select. * [SR] Add client-side validation to repository form and link to snapshots from details (elastic#35238) * Add client side repository form validation, extract `flatten` into common lib * Add snapshot count to repository details and link to snapshot list * Reset validation when changing repository type * Fix snapshot list filter deep linking for repository names with slashes and spaces * Fix imports * PR feedback * [SR] Design and copywriting fixes (elastic#35591) * Split repository form into two steps; move `clean_settings.ts` to server * Default to snapshots tab, adjust snapshot empty prompt, add app description * Add minimum timeout to list view requests to avoid flicker, use EuiEmptyPrompt for loading screen, add doc link to settings step * Add information about snapshots to delete repository behavior, add doc link for source only toggle, add size notation help text * Add main doc link * Copywriting and i18n fixes, and add some common settings to third party repo types * Add fields to third party repo detail panel * More copywriting fixes * Use spinner for duration and end time if snapshotting is still in progress * Show all repository type options, mark missing plugins * Revert "Show all repository type options, mark missing plugins" This reverts commit e34ee47. * Fix space * [SR] Add permissions UI and Cloud-specific repository type UI branch (elastic#35833) * Add missing permissions UI and cloud-specific repository type UI branch * Add ES UI as owners of /snapshot_restore directory * Add no repository types callout for Cloud edge case * Redirect invalid section param to repositories * Add warning empty prompt if all repositories have errrors * Replace repository cards with EuiCard * Add snapshot doc link to repository error empty prompt * Remove auto-verification from list and get routes, add separate verification route, add manual verification to repository detail panel * Update copy and remove obsolete test * Remove unused scss files * Final changes to repository cards
* [SR] Snapshot and restore plugin boilerplate (#32276) * Initial plugin set up * Set up client shell * Add initial repository list routes * Fix merge issues and some typings * Decouple server from plugin.ts files, tighten up typings * Use exported constant for required license * Translate plugin name, more typings * Fix more types, move list components under /home * Remove unused var * Change scss prefix * Uncouple unmount logic from routing shim, and some other PR feedback * [SR] Repository list and details UI (#33367) * Initial pass at repositories list UI * Add detail panel for file system repositories, and a generic detail panel with json settings view * Add detail components for other types * Add detail panel footer, rename `useStateValue` to `useAppState` * Fix detail panel footer * Fix unused vars * PR feedback * PR feedback * [SR] Refactor proposal (#33690) * Move app dependencies to its own context provider * Add index.ts barrel file for common types * Move Enums to constants.ts file * Refactor function component using `React.FunctionComponent<Props>` * Refactor service folder structure * Fix type import * Move REPOSITORY_DOC_PATHS from common to public constants * Move AppCore and AppPlugins interfaces back to shim and re-export them from app types * [SR] Create and edit repositories UI (#34020) * Add routing and placeholder form * Fix typings * Set up edit repository route, and basic form UI * Add typings for wrapCustomError, and copy extractCausedByChain from CCR wrapEsError * Throw errors that are already boomified * Create and edit for basic repository types (fs, url, source) * Add repository verification UI to table and details * Create and edit for plugin repository types (hdfs, azure, s3, gcs) * Fix linting * Fix test * Fix test * Remove unused import * Fix duplicate i18n key * Fix details opening on cancel edit, remove unnecessary Fragments, definition file for some EUI components to x-pack, rename saveError * Remove breaks * Adjust add and edit repo routes so they don't conflict with list route * Add repo plugin and types doc links to form * Bootstrap documentation service * Bootstrap text service and replace RepositoryTypeName component with it * Bootstrap breadcrumb service and replace usages * Bootstrap httpService, remove chrome and http from app dependencies(!) * Add request creator and replace all instances of useRequest and sendRequest with it * Fix typo * Simplify update repository and update repository setting methods * Adjust copy * Lint * Remove unused var * Remove unused import * [SR] Add API for retrieving snapshots. (#34598) * [SR] Single and multiple repository delete (#34593) * Add single/multi repository delete API and UI * Address PR feedback * [SR] Add SnapshotTable and SnapshotDetails. (#34837) * Remove associations between multiple repositories with a single snapshot. * Retrieve complete snapshot details in getAllHandler. * Fix cleanup function bug in useRequest hook. * Fix bug in useRequest which prevented old data from being cleared when subsequent requests returned errors. * Add initialValue config option to useRequest. * Add formatDate service to text module. * [SR] Fix linting and add (de)serialization for repositories (#35031) * Fix eslint issues and add (de)serialization for repositories * Add comment about flattening settings * [SR] Surface repository errors and index failures more prominently (#35042) * Add links to repositories from Snapshot Table and Snapshot Details. - Rename services/breadcrumbs to services/navigation and add linkToRepository function. - Refactor home component to update active tab when URL was changed. * Add warning callout to let user know when their repositories contain errors. * Sort failures by shard and add test for snapshot serialization. * Sort failures and indices. * Add filter for filtering snapshots by their repository. * Surface states with humanized text, icons, and tooltips where necessary. * Fix pluralization of seconds. * Surface failures tab even if there are none. - Display a '-' for missing times and durations. - Create DataPlaceholder component. * [SR] Polish repositories UX (#35123) * Refactor repository detail panel to load repository based directly on route param. * Display repository detail panel while table is loading. * Make 'Edit repository' table action a link instead of a button. * Render disabled EuiSelect as a readonly EuiFieldText. * Prepend HDFS URI with hdfs:// protocol. * Present scheme options for Read-Only URL repository as a select. * [SR] Add client-side validation to repository form and link to snapshots from details (#35238) * Add client side repository form validation, extract `flatten` into common lib * Add snapshot count to repository details and link to snapshot list * Reset validation when changing repository type * Fix snapshot list filter deep linking for repository names with slashes and spaces * Fix imports * PR feedback * [SR] Design and copywriting fixes (#35591) * Split repository form into two steps; move `clean_settings.ts` to server * Default to snapshots tab, adjust snapshot empty prompt, add app description * Add minimum timeout to list view requests to avoid flicker, use EuiEmptyPrompt for loading screen, add doc link to settings step * Add information about snapshots to delete repository behavior, add doc link for source only toggle, add size notation help text * Add main doc link * Copywriting and i18n fixes, and add some common settings to third party repo types * Add fields to third party repo detail panel * More copywriting fixes * Use spinner for duration and end time if snapshotting is still in progress * Show all repository type options, mark missing plugins * Revert "Show all repository type options, mark missing plugins" This reverts commit e34ee47. * Fix space * [SR] Add permissions UI and Cloud-specific repository type UI branch (#35833) * Add missing permissions UI and cloud-specific repository type UI branch * Add ES UI as owners of /snapshot_restore directory * Add no repository types callout for Cloud edge case * Redirect invalid section param to repositories * Add warning empty prompt if all repositories have errrors * Replace repository cards with EuiCard * Add snapshot doc link to repository error empty prompt * Remove auto-verification from list and get routes, add separate verification route, add manual verification to repository detail panel * Update copy and remove obsolete test * Remove unused scss files * Final changes to repository cards

Just an API so far. You can test by creating repositories using the steps in the dev issue, and then creating snapshots via the snapshots API.