This change is 

Hello! 😄 Could I get help with understanding the endpoint + requirements?

Specifically, #75556 adds an execution_failures field to the API, which contains start and end times in addition to the error message and allows us to populate the three UI table columns. However, this execution_failures field only contains retriable execution errors (since it is populated by the log of retriable execution failures), and does not include errors that are not retriable.

For development purposes, I have managed to cause a job to fail by adding an error in the normal job flow. But it does not populate the execution_failures field, because the error I introduced is not retriable. I haven't figured out how to induce a retriable failure, and am thus not sure how the execution_failures field and the separate error field (string message only, no start or end times) would be in populated in a real use case.

Lacking the example of a real use case, I have some starting questions:

1. @ajwerner, I'm assuming this is intentional, to only populate execution_failures with retriable errors?
2. @kevin-v-ngo / @Annebirzin: I assume that we want to show both retriable, and non-retriable errors in the UI table? Or am I mistaken, and we only want to show those that are retriable? Currently, if a job fails in a non-retriable way, it already shows the error message under the "FAILED" badge (this message existed before this PR). Should we repeat that in the table, or just show it once in the table/under the badge?

Assuming I'm correct on the above two assumptions, my further questions are:
3) @ajwerner, do you have an idea how one could induce a retriable error? E.g. via temporarily compiling errors somewhere in a normal job flow, for UI dev purposes. If we can't simulate this, my question is then:
- When a retriable error occurs, is the latest error populated into the error (string) column of crdb_internal.jobs? Assuming we should show both retriable and non-retriable errors in the UI table, I'm trying to understand if I should concatenate the error (string) field with errors in the execution_failures field, or if that would cause duplicates if execution_failures is populated.
4) @kevin-v-ngo / @ajwerner, for non-retriable errors, what should the start and end time columnns be? I could make them the job start and end time, though since the job is the first thing to start (before any retriable errors), the "final" error will be the bottommost one if we are sorting by most recent error start time as default. Or should they be empty?
5) @kevin-v-ngo / @Annebirzin, on that note: should the table default to sorting by most recent error start time, or something else?

Regardless of the retriable/non-retriable errors stuff,
6) @kevin-v-ngo / @ajwerner, do we have to worry about the error messages being too long to reasonably contain in the table cell? E.g., the dump of some JSON payload.
7) @kevin-v-ngo / @ajwerner, would we potentially expect more than, e.g., 20 errors in a real use case, and if so how many? If we're expecting many errors for a single job, @Annebirzin I'm wondering if we should paginate the table.
8) @kevin-v-ngo / @ajwerner, should we link to any resources to help with the error messages?

Thanks, everyone!

Hello! 😄 Could I get help with understanding the endpoint + requirements?

Specifically, #75556 adds an execution_failures field to the API, which contains start and end times in addition to the error message and allows us to populate the three UI table columns. However, this execution_failures field only contains retriable execution errors (since it is populated by the log of retriable execution failures), and does not include errors that are not retriable.

For development purposes, I have managed to cause a job to fail by adding an error in the normal job flow. But it does not populate the execution_failures field, because the error I introduced is not retriable. I haven't figured out how to induce a retriable failure, and am thus not sure how the execution_failures field and the separate error field (string message only, no start or end times) would be in populated in a real use case.

Lacking the example of a real use case, I have some starting questions:

1. @ajwerner, I'm assuming this is intentional, to only populate execution_failures with retriable errors?
2. @kevin-v-ngo / @Annebirzin: I assume that we want to show both retriable, and non-retriable errors in the UI table? Or am I mistaken, and we only want to show those that are retriable? Currently, if a job fails in a non-retriable way, it already shows the error message under the "FAILED" badge (this message existed before this PR). Should we repeat that in the table, or just show it once in the table/under the badge?

Assuming I'm correct on the above two assumptions, my further questions are: 3) @ajwerner, do you have an idea how one could induce a retriable error? E.g. via temporarily compiling errors somewhere in a normal job flow, for UI dev purposes. If we can't simulate this, my question is then: - When a retriable error occurs, is the latest error populated into the error (string) column of crdb_internal.jobs? Assuming we should show both retriable and non-retriable errors in the UI table, I'm trying to understand if I should concatenate the error (string) field with errors in the execution_failures field, or if that would cause duplicates if execution_failures is populated. 4) @kevin-v-ngo / @ajwerner, for non-retriable errors, what should the start and end time columnns be? I could make them the job start and end time, though since the job is the first thing to start (before any retriable errors), the "final" error will be the bottommost one if we are sorting by most recent error start time as default. Or should they be empty? 5) @kevin-v-ngo / @Annebirzin, on that note: should the table default to sorting by most recent error start time, or something else?

Regardless of the retriable/non-retriable errors stuff, 6) @kevin-v-ngo / @ajwerner, do we have to worry about the error messages being too long to reasonably contain in the table cell? E.g., the dump of some JSON payload. 7) @kevin-v-ngo / @ajwerner, would we potentially expect more than, e.g., 20 errors in a real use case, and if so how many? If we're expecting many errors for a single job, @Annebirzin I'm wondering if we should paginate the table. 8) @kevin-v-ngo / @ajwerner, should we link to any resources to help with the error messages?

Thanks, everyone!

2. I'd suggest we show all retriable errors that the job experienced in the UI table below. The non-retriable error that caused the jobs to fail would be part of the status section of the job.

3. With 2, we'd have the non-retriable error as the status. Ideally it should have the time the job failed and experienced that error there as well.

4. Sort by most recent error in the UI table sounds good to me

5. I'm not sure how long error messages can be. Each job type would probably have its own set of error messages but it would be great if we can search for the longest in the code base. If it's not feasible, perhaps we should have fixed width and truncate as we have with other tables in the UI. On hover would show the full error message. Thoughts @Annebirzin? I remember doing this for the new statements/transaction format.

7. I'm not completely sure the max number of retriable errors we'd see. I'd suspect it's not many to warrant pagination but adding @vy-ton for schema and statistics and @amruss for CDC for their feedback.

8. Short answer not yet. Each job type would probably have its own set of error messages that would need descriptions and how to mitigate/prevent. We could probably have something similar to this doc. @amruss, is there a Jobs docs writer or board we can file a tracking docs improvement here?

Thanks, @kevin-v-ngo!

I've gotten answers to 5 and 7 from Andrew:
6. Error messages are capped at 16k (i.e., they could be very long)
7. Default to retaining only 3, controlled by this cluster setting.

Additionally, Andrew said that

• execution_failures was really added to try to make it clear what’s going on in the face of exponential backoff.
• Unclear whether the database duplicates the final retriable error message into the top-level error field. I filed, and am blocked on Jobs: Add a way to inject retriable and non-retriable errors #81474 to find this out

From Kevin, regarding error message length:

If it's not feasible, perhaps we should have fixed width and truncate as we have with other tables in the UI. On hover would show the full error message. Thoughts @Annebirzin? I remember doing this for the new statements/transaction format.

@Annebirzin, just bringing up that another option could be to fix cell size, and scroll the contents.

Additionally, @Annebirzin, regarding Kevin's comment that:

I'd suggest we show all retriable errors that the job experienced in the UI table below. The non-retriable error that caused the jobs to fail would be part of the status section of the job.

We discussed off-thread, and it sounds like we want to convey the difference between the "final" non-retriable error in the status, and the retriable errors in the job table. A suggestion was to name the job errors table "Previous Job errors" - this table might be present while the job is running, reverting, or moved on to some other state e.g. succeeded/failed. I've updated the screenshot in the description with these text updates.

@jocrl

• For long error messages, I wonder if we can introduce a pattern to truncate after 2 lines, and then include a 'Show More/Less' action to vertically expand or hide the additional text in the cell (designs here)
  
  

• For the title and empty state text, I'm not sure adding the word 'Previous' really tells the user much. I would suggest leaving as 'Job Errors' and maybe add some description text below the title to convey that only retriable errors will show in the table. (designs here) I'm open to suggestions on this description copy as I'm not sure I captured the distinction between the types of job errors that will show up here cc @kevin-v-ngo

Hi @amruss! Could we get your help on how to convey that the errors table only contains retriable errors, and not non-retriable ones? E.g., whether that's a title for the table, or some description text.

It looks like #82562 could unblock this. See #81474 (comment).