Skip to content

feat: Try these#215

Merged
kim-em merged 62 commits intoleanprover-community:mainfrom
thorimur:try-these
Oct 24, 2023
Merged

feat: Try these#215
kim-em merged 62 commits intoleanprover-community:mainfrom
thorimur:try-these

Conversation

@thorimur
Copy link
Copy Markdown
Contributor

@thorimur thorimur commented Aug 10, 2023

This PR introduces Try these: functionality to the tryThis widget and associated definitions, such as addSuggestions, which renders an array of replacement suggestions as a bulleted list in the infoview preceded by "Try these:".
An image of a bulleted list of tactic replacements in the infoview.

Currently, tactics in Mathlib which return many different "Try this:" suggestions, such as apply? and rw?, work by creating a separate "Try this:" widget for each individual suggestion. This takes up a lot of space in the infoview—each suggestion includes its own "Tactic replacement" summary element and the words Try this:.

This was discussed briefly and requested as a std4 PR on zulip.

This PR also provides both widgets with configurable headers—the text Try these: and Try this: can now be modified at will. (This is motivated by upcoming tactics like hint, which may want different lists for different things, e.g. a list of tactics which close the goal, and a list of tactics which don't.)

This PR also modifies the form of the code actions. Currently, tryThis code actions all read simply "Apply 'Try this'". We change this to the more informative "Try this: {replacementText}". This is necessary in order to have distinguishable code actions now that we're handling a list of different possible replacements instead of just one.

Finally, this PR also allows for a bare String (instead of TSyntax) to be used as a suggested replacement via SuggestionText, and provides 6 basic styling options for each entry via SuggestionStyle: .error, .warning, .success; .asHypothesis, .asInaccessible; and .value (t : Float), which draws from a red-yellow-green color gradient with red at 0.0 and green at 1.0.


I'm not 100% sure these are the "right" HTML classes to be using to style the text in a code-like fashion, even though they look alright so far.

@kim-em kim-em added the awaiting-review This PR is ready for review; the author thinks it is ready to be merged. label Aug 11, 2023
@kim-em
Copy link
Copy Markdown
Collaborator

kim-em commented Aug 18, 2023

The LLMStep project has implemented something very similar, see https://github.com/wellecks/llmstep/blob/master/LLMstep/LLMstep.lean#L94 and also my PR refactoring it a bit wellecks/llmstep#4.

Can we support their use case?

They are doing something a bit extra, which is actually running the tactics against the goal and putting in 🎉 emojis where appropriate.

They also need to cope with the case of "bad suggestions" for which they only have Strings that can't be parsed into Syntax.

@thorimur
Copy link
Copy Markdown
Contributor Author

thorimur commented Aug 19, 2023

@semorrison I think so! There are two additional pieces of functionality I see that we'd need to supply here:

  • ability to customize the classname/highlight of the clickable link (to some extent; iirc there are specific highlight colors allowed and a type somewhere encoding them)
  • ability to prepend text (e.g. 🎉)

If these both become fields of the Suggestion structure, then the running of tactics against the goal (and sorting, etc.) can all be done before feeding the array of Suggestions to addSuggestions. I think that might be a good pattern in general: put all of the customization data in the Suggestions, then feed them to addSuggestion(s), which only pretty-prints and prepares props for the infoview (and nothing more).

Then, there are two ways I can think of for modifying the setup to handle the "bad suggestions".

  1. I think there's a case to be made that we should only ever insert valid Syntax, and thus only Syntax should be clickable. (But maybe I'm not anticipating some cases...) (Is there a reason they ought to be clickable in the LLMStep case?) This would mean that the bad suggestion strings should be relegated to the info field(s) when constructing the Suggestions. We'd abstain from displaying syntax by either:
    a. making the suggestion field Option al
    b. asking the LLMStep project to take the suggestion field to be ⟨Syntax.missing⟩, and special-casing missing syntax to simply not display (instead of displaying as <missing>)
  2. Have the suggestion field be upgraded fromTSyntax kind to SuggestionText or SuggestionText kind, which would more or less just be a descriptive version of TSyntax kind ⊕ String (with appropriate coercions from the two ponents, each of which would be handled appropriately, etc.). kind could either be a parameter to SuggestionText or an implicit argument to one of the constructors of SuggestionText (in which case it wouldn't need to be a parameter to Suggestion any longer, and suggestion arrays could be heterogeneously in their syntax kinds). (I tried the latter out just to make sure it's viable and it seems so.)

I'll think about this a bit, but if you have an opinion on which of these options would be the best way to handle the invalid suggestions (or favor a different option), please let me know! :)

@kim-em
Copy link
Copy Markdown
Collaborator

kim-em commented Aug 20, 2023

The reason there are non-parseable suggestions in the llmstep example is that the suggestions are coming from a language model, which sometimes has no idea what it is doing! :-) Nevertheless, the thinking is that sometimes there might be the germ of a good idea, which the human could still make use of. So for their application, they do want to be able to show plain text suggestions (whether or not they are clickable is probably less important: users will use these ones less often, and perhaps can be expected to copy and paste).

The wrapper for TSyntax kind ⊕ String seems the right way to go.

And yes, absolutely addSuggestions itself should not be doing any of the preparation of the Suggestion, just displaying it.

@thorimur thorimur added WIP work in progress and removed awaiting-review This PR is ready for review; the author thinks it is ready to be merged. labels Aug 20, 2023
* replaces `info` field
* used in place of `TSyntax`
* `Suggestion` loses its `kind` parameter
* instead of "Tactic replacement"("s")
* reflects fact that we might not be suggesting a tactic
* `style` can be used to specify replacement text style
* this handles e.g. recolors correctly on mouseover
* appending `red` to the class behaved incorrectly on mouseover
* `className?` simply overrides the attr instead of appending
* no need to hem the user in; provides greater flexibility
* at the cost of duplicate code
@thorimur thorimur removed the WIP work in progress label Aug 22, 2023
@thorimur
Copy link
Copy Markdown
Contributor Author

(Apologies for the force push, accidentally pushed some clutter.)

@kim-em
Copy link
Copy Markdown
Collaborator

kim-em commented Sep 21, 2023

@digama0, what do you think of this PR at this point? It would be nice to get this in so that both LeanInfer and LLMStep can adopt it, rather than having a version in each repository.

I'd also like it to restore the hint tactic.

@ghost ghost added the merge-conflict This PR has merge conflicts with the `main` branch which must be resolved by the author. label Oct 18, 2023
@ghost ghost removed the merge-conflict This PR has merge conflicts with the `main` branch which must be resolved by the author. label Oct 23, 2023
Copy link
Copy Markdown
Contributor

@joehendrix joehendrix left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. Thanks for all the comments.

Copy link
Copy Markdown
Member

@digama0 digama0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for not giving this a proper review for so long. I've made several changes, most notably to remove all of the getStyle logic and just let users pass in a json object with the style they want (and retaining the existing styles, as presets). Anyway it is now ready to merge IMO, but I will give @joehendrix @thorimur @semorrison time to speak up since I made some significant modifications.

@kim-em
Copy link
Copy Markdown
Collaborator

kim-em commented Oct 24, 2023

For ease of reviewing, Mario's changes are:

@thorimur
Copy link
Copy Markdown
Contributor Author

@digama0 I quite like the refactor you've made to the style logic! I was also wary of all that JS and reluctant to constrain the user to my choices. This is much better. (And I don't think we had the json% elaborator in std when I started, so that's nice too!) The default style setting is a sensible addition as well.

I went through the code and only found an incorrect test description from myself, which I fixed. LGTM. :)

@kim-em kim-em merged commit fb56324 into leanprover-community:main Oct 24, 2023
@digama0
Copy link
Copy Markdown
Member

digama0 commented Oct 24, 2023

(And I don't think we had the json% elaborator in std when I started, so that's nice too!)

There's a reason it was merged just a few hours ago...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awaiting-review This PR is ready for review; the author thinks it is ready to be merged.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants