Add support fuzzy matching a list of dictionaries using either a key or a callback function by yegappan · Pull Request #6947 · vim/vim

yegappan · 2020-09-13T05:42:36Z

No description provided.

codecov · 2020-09-13T06:06:40Z

Codecov Report

Merging #6947 into master will decrease coverage by 0.20%.
The diff coverage is 94.77%.

@@            Coverage Diff             @@
##           master    #6947      +/-   ##
==========================================
- Coverage   88.58%   88.38%   -0.21%     
==========================================
  Files         148      144       -4     
  Lines      161528   158442    -3086     
==========================================
- Hits       143096   140040    -3056     
+ Misses      18432    18402      -30

Impacted Files	Coverage Δ
src/evalfunc.c	`96.27% <ø> (+0.01%)`	⬆️
src/search.c	`91.27% <94.77%> (-0.60%)`	⬇️
src/json.c	`90.95% <0.00%> (-1.30%)`	⬇️
src/mbyte.c	`79.29% <0.00%> (-1.29%)`	⬇️
src/gui.c	`62.27% <0.00%> (-1.19%)`	⬇️
src/help.c	`90.05% <0.00%> (-0.91%)`	⬇️
src/textobject.c	`91.69% <0.00%> (-0.71%)`	⬇️
src/gui_gtk_x11.c	`58.28% <0.00%> (-0.71%)`	⬇️
src/message.c	`87.84% <0.00%> (-0.66%)`	⬇️
src/if_python.c	`82.65% <0.00%> (-0.62%)`	⬇️
... and 127 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 10e4f12...0e1bee5. Read the comment docs.

runtime/doc/eval.txt

-		based on the matching score. {str} is treated as a literal
-		string and regular expression matching is NOT supported.
-		The maximum supported {str} length is 256.
+matchfuzzy({list}, {str} [, {key}])


prabirshrestha · 2020-09-15T17:39:07Z

src/testdir/test_functions.vim

+  call assert_fails("let x = matchfuzzy(l, 'foo', {'key' : 'name'})", 'E730:')
+
+  " Test for the fuzzymatchpos() function
+  call assert_equal([['curl', 'world'], [[2,3], [2,3]]], matchfuzzypos(['world', 'curl'], 'rl'))


missing tests for key and text_cb for matchfuzzypost

prabirshrestha · 2020-09-15T17:40:35Z

src/testdir/test_functions.vim

+  call assert_fails("let x = matchfuzzy(l, 'foo', {'key' : 'name'})", 'E730:')
+
+  " Test for the fuzzymatchpos() function
+  call assert_equal([['curl', 'world'], [[2,3], [2,3]]], matchfuzzypos(['world', 'curl'], 'rl'))


can we also have a test case for multiple matches in the same item.

matchfuzzypos(['hello world hello world', 'hello', 'world]`, 'hello')

prabirshrestha · 2020-09-15T17:42:12Z

src/testdir/test_functions.vim

+endfunc
+
+" Test for the fuzzymatchpos() function
+func Test_matchfuzzypos()


missing tests for empty array. matchfuzzypos([], 'foo')

prabirshrestha · 2020-09-15T17:43:27Z

runtime/doc/eval.txt

 		   :let l = readfile("buffer.c")->matchfuzzy("str")
 <		results in a list of lines in "buffer.c" fuzzy matching "str".

+matchfuzzypos({list}, {str} [, {dict}])			*matchfuzzypos()*


add example for dict or might be add a comment to refer to matchfuzzy dict option.

prabirshrestha

added few comments.

prabirshrestha · 2020-09-15T17:49:12Z

src/testdir/test_functions.vim

+" Test for the fuzzymatchpos() function
+func Test_matchfuzzypos()
+  call assert_equal([['curl', 'world'], [[2,3], [2,3]]], matchfuzzypos(['world', 'curl'], 'rl'))
+  call assert_equal([['curl', 'world'], [[2,3], [2,3]]], matchfuzzypos(['world', 'one', 'curl'], 'rl'))


shouldn't this response be this instead? so that if there are multiple highlights it can still be represented correctly?

[ [ "curl", "world", "curl world"], [ [ [2,3] ], [ [2,3 ] ], [ [2,3], [7,8] ] ] ]

vim-ml · 2020-09-15T17:57:08Z

Hi,

On Tue, Sep 15, 2020 at 10:50 AM Prabir Shrestha ***@***.***> wrote: ***@***.**** requested changes on this pull request. added few comments. ------------------------------ In src/testdir/test_functions.vim <#6947 (comment)>: > + call assert_fails("let x = matchfuzzy(l, 'day', {'text_cb' : {a, b -> 1}})", 'E119:') + call assert_equal([], matchfuzzy(l, 'cam')) + call assert_fails("let x = matchfuzzy(l, 'cam', {'text_cb' : []})", 'E921:') + call assert_fails("let x = matchfuzzy(l, 'foo', {'key' : []})", 'E730:') + call assert_fails("let x = matchfuzzy(l, 'cam', test_null_dict())", 'E715:') + call assert_fails("let x = matchfuzzy(l, 'foo', {'key' : test_null_string()})", 'E475:') + call assert_fails("let x = matchfuzzy(l, 'foo', {'text_cb' : test_null_function()})", 'E475:') + + let l = [{'id' : 5, 'name' : 'foo'}, {'id' : 6, 'name' : []}, {'id' : 7}] + call assert_fails("let x = matchfuzzy(l, 'foo', {'key' : 'name'})", 'E730:') +endfunc + +" Test for the fuzzymatchpos() function +func Test_matchfuzzypos() + call assert_equal([['curl', 'world'], [[2,3], [2,3]]], matchfuzzypos(['world', 'curl'], 'rl')) + call assert_equal([['curl', 'world'], [[2,3], [2,3]]], matchfuzzypos(['world', 'one', 'curl'], 'rl')) shouldn't this response be this instead? so that if there is multiple highlights it can still be represented correctly? [ [ "curl", "world", "curl world"], [ [ [2,3] ], [ [2,3 ] ], [ [2,3], [7,8] ] ] ] The fuzzy matching currently looks for only the first match. If there are

multiple matches in the same string, it currently returns only the first match. - Yegappan

prabirshrestha · 2020-09-18T00:36:01Z

@yegappan any plans to return multiple highlights for an item? Is it currently the limitation in the library?
Would be good if it supported it even if the library didnt support it so in future can update the library without updating the public api.

vim-ml · 2020-09-18T02:48:16Z

Hi,

On Thu, Sep 17, 2020 at 5:36 PM Prabir Shrestha ***@***.***> wrote: @yegappan <https://github.com/yegappan> any plans to return multiple highlights for an item? Is it currently the limitation in the library? Would be good if it supported it even if the library didnt support it so in future can update the library without updating the public api.

I am not sure about the use case for returning multiple matches within a given string. Why is it not good enough to just highlight the best match within a string? How will highlighting multiple matches within a given string help a user? For example, consider the case of searching for 'aa' in the string 'aaaaaaaa' (worst case scenario). If all the matches are highlighted, then the entire string will be highlighted, which doesn't add much value. Regards, Yegappan

prabirshrestha · 2020-09-18T20:22:50Z

It is very common if you are trying to create a fuzzy matching for files.

Here is a real example where I'm searching for sys\dri\hosts. In this case I'm actually interested in hosts file and not sys. this means hosts needs to be highlighted.

In projects I can see this using as controller/feature/featuresA/component.tsx, controller/features/featuresA/service.tsx I might want to use featureA/service. which means I want both to highlight.

vim-ml · 2020-09-18T21:45:13Z

Hi,

On Fri, Sep 18, 2020 at 1:23 PM Prabir Shrestha ***@***.***> wrote: It is very common if you are trying to create a fuzzy matching for files. Here is a real example where I'm searching for sys\dri\hosts. In this case I'm actually interested in hosts file and not sys. this means hosts needs to be highlighted.

This is already supported. For example, using your sample input: let f = ['System32\drivers\etc\lmhosts.sam', 'System32\drivers\mausbhost.sys', 'System32\drivers\etc\hosts.ics', 'System32\drivers\etc\hosts'] echo matchfuzzypos(f, 'Sys\dri\hosts') produces the following output [['System32\drivers\etc\hosts', 'System32\drivers\etc\hosts.ics', 'System32\driv ers\etc\lmhosts.sam', 'System32\drivers\mausbhost.sys'], [[0, 1, 2, 8, 9, 10, 11 , 16, 21, 22, 23, 24, 25], [0, 1, 2, 8, 9, 10, 11, 16, 21, 22, 23, 24, 25], [0, 1, 2, 8, 9, 10, 11, 16, 23, 24, 25, 26, 27], [0, 1, 2, 8, 9, 10, 11, 16, 22, 23, 24, 25, 27]]] Using the matching positions in the above output, you can highlight the characters like shown below in your example. Regards, Yegappan

…

[image: image] <https://user-images.githubusercontent.com/287744/93641752-aae6c700-f9b1-11ea-92df-8113c97664b9.png> In projects I can see this using as controller/feature/featuresA/component.tsx, controller/features/featuresA/service.tsx I might want to use featureA/service. which means I want both to highlight.

prabirshrestha · 2020-09-18T22:43:26Z

If I have 100 matches with each items having 2 highlights and only want to highlight 39:49 items do I need to loop through all the highlights which is 200 iterations or I can just ask for items[39:49] and iterate only 20 higlights?

I'm looking something like ios api for virtualization.

vim-ml · 2020-09-19T02:30:16Z

Hi,

On Fri, Sep 18, 2020 at 3:43 PM Prabir Shrestha ***@***.***> wrote: If I want to 100 matches with each items having 2 highlights and only want to highlight 39:49 items do I need to loop through all the highlights which is 200 iterations or I can just ask for items[39:49] and iterate only 20 higlights?

In the list (second item) returned by matchfuzzypos() you can use list slices to get only the matching positions for specific strings. For example, if there are 100 matching strings and you want to highlight only strings 39 to 49, then you can slice the second list using 39:49 and use only those items for highlighting. - Yegappan

…

I'm looking something like ios api <https://developer.apple.com/documentation/uikit/uitableviewdatasource/1614861-tableview?language=objc> for virtualization.

prabirshrestha · 2020-09-19T07:31:01Z

Now it makes sense. I was looking for range but seems like you are giving character positions so should be possible to do it.

For this text System32\\drivers\\mausbhost.sys when I search for Sys\dri\hosts I get 0, 1, 2, 8, 9, 10, 11, 16, 22, 23, 24, 25, 27. If I want to use prop_add I need to calculate the length on my own as I most likely want to avoid adding prop_add for each character.

let matches = [ 0, 1, 2, 8, 9, 10, 11, 16, 22, 23, 24, 25, 27 ]

call prop_add(lnum, 0, { 'length': 3 })
call prop_add(lnum, 8, { 'length': 4 })
call prop_add(lnum, 16, { 'length': 1 })
call prop_add(lnum, 22, { 'length': 4 })
call prop_add(lnum, 27, { 'length': 1 })

might be what I need now is another function to convert it to range.

let matches = [ 0, 1, 2, 8, 9, 10, 11, 16, 22, 23, 24, 25, 27 ]
let matchrange = matchpos2range(matches) 
" matchrange = [[0, 3], [8, 4], [16, 1], [22, 4], [27, 1]]

for range in matchrange
    call prop_add(lnum, range[0], { 'length': range[1] })
endfor

bfrg · 2020-09-19T07:44:18Z

Here's a very simple command that I used for testing matchfuzzypos():

if prop_type_get('test')->empty()
    call prop_type_add('test', {'highlight': 'Search'})
endif

function s:fuzzyfind(bang, mod, ...) abort
    if a:0 != 2
        return
    endif

    const files = systemlist('cd ' .. a:1 .. '; find -type f | sed "s/^\.\///"')
    const matches = matchfuzzypos(files, a:2)
    const info = printf('%s [dir: %s]', a:2, fnamemodify(a:1, ':~:.'))

    if empty(matches)
        return
    endif

    silent execute a:mod 'new' fnameescape(info)
    setlocal buftype=nofile noswapfile bufhidden=wipe
    call setline(1, matches[0])

    if a:bang
        for i in len(matches[1])->range()[: winheight(0) - 1]
            call len(matches[1][i])
                    \ ->range()
                    \ ->map({_,j ->
                    \   prop_add(i + 1, matches[1][i][j] + 1, {
                    \     'length': 1,
                    \     'type': 'test',
                    \     'bufnr': bufnr('%')
                    \   })
                    \ })
        endfor
    endif
    setlocal nomodifiable
endfunction

command -nargs=+ -bang -complete=dir Fuzzy call s:fuzzyfind(<bang>0, <q-mods>, <f-args>)

Use it like :Fuzzy! ~/path/to/project/ syshost.

…or a callback function and the matchfuzzypos() function. Add support for fuzzy matching multibyte characters.

Problem: Fuzzy matching only works on strings. Solution: Support passing a dict. Add matchfuzzypos() to also get the match positions. (Yegappan Lakshmanan, closes vim/vim#6947) vim/vim@4f73b8e

Problem: Fuzzy matching only works on strings. Solution: Support passing a dict. Add matchfuzzypos() to also get the match positions. (Yegappan Lakshmanan, closes vim/vim#6947) vim/vim@4f73b8e Also remove some N/A and dead checks.

Problem: Fuzzy matching only works on strings. Solution: Support passing a dict. Add matchfuzzypos() to also get the match positions. (Yegappan Lakshmanan, closes vim/vim#6947) vim/vim@4f73b8e Also remove some N/A and dead checks. Adjust tests. Note that the text_cb tests actually throw E6000 in Nvim, but we also can't assert that error due to v8.2.1183 not being ported yet.

Problem: Fuzzy matching only works on strings. Solution: Support passing a dict. Add matchfuzzypos() to also get the match positions. (Yegappan Lakshmanan, closes vim/vim#6947) vim/vim@4f73b8e Also remove some N/A and seemingly useless NULL checks (Nvim allocs can't return NULL. I'm not sure why the retmatchpos stuff in match_fuzzy checks for NULL too, given that Vim checks for NULL alloc in do_fuzzymatch...) Adjust tests. Note that the text_cb tests actually throw E6000 in Nvim, but we also can't assert that error due to v8.2.1183 not being ported yet.

Problem: Fuzzy matching only works on strings. Solution: Support passing a dict. Add matchfuzzypos() to also get the match positions. (Yegappan Lakshmanan, closes vim/vim#6947) vim/vim@4f73b8e Also remove some N/A and seemingly useless NULL checks -- Nvim allocs can't return NULL. I'm not sure why the retmatchpos stuff in match_fuzzy checks for NULL too, given that Vim checks for NULL alloc in do_fuzzymatch; assert that the li stuff is not NULL as that's the one check I'm ever-so-slightly unsure about. Adjust tests. Note that the text_cb tests actually throw E6000 in Nvim, but we also can't assert that error due to v8.2.1183 not being ported yet.

yegappan requested a review from brammool as a code owner September 13, 2020 05:42

vim-ml mentioned this pull request Sep 13, 2020

Add the matchfuzzy() function #6932

Closed

prabirshrestha reviewed Sep 13, 2020

View reviewed changes

runtime/doc/eval.txt Outdated

based on the matching score. {str} is treated as a literal

string and regular expression matching is NOT supported.

The maximum supported {str} length is 256.

matchfuzzy({list}, {str} [, {key}])

This comment was marked as resolved.

Sign in to view

yegappan force-pushed the fuzzy branch 4 times, most recently from c145346 to 070ce66 Compare September 14, 2020 03:01

prabirshrestha approved these changes Sep 14, 2020

View reviewed changes

yegappan force-pushed the fuzzy branch 2 times, most recently from 31d8dfa to c6c41f8 Compare September 15, 2020 17:41

prabirshrestha reviewed Sep 15, 2020

View reviewed changes

prabirshrestha suggested changes Sep 15, 2020

View reviewed changes

yegappan force-pushed the fuzzy branch 5 times, most recently from 304776a to 2d75b3f Compare September 17, 2020 23:50

yegappan force-pushed the fuzzy branch from 27fb55a to e5ef298 Compare September 18, 2020 03:53

yegappan force-pushed the fuzzy branch 2 times, most recently from 714ef10 to 113e4ca Compare September 19, 2020 06:45

yegappan force-pushed the fuzzy branch from 8755649 to 7a197aa Compare September 19, 2020 07:32

yegappan force-pushed the fuzzy branch 2 times, most recently from 6bcbbbe to 98ca316 Compare September 20, 2020 06:02

Add support fuzzy matching a list of dictionaries using either a key …

a940fba

…or a callback function and the matchfuzzypos() function. Add support for fuzzy matching multibyte characters.

yegappan force-pushed the fuzzy branch from 98ca316 to a940fba Compare September 20, 2020 23:10

Move the fuzzy matching tests to a separate test file

0e1bee5

brammool closed this in 4f73b8e Sep 22, 2020

vigoux mentioned this pull request Sep 27, 2020

[WIP]vim-patch: 8.2.{1665, 1726} matchfuzzy() neovim/neovim#12995

Closed

Uh oh!

Conversation

yegappan commented Sep 13, 2020

Uh oh!

codecov bot commented Sep 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

This comment was marked as resolved.

Uh oh!

prabirshrestha Sep 15, 2020

Choose a reason for hiding this comment

Uh oh!

prabirshrestha Sep 15, 2020

Choose a reason for hiding this comment

Uh oh!

prabirshrestha Sep 15, 2020

Choose a reason for hiding this comment

Uh oh!

prabirshrestha Sep 15, 2020

Choose a reason for hiding this comment

Uh oh!

prabirshrestha left a comment

Choose a reason for hiding this comment

Uh oh!

prabirshrestha Sep 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vim-ml commented Sep 15, 2020 via email

Uh oh!

prabirshrestha commented Sep 18, 2020

Uh oh!

vim-ml commented Sep 18, 2020 via email

Uh oh!

prabirshrestha commented Sep 18, 2020

Uh oh!

vim-ml commented Sep 18, 2020 via email

Uh oh!

prabirshrestha commented Sep 18, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vim-ml commented Sep 19, 2020 via email

Uh oh!

prabirshrestha commented Sep 19, 2020

Uh oh!

bfrg commented Sep 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Sep 13, 2020 •

edited

Loading

prabirshrestha Sep 15, 2020 •

edited

Loading

prabirshrestha commented Sep 18, 2020 •

edited

Loading

bfrg commented Sep 19, 2020 •

edited

Loading