It might make more sense to read this data from a user-configurable place: a user might want to modify the standard command, or add a missing one.

Sure, I can look into that.

What configuration do you expect users to do? Replace the command entirely, always add specific arguments, etc. Off the top of my head, I'm thinking we add a field to config.json that covers cases we want to start with (testCommand, args, etc). Or do you mean something central? They could maybe override commands (and args, etc) per-track in a central place.

I think it would make sense for it to be user-overridable, but to include these defaults in the CLI. That way the command will work out of the box for most people without having to modify the config file, but the option is there if people want it.

Here's a thought: use ~/.config/exercism/user.json
Extend exercism configure

exercism configure --test --track go --script "go test"

exercism configure --test --track ruby --script "(cat <<'END'
  sh -c '
    for t in *_test.rb; do
      gawk -i inplace '1; /< Minitest::Test/ {print "  def skip; end"}' "$t"
      ruby "$t"
    done
  '
END
)"

And then ~/.config/exercism/user.json contains

{
  "apibaseurl": "https://api.exercism.io/v1",
  "token": "my-uuid",
  "workspace": "/Users/glennj/src/exercism/exercism.io"
  "testing": {
    "go": "go test",
    "ruby": "  sh -c '\n    for t in *_test.rb; do\n      gawk -i inplace '1; /< Minitest::Test/ {print \"  def skip; end\"}' \"$t\"\n      ruby \"$t\"\n    done\n  '"
  }
}

I'm not opposed to making it so configurable, but I wonder if we shouldn't release a simpler version initially and then add configuration options later?

With the universal test runner, one of my main design goals was "it just works". I think being able to run every test suite out of the box without configuration is a great starting point and will cover a lot of use cases and we can iterate from there.

Nice to point to the HELP.md file

I also noticed that every track's testing instructions are at https://exercism.org/docs/tracks/{TRACK}/tests, so I could link out to that too (or instead).

You could, but I think HELP.md is even better in this case

I wonder if we should maybe no use "test handler", which is somewhat technical, but use a more functional description. Maybe:

yeah, great call. I went with the \"%s\" track does not yet support running tests using the Exercism CLI. Please see HELP.md for testing instructions, to be clear tests can be run via a CLI, just not the exercism one. Also I kept the track name in, since that reminds them where they are (to reduce confusion vs "this track")

I like how this is now "a thing". Maybe the getExerciseSolutionFiles function in workspace/submit.go can also be updated once this PR is merged.

yeah! that would make sense.

CI was complaining about this block, so I ran go fmt ./... and this was the change. I realize you want to keep the PR focused, but I also assume you want to keep the CI green

I experimented with putting {{test_files}} in a const (which would improve the replacement code), but it made writing the commands less clear (one of "ruby " + TEST_FiLES or fmt.Sprintf("ruby %s", TEST_FILES) or injectTestFilesPlaceholder("ruby %s"), all of which seemed to defeat the point).

As written, there's a higher chance that someone adding a new command will typo the placeholder and the replacement won't happen, but it provides the most clarity when reading the actual commands.

I think this is nice!

I've updated, but was just copying from the docs. they may need to be updated: https://exercism.org/docs/tracks/cobol/tests#h-windows

I wonder if we should maybe no use "test handler", which is somewhat technical, but use a more functional description. Maybe:

I really like the logging! My only concern is that it might be too subtle for the student to notice. Maybe we could make it more explicit? To instead of --> <command>, something like Running tests via: <command> or Running test command: <command>

Sure! updated to:

Running tests via `<command>`\n\n

which makes it clear exactly what's running and gives it some space to breathe

I've just done a quick double check to see which active tracks are missing.
This is the list of missing tracks:

• abap
• common-lisp
• cpp
• delphi
• fortran
• objective-c
• pharo-smalltalk
• plsql
• powershell
• r
• scheme
• unison
• vimscript

Could you double check these tracks? If they're not supported, it might be useful to add a comment somewhere noting that those tracks are not supported (and why)

Sure, added in ad33864! I've confirmed that most of the remaining tracks are N/A (tests are run in IDE, etc).

For common-lisp, I used

sbcl --load "${slug}-test" \
     --eval "(exit :code (if (${slug}-test:run-tests) 0 5))"

That assumes a particular distribution: there are several suggested on the track's intstallation page.

For fortran, I have this fish snippet

    if not test -d ./.exercism
        echo run me from the exercise root >&2
        return 1
    end
    mkdir -p build
    cd build
    test -d ./testlib; or cmake -G "Unix Makefiles" ..
    make; and ctest -V
    cd ..

R is just

Rscript test_*.R

R is just
Rscript test_*.R

Could you update the configuration to use this command? I can confirm that this is also how the test runner does it.

Ah, good call! I didn't even think about the test runners - I've been going off the public docs

Looking at how the test runner does it, it looks like this might be possible. Could you double-check?

While this does seem to work, the output is super long and ugly. There doesn't seem to be a --quiet flag and I'm disinclined to ship it as is:

% vim -XN -i NONE -c ":so hello_world.vim" -c "Vader! hello_world.vader"
Vader note: cannot print to stderr reliably/directly.  Please consider using Vim's -es/-Es option (mode=n).
VIM - Vi IMproved 9.0 (2022 Jun 28, compiled Apr 15 2023 04:26:46)
macOS version - arm64
Included patches: 1-1424
Compiled by root@apple.com
Normal version without GUI.  Features included (+) or not (-):
+acl               -dnd               +listcmds          +postscript        +termresponse
-arabic            -ebcdic            +localmap          +printer           +textobjects
+autocmd           -emacs_tags        -lua               -profile           +textprop
+autochdir         +eval              +menu              -python            +timers
-autoservername    +ex_extra          +mksession         -python3           +title
-balloon_eval      +extra_search      +modify_fname      +quickfix          -toolbar
-balloon_eval_term -farsi             +mouse             +reltime           +user_commands
-browse            +file_in_path      -mouseshape        -rightleft         -vartabs
++builtin_terms    +find_in_path      +mouse_dec         -ruby              +vertsplit
+byte_offset       +float             -mouse_gpm         +scrollbind        +vim9script
+channel           +folding           -mouse_jsbterm     +signs             +viminfo
+cindent           -footer            +mouse_netterm     +smartindent       +virtualedit
-clientserver      +fork()            +mouse_sgr         -sodium            +visual
+clipboard         -gettext           -mouse_sysmouse    -sound             +visualextra
+cmdline_compl     -hangul_input      +mouse_urxvt       +spell             +vreplace
+cmdline_hist      +iconv             +mouse_xterm       +startuptime       +wildignore
+cmdline_info      +insert_expand     +multi_byte        +statusline        +wildmenu
+comments          +ipv6              +multi_lang        -sun_workshop      +windows
+conceal           +job               -mzscheme          +syntax            +writebackup
+cryptv            +jumplist          +netbeans_intg     +tag_binary        -X11
+cscope            -keymap            +num64             -tag_old_static    -xfontset
+cursorbind        +lambda            +packages          -tag_any_white     -xim
+cursorshape       -langmap           +path_extra        -tcl               -xpm
+dialog_con        +libcall           -perl              +termguicolors     -xsmp
+diff              +linebreak         +persistent_undo   +terminal          -xterm_clipboard
+digraphs          +lispindent        +popupwin          +terminfo          -xterm_save
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/share/vim"
Compilation:
gcc -c -I. -Iproto -DHAVE_CONFIG_H   -DMACOS_X_UNIX  -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: gcc   -L/usr/local/lib -o vim        -lm -lncurses  -liconv -framework Cocoa

Starting Vader: 1 suite(s), 1 case(s)
  Starting Vader: /Users/david/projects/exercism/vimscript/hello-world/hello_world.vader
    (1/1) [EXECUTE] Say Hi!
  Success/Total: 1/1
Success/Total: 1/1 (assertions: 1/1)
Elapsed time: 0.01 sec.

Do you think it's common that users are running this standalone vs inside vim itself?

Probably not. The verbosity is indeed a bit much. Let's leave it out

R is just
Rscript test_*.R

Could you update the configuration to use this command? I can confirm that this is also how the test runner does it.

Could we do cmake and then make via &&? See https://github.com/exercism/cpp-test-runner/blob/main/bin/run.sh#L43 We'd have to test if this works in Windows of course.

It won't work as written. It works if I run cmake . && make manually, but because we're passing the command into a single exec.Command, it's being run as cmake . make, and make is ignored as an unknown path:

CMake Warning:
  Ignoring extra path from command line:

   "make"

To make that work, we'd need to make every command a slice of strings (most of which will only have 1 item) and then run each in order (which will work cross-platform).

I think that's a reasonable improvement to make, but I'd recommend putting it in its own PR; this one has gotten big. I also have no idea if/how this build script works on windows- I left off the -G "Unix Makefiles" part of the command and I have no idea how important that is. Verifying this is probably best left to someone who knows more about those tracks. I'd rather ship a TODO than a solution we're not sure works

Actually, it looks like the cmake portion is 1-time setup. Once it's run successfully, the only thing you need to run tests is make. I think it's reasonable to treat the cmake portion as user-required setup/installation and the actual test run as make. I'll update accordingly.

I was just appreciating this new feature and noticed this file was added. I’m assuming it wasn’t meant to be committed so I thought I’d point it out 🤓

good call, removed 😅

Probably not. The verbosity is indeed a bit much. Let's leave it out