I've tested this with a fake tool and it works correctly if the test-data directory is a symlink.
On the other hand, if test-data is a normal directory and some output test files are symlinks, with this PR the symlinks are replaced with updated real files, so I'm not sure it's the proper fix for #857 (according to @bernt-matthias's comment on #865).

Yes, that is what I intended, I don't think you'll want to overwrite symlink targets.

As far as I understand #857 this was mostly an issue with having test data point out of the runnable directory, which Galaxy doesn't allow. That will work fine with this PR, you'll just have to take care with --update_test_data

OK, let's see if @bernt-matthias has anything to add.

I seem to be able to replicate the same behaviour with just these changes:

diff --git a/planemo/galaxy/test/actions.py b/planemo/galaxy/test/actions.py
index 92cda85..4cf9c8b 100644
--- a/planemo/galaxy/test/actions.py
+++ b/planemo/galaxy/test/actions.py
@@ -3,9 +3,9 @@
 import io
 import json
 import os
+from distutils.dir_util import copy_tree
 
 import click
-from galaxy.tool_util.deps.commands import shell
 from galaxy.util import unicodify
 
 from planemo.exit_codes import (
@@ -96,8 +96,7 @@ def run_in_config(ctx, config, run=run_galaxy_command, **kwds):
         action
     )
     if kwds.get('update_test_data', False):
-        update_cp_args = (job_output_files, config.test_data_dir)
-        shell('cp -r "%s"/* "%s"' % update_cp_args)
+        copy_tree(job_output_files, config.test_data_dir)
 
     _check_test_outputs(xunit_report_file_tracker, structured_report_file_tracker)
     test_results = test_structures.GalaxyTestResults(

Are the other changes needed? In case, if you can add some comments explaining the reason, thanks!

I seem to be able to replicate the same behaviour with just these changes:

The diff you posted will not expand symlinks in the runnable location, so when the test tool interactor asks for the test files from Galaxy Galaxy will deny this https://github.com/mvdbeek/galaxy/blob/fdfdcc50b45ea692e747db7cf05f9adac235ae72/lib/galaxy/tools/__init__.py#L972. That is the cause of #857 and #865.

I mostly wrote this for dockerized tests where on top of these restrictions the symlink target won't be mounted in leading to broken symlinks, not just in the test data but also for symlinked scripts as in ruvseq. If you want to check this for yourself I'd suggest running planemo test --biocontainers on ruvseq (symlinked script) or samtools_cram_to_bam test 4 (symlinked test table data).

We don't have any iuc tools around that would elicit #857, but you can simulate that with

cd tools/column_maker/test-data
mv column_maker_out1.interval ../../
ln -s ../../column_maker_out1.interval .
planemo test column_maker.xml

That'll fail with

Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittest/case.py", line 329, in run
    testMethod()
  File "/private/var/folders/df/6xqpqpcd7h73b6jpx9t6cwhw0000gn/T/tmpwttc77ji/galaxy-dev/test/functional/test_toolbox.py", line 99, in test_tool
    self.do_it(tool_version=tool_version, test_index=test_index)
  File "/private/var/folders/df/6xqpqpcd7h73b6jpx9t6cwhw0000gn/T/tmpwttc77ji/galaxy-dev/test/functional/test_toolbox.py", line 36, in do_it
    verify_tool(tool_id, self.galaxy_interactor, resource_parameters=resource_parameters, test_index=test_index, tool_version=tool_version, register_job_data=register_job_data)
  File "/private/var/folders/df/6xqpqpcd7h73b6jpx9t6cwhw0000gn/T/tmpwttc77ji/galaxy-dev/lib/galaxy/tool_util/verify/interactor.py", line 789, in verify_tool
    raise e
JobOutputsError: Test file (column_maker_out1.interval) is missing. If you use planemo try --update_test_data to generate one.

This is a lot of magic outside of Galaxy 😟. I don't know what to do otherwise though, thanks for the fixes.

Thanks @jmchilton! If it is any consolation this brings the test closer to what is being uploaded to the toolshed. Another alternative would have been to do some staging on the Galaxy side, which I think pulsar does anyway ?