I decided the problem I was addressing for the debugger
was more general and that the BUILT_PATH_PREFIX_MAP
spec should be generalized to handle it.

See richardlford/build-path-prefix-map-spec#1.

1. Modified BUILT_PATH_PREFIX_MAP handling accordingly
2. Used the new facilities in ocamldebug
3. Updated one test for the new spec.
4. Modify encoding and decoding functions in ocamltest.

Note this is a work in progress. The changes will be split
into multiple PRs.

In order to produce reproducible builds (independent of the location
of the build), starting from Dune 3.0, Dune has mapped references to
the workspace directory to “/workspace_root”.

This change enables ocamldebug to recover the locations of the source
code. The ocamldebug user must set the BUILD_PATH_PREFIX_MAP
environment variable to effect the mapping.

If we only want to map "/workspace_root" to a single directory, say
mydir, we would do:

export BUILD_PATH_PREFIX_MAP="mydir=/workspace_root"

If we want to map to multiple directories, we must separate
them with ';'.

So to map to mydir1 and mydir2, the setting would be:

export BUILD_PATH_PREFIX_MAP="mydir1;mydir2=/workspace_root"

Then if ocamldebug has a path like

"/workspace_root/stuff",

this is converted to the list

["mydir1/stuff" "mydir2/stuff"]

Add "mapping" debugger variable to control BUILD_PATH_PREFIX_MAP.

In order to allow the BUILD_PATH_PREFIX_MAP environment variable to be
set from within ocamldebug, a new debugger variable, "mapping" has
been added. It works with the usual set and show commands. When set,
the value is put into BUILD_PATH_PREFIX_MAP in ocamldebug's
environment, just as if it had been set before entering ocamldebug.
And showing it give the current value, including the value on
entry, if set.

This also makes it possible to set it from .ocamldebug files.

The manual and man pages have been updated to describe
the change.

1. I added four tests for the ocamldebug support for
BUILD_PATH_PREFIX_MAP. One test shows that if
debug paths are sanitized by the compiler, then
the debugger is unable to find the sources.
The other three tests show three ways to provide
the BUILD_PATH_PREFIX_MAP to ocamldebug:
a. Setting environment variable externally.
b. Setting the "mapping" ocamldebug variable
in an input script.
c. Setting the "mapping" ocamldebug variable
in a ".ocamldebug" file.

2. To write the tests I needed a way to have
script or ".ocamldebug" files that had ocamldebug
variables expanded. I did not see a way to do
that, so I added a new "expand" action to
ocamltest. It is mostly like the "copy" action,
but the source file is read a line at a time
and the lines are expanded. It does not
support source directories, but the
destination can be a directory.

3. I used a dumper to look at the directories
in the debug information and noticed that
there were still some unsanitized directories.
I tracked the source of these in the compiler
and modified so all debug directories are
sanitized.

More ocamltest enhancements, add/fix tests

1. I enhanced ocamltest to have a facility for making
builtin functions. These are like ocamltest variables
but have a function attached to them, and when
the variable is expanded, the arguments are first
expanded and then the function is called and its
result is returned.

2. Added a new "dumpenv_expanded" action which not
only shows the value of each variable, but also what
the they expand to. When variables are originally
assigned, their RHS is not expanded, but only
later when the variable itself is looked up.

3. Two builtin functions were defined:

- bppm_decode does BUILD_PATH_PREFIX_MAP decoding
- bppm_encode does BUILD_PATH_PREFIX_MAP encoding

4. Tests of the new builtin functions were added.

5. The debugger prefix mapping tests were
modified to use bppm_encode. They, in fact,
where the motivation for the ocamltest changes.

Use load/install printer trick to add ocamldebug unit test

When a printer is loaded and installed into the debugger,
it has access to the internals of the debugger.

I factored out the function that does the
BUILD_PATH_PREFIX_MAP processing in ocamldebug and was able to
unit test it with code in the fake printer.

Modify for revised BUILT_PATH_PREFIX_MAP spec

I decided the problem I was addressing for the debugger
was more general and that the BUILT_PATH_PREFIX_MAP
spec should be generalized to handle it.

See richardlford/build-path-prefix-map-spec#1.

1. Modified BUILT_PATH_PREFIX_MAP handling accordingly
2. Used the new facilities in ocamldebug
3. Updated one test for the new spec.
4. Modify encoding and decoding functions in ocamltest.

Avoid adding dir to Load_path if already their, some test changes

Sanitize paths in debug events

Previously we were sanitizing the debug directories
but were assuming the paths in debug events were
relative. But if the user passes an absolute path
to the compiler, it was putting absolute paths
in the debug events.

Fixes ocaml#12083

Note this is a work in progress. The changes will be split
into multiple PRs.

In order to produce reproducible builds (independent of the location
of the build), starting from Dune 3.0, Dune has mapped references to
the workspace directory to “/workspace_root”.

This change enables ocamldebug to recover the locations of the source
code. The ocamldebug user must set the BUILD_PATH_PREFIX_MAP
environment variable to effect the mapping.

If we only want to map "/workspace_root" to a single directory, say
mydir, we would do:

export BUILD_PATH_PREFIX_MAP="mydir=/workspace_root"

If we want to map to multiple directories, we must separate
them with ';'.

So to map to mydir1 and mydir2, the setting would be:

export BUILD_PATH_PREFIX_MAP="mydir1;mydir2=/workspace_root"

Then if ocamldebug has a path like

"/workspace_root/stuff",

this is converted to the list

["mydir1/stuff" "mydir2/stuff"]

Add "mapping" debugger variable to control BUILD_PATH_PREFIX_MAP.

In order to allow the BUILD_PATH_PREFIX_MAP environment variable to be
set from within ocamldebug, a new debugger variable, "mapping" has
been added. It works with the usual set and show commands. When set,
the value is put into BUILD_PATH_PREFIX_MAP in ocamldebug's
environment, just as if it had been set before entering ocamldebug.
And showing it give the current value, including the value on
entry, if set.

This also makes it possible to set it from .ocamldebug files.

The manual and man pages have been updated to describe
the change.

1. I added four tests for the ocamldebug support for
BUILD_PATH_PREFIX_MAP. One test shows that if
debug paths are sanitized by the compiler, then
the debugger is unable to find the sources.
The other three tests show three ways to provide
the BUILD_PATH_PREFIX_MAP to ocamldebug:
a. Setting environment variable externally.
b. Setting the "mapping" ocamldebug variable
in an input script.
c. Setting the "mapping" ocamldebug variable
in a ".ocamldebug" file.

2. To write the tests I needed a way to have
script or ".ocamldebug" files that had ocamldebug
variables expanded. I did not see a way to do
that, so I added a new "expand" action to
ocamltest. It is mostly like the "copy" action,
but the source file is read a line at a time
and the lines are expanded. It does not
support source directories, but the
destination can be a directory.

3. I used a dumper to look at the directories
in the debug information and noticed that
there were still some unsanitized directories.
I tracked the source of these in the compiler
and modified so all debug directories are
sanitized.

More ocamltest enhancements, add/fix tests

1. I enhanced ocamltest to have a facility for making
builtin functions. These are like ocamltest variables
but have a function attached to them, and when
the variable is expanded, the arguments are first
expanded and then the function is called and its
result is returned.

2. Added a new "dumpenv_expanded" action which not
only shows the value of each variable, but also what
the they expand to. When variables are originally
assigned, their RHS is not expanded, but only
later when the variable itself is looked up.

3. Two builtin functions were defined:

- bppm_decode does BUILD_PATH_PREFIX_MAP decoding
- bppm_encode does BUILD_PATH_PREFIX_MAP encoding

4. Tests of the new builtin functions were added.

5. The debugger prefix mapping tests were
modified to use bppm_encode. They, in fact,
where the motivation for the ocamltest changes.

Use load/install printer trick to add ocamldebug unit test

When a printer is loaded and installed into the debugger,
it has access to the internals of the debugger.

I factored out the function that does the
BUILD_PATH_PREFIX_MAP processing in ocamldebug and was able to
unit test it with code in the fake printer.

Modify for revised BUILT_PATH_PREFIX_MAP spec

I decided the problem I was addressing for the debugger
was more general and that the BUILT_PATH_PREFIX_MAP
spec should be generalized to handle it.

See richardlford/build-path-prefix-map-spec#1.

1. Modified BUILT_PATH_PREFIX_MAP handling accordingly
2. Used the new facilities in ocamldebug
3. Updated one test for the new spec.
4. Modify encoding and decoding functions in ocamltest.

Avoid adding dir to Load_path if already their, some test changes

Sanitize paths in debug events

Previously we were sanitizing the debug directories
but were assuming the paths in debug events were
relative. But if the user passes an absolute path
to the compiler, it was putting absolute paths
in the debug events.

Alway rewrite in Location.absolute_path

1. Modified Location.absolute_path to do mapping for
BUILD_PATH_PREFIX_MAP regardless of whether the
path is relative or absolute. Previously it only
did it for relative paths.

2. Because of ocaml#1, deleted Location.absolute_path_always.

3. In bytelink.ml, move make_absolute locally within link_bytecode
and do not do BUILD_PATH_PREFIX_MAP rewriting.
In this case the path is what goes in the shebang, and
I do not see how we would want rewriting in that case,
unless there was a later processor to rewrite the
shebangs.

4. Update man page and manual for the revised
BUILD_PATH_PREFIX_MAP spec (approval pending).

Prepare to be able to do Dune tests

1. Added a action helper for testing if a program
is available in PATH.

2. Use the helper to make an action, has_dune,
that detects whether dune is available.

Fixes ocaml#12083

The BUILD_PATH_PREFIX_MAP specification tells how to use that
environment variable to achieve reproducible build, i.e. builds of
products that do not leak absolute paths from the build
environment. See
https://reproducible-builds.org/specs/build-path-prefix-map.

However, that specification only covers half of the story.  Let us
call the building of reproducible products the "Build Phase". That is
the phase covered by the existing specification.

Let us defined the "Deployment phase" as the phase where you accept a
built reproducible product and make use of it, i.e. deploy it. An
example would be the debugger taking a reproducible binary and letting
the user debug it, showing the user the source code, etc. These
changes were originally being made for the benefit of the debugger,
but I realized they are more generally applicable and that a
generalization of the spec was desirable.

I have a proposed revision of that specification.  I have capture the
original specification on github, and then created a pull requests
with my proposed changes. See

richardlford/build-path-prefix-map-spec#1

This PR implements the revised specification.

Also, this is part of a larger PR,
ocaml#12085, that is being split up
because it was too large. In addition to these changes, that PR
includes ocamltest enhancements plus the tests for
these changes. See it for prior discussion and the other changes.

I will now describe the essence of the changes. See the
revised document for motivation.

1. utils/build_path_prefix_map.{ml,mli}

The essential change to the spec is that left-hand side of
a mapping pair (formerly the target) now may have a
list of targets, separated by ';'. Because ';' is
now used as a delimiter, it is escaped when
a path is encoded (to "%,"). Because of the encoding,
there is no restriction on characters in a path.

A list of targets that may be returned is called a
"search list". The API is extended with functions
that return search lists.

During the build phase we expect that the maps will only
have a single target. An open issue is what to do
if there is more than one target during the
build phase. Currently we ignore the mapping.

2. parsing/location.{ml, mli}

This has convenience functions for using the above
mapping API. It reads and caches the environment
variable so the end-user does not need to.

a. absolute_path was modified so that
BUILD_PATH_PREFIX_MAP rewriting is done for both
absolute and relative paths. Relative paths
are made absolute by appending to the cwd.
Previously only relative paths were rewritten.
One discovery during testing was that if the
compiler is given an absolute path as the
input source, the debug event information
had that absolute path (see below).

b. rewrite_to_search_list
New function which returns search list from the mapping.

c. search_list_find
New function that uses rewrite_to_search_list to get
a list of paths and then looks for the first one that
exists.

3. bytecomp/emitcode.ml
Added sanitizing of paths in the debug events.
If the compiler is given absolute source paths
this was leaking absolute build paths.

4. bytecomp/bytelink.ml
Never rewrite when producing the path for a shebang.
The value written is from the user "-use-runtime"
option, and I do not believe that should be rewritten.

5. utils/load_path.ml
Modified so that a directory is not added to the load path
if it is already in the load path. I was concerned that
failing to do this might not preserve the desired order.

6. debugger/command_line.ml

A debugger variable, "mapping", was added, which is tied to the
BUILD_PATH_PREFIX_MAP environment variable.  So "set mapping value"
will set BUILD_PATH_PREFIX_MAP, and "show mapping" will show the value
of BUILD_PATH_PREFIX_MAP. This allows the user to set it
programmaticaly, possibly from the ".ocamldebug" file.

7. debugger/program_management.ml
Reverse the list of directories so the ones with highest
priority are first.

8. debugger/source.ml
In source_of_module, use the Location.search_list_find API.

9. debugger/symbols.{ml, mli}
a. Take care to preserve the order of directories, so the
highest priority will come first.
b. Expose bppm_expand_path and get_load_path so they
can be called from a fake printer which, when
loaded and installed in a test, can do unit testing
of ocamldebug.

10. utils/misc.ml
When printing mapping, take into account multiple targets.

11. Man page and manual updated.

Fixes ocaml#12083

This is part 2 of a larger PR, ocaml#12085, which includes a part 1
(compiler and debugger changes), this part, and a third part which is
tests of part 1 that make use of (and hence motivate and test) these
changes. Please see that PR for example uses of these changes.

1. Add the ability to have scripts or files with ocamltest variables
references expanded. Added a new "expand" action to ocamltest. It is
mostly like the "copy" action, but the source file is read a line at a
time and the lines are expanded. It does not support source
directories, but the destination can be a directory.

2. Enhanced ocamltest to have a facility for making builtin
functions. These are like ocamltest variables but have a function
attached to them, and when the variable is expanded, the arguments are
first expanded and then the function is called and its result is
returned. Currently they only take one parameter, but it should not be
too hard to add the ability to have multiply arguments.

3. Added a new "dumpenv_expanded" action which not
only shows the value of each variable, but also what
they expand to. When variables are originally
assigned, their RHS is not expanded, but only
later when the variable itself is looked up.

4. Two builtin functions were defined:

- bppm_decode does BUILD_PATH_PREFIX_MAP decoding

- bppm_encode does BUILD_PATH_PREFIX_MAP encoding

See https://reproducible-builds.org/specs/build-path-prefix-map/
and richardlford/build-path-prefix-map-spec#1.

5. Prepare to be able to do Dune tests:

5a. Added a action helper for testing if a program
is available in PATH.

5b. Use the helper to make an action, has_dune,
that detects whether dune is available.