Skip to content

WIP [DO NOT MERGE] Test attempt to put existing FSMonitor code on top of multi-threaded simple-ipc code#282

Closed
jeffhostetler wants to merge 81 commits intomicrosoft:features/fsmonitor-2.27.0from
jeffhostetler:fsmonitor
Closed

WIP [DO NOT MERGE] Test attempt to put existing FSMonitor code on top of multi-threaded simple-ipc code#282
jeffhostetler wants to merge 81 commits intomicrosoft:features/fsmonitor-2.27.0from
jeffhostetler:fsmonitor

Conversation

@jeffhostetler
Copy link

@jeffhostetler jeffhostetler commented Jul 9, 2020

This PR moves the FSMonitor branch thicket onto the multi-threaded simple-ipc code.
A single fixup commit on top was added to switch to the multi-threaded IPC API.

FSMonitor builds and tests pass on both Windows and Mac, but I haven't actually made
any changes to the existing branch thicket.

This PR should replace #273.

TODO
[x] I hard-coded the number of threads in the call in FSMonitor to ipc_server_run(). This should be a config setting or something. -- I added fsmonitor.ipcthreads and --ipcthreads.

Depends on #281.

dscho and others added 30 commits June 4, 2020 16:24
In 897d68e (Makefile: use curl-config --cflags, 2020-03-26), we
taught the build process to use `curl-config --cflags` to make sure that
it can find cURL's headers.

In the MSVC build, this is completely bogus because we're running in a
Git for Windows SDK whose `curl-config` supports the _GCC_ build.

Let's just ignore each and every `-I<path>` option where `<path>` points
to GCC/Clang specific headers.

Reported by Jeff Hostetler in
microsoft#275.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Teach packet_write_gently() to use a stack buffer rather than a static
buffer when composing the packet line message.  This helps get us ready
for threaded operations.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
So far, the (possibly indirect) callers of `get_packet_data()` can ask
that function to return an error instead of `die()`ing upon end-of-file.

However, when we call this function in a long-running daemon, we
absolutely want the daemon to live, still, even if there was a read
error when one random read failed.

So let's introduce an explicit option to tell the packet reader
machinery to please be nice and only return an error.

This change prepares pkt-line for the internal fsmonitor (which is
precisely such a daemon).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
…uf()

This function currently has only one caller: `apply_multi_file_filter()`
in `convert.c`. That caller wants a flush packet to be written after
writing the payload.

However, we are about to introduce a user that wants to write many
packets before a final flush packet, so let's extend this function to
prepare for that scenario.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `read_packetized_to_strbuf()` function reads packets into a strbuf
until a flush packet has been received. So far, it has only one caller:
`apply_multi_file_filter()` in `convert.c`. This caller really only
needs the `PACKET_READ_GENTLE_ON_EOF` option to be passed to
`packet_read()` (which makes sense in the scenario where packets should
be read until a flush packet is received).

We are about to introduce a caller that wants to pass other options
through to `packet_read()`, so let's extend the function signature
accordingly.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
TODO words here

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Define client-side prototypes for simple-ipc API.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Define server-side prototypes for simple-ipc API.  This is a simplified
and synchronous interface within the server.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Define server-side prototypes for simple-ipc API.  This extends the API
with an asynchronous interface.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The idea is to provide an efficient default implementation of the
`fsmonitor` hook for the major platforms, starting on the platform that
benefits from it most: Windows (because there is no native `stat()`, it
has to be emulated, which means that refreshing the Git index is a
rather hefty operation on that platform).

Originally, the intention of the `fsmonitor` extension was quite
obviously to use Facebook's `watchman`. However, it turned out that it
is just too hard to make it work reliably on Windows: none of the core
`watchman` developers has access to a Windows machine, and it is even
non-trivial to make the project build on Windows (which is not helped by
the hierarchy of build dependencies).

Besides: when it comes to Git's `fsmonitor` hook, `watchman` does both
too much (it allows pretty complicated queries, which we do not need in
the `fsmonitor` hook) and too little (`watchman` is totally unaware of
Git's internals, e.g. cannot use the `untracked` extension to initialize
its state efficiently upon start-up, therefore the first `git status` is
still slow, even if `watchman` is used via the `fsmonitor`).

Given all these considerations, it makes sense to "roll our own".

This commit merely adds the command without any functionality; Over the
course of the next commits we will implement it in bits and pieces.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This patch adds a scaffolding for an internal Git-aware FSMonitor, in
preparation for adding platform-specific backends.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
By nature, the backends for the fsmonitor daemon need to be
platform-specific: there is no common API (not even a POSIX one) for
monitoring directories for changes.

This patch adds the first such backend: a Windows one, based on the
Win32 API function `ReadDirectoryChangesW()`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
For the moment, there are only two more command modes (in addition to
the already-existing `--is-supported` one): `--run`. one mode to start
the daemon in the foreground, and another one to query a running daemon.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It is quite useful to have a way to check whether there is an
already-running FSMonitor daemon; This functionality will be used in one
of the next commits to auto-start one when querying via the FSMonitor
hook.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Kevin Willford <Kevin.Willford@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
TODO: test!

It is quite common that the internal fsmonitor receives multiple events
for the same path. There is no need for the fsmonitor to list the path
multiple times, though, as Git just wants to verify via an `lstat()`
whether the file changed or not.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
TODO: add a test case

The `core.fsmonitor` setting is supposed to be set to a path pointing to
a script or executable that queries an fsmonitor such as watchman.

We are about to implement our own backend, and do not want to spawn an
extra process just to query it, but want to use Inter-Process
Communication to talk to the daemon (and start it if necessary).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Those paths are not for reportin' ;-)

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
…Events

Signed-off-by: Kevin Willford <Kevin.Willford@microsoft.com>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Convert simple-ipc calls to ipc_server_* and ipc_client_* versions

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach fsmonitor--daemon to accept the number of IPC threads to start
from the config and the command line.

TODO I selected `fsmonitor.ipcThreads` for now.  Decide on final config
TODO key name and document somewhere in `Documentation/config/*.txt`.

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
…eleting is-active logic

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
… deleting is-active logic

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
… deleting is-active logic

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
…ompat/fsmonitor/macos.c

Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
@jeffhostetler
Copy link
Author

Closing this obsolete FSMonitor PR in favor of #282.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants