WIP [DO NOT MERGE] Test attempt to put existing FSMonitor code on top of multi-threaded simple-ipc code#282
Closed
jeffhostetler wants to merge 81 commits intomicrosoft:features/fsmonitor-2.27.0from
Closed
Conversation
In 897d68e (Makefile: use curl-config --cflags, 2020-03-26), we taught the build process to use `curl-config --cflags` to make sure that it can find cURL's headers. In the MSVC build, this is completely bogus because we're running in a Git for Windows SDK whose `curl-config` supports the _GCC_ build. Let's just ignore each and every `-I<path>` option where `<path>` points to GCC/Clang specific headers. Reported by Jeff Hostetler in microsoft#275. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Teach packet_write_gently() to use a stack buffer rather than a static buffer when composing the packet line message. This helps get us ready for threaded operations. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
So far, the (possibly indirect) callers of `get_packet_data()` can ask that function to return an error instead of `die()`ing upon end-of-file. However, when we call this function in a long-running daemon, we absolutely want the daemon to live, still, even if there was a read error when one random read failed. So let's introduce an explicit option to tell the packet reader machinery to please be nice and only return an error. This change prepares pkt-line for the internal fsmonitor (which is precisely such a daemon). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
…uf() This function currently has only one caller: `apply_multi_file_filter()` in `convert.c`. That caller wants a flush packet to be written after writing the payload. However, we are about to introduce a user that wants to write many packets before a final flush packet, so let's extend this function to prepare for that scenario. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
The `read_packetized_to_strbuf()` function reads packets into a strbuf until a flush packet has been received. So far, it has only one caller: `apply_multi_file_filter()` in `convert.c`. This caller really only needs the `PACKET_READ_GENTLE_ON_EOF` option to be passed to `packet_read()` (which makes sense in the scenario where packets should be read until a flush packet is received). We are about to introduce a caller that wants to pass other options through to `packet_read()`, so let's extend the function signature accordingly. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Fix the MSVC build
TODO words here Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Define client-side prototypes for simple-ipc API. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Define server-side prototypes for simple-ipc API. This is a simplified and synchronous interface within the server. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Define server-side prototypes for simple-ipc API. This extends the API with an asynchronous interface. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
The idea is to provide an efficient default implementation of the `fsmonitor` hook for the major platforms, starting on the platform that benefits from it most: Windows (because there is no native `stat()`, it has to be emulated, which means that refreshing the Git index is a rather hefty operation on that platform). Originally, the intention of the `fsmonitor` extension was quite obviously to use Facebook's `watchman`. However, it turned out that it is just too hard to make it work reliably on Windows: none of the core `watchman` developers has access to a Windows machine, and it is even non-trivial to make the project build on Windows (which is not helped by the hierarchy of build dependencies). Besides: when it comes to Git's `fsmonitor` hook, `watchman` does both too much (it allows pretty complicated queries, which we do not need in the `fsmonitor` hook) and too little (`watchman` is totally unaware of Git's internals, e.g. cannot use the `untracked` extension to initialize its state efficiently upon start-up, therefore the first `git status` is still slow, even if `watchman` is used via the `fsmonitor`). Given all these considerations, it makes sense to "roll our own". This commit merely adds the command without any functionality; Over the course of the next commits we will implement it in bits and pieces. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
This patch adds a scaffolding for an internal Git-aware FSMonitor, in preparation for adding platform-specific backends. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
By nature, the backends for the fsmonitor daemon need to be platform-specific: there is no common API (not even a POSIX one) for monitoring directories for changes. This patch adds the first such backend: a Windows one, based on the Win32 API function `ReadDirectoryChangesW()`. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
For the moment, there are only two more command modes (in addition to the already-existing `--is-supported` one): `--run`. one mode to start the daemon in the foreground, and another one to query a running daemon. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
It is quite useful to have a way to check whether there is an already-running FSMonitor daemon; This functionality will be used in one of the next commits to auto-start one when querying via the FSMonitor hook. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Kevin Willford <Kevin.Willford@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
TODO: test! It is quite common that the internal fsmonitor receives multiple events for the same path. There is no need for the fsmonitor to list the path multiple times, though, as Git just wants to verify via an `lstat()` whether the file changed or not. Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
TODO: add a test case The `core.fsmonitor` setting is supposed to be set to a path pointing to a script or executable that queries an fsmonitor such as watchman. We are about to implement our own backend, and do not want to spawn an extra process just to query it, but want to use Inter-Process Communication to talk to the daemon (and start it if necessary). Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Those paths are not for reportin' ;-) Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
…Events Signed-off-by: Kevin Willford <Kevin.Willford@microsoft.com> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Convert simple-ipc calls to ipc_server_* and ipc_client_* versions Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Teach fsmonitor--daemon to accept the number of IPC threads to start from the config and the command line. TODO I selected `fsmonitor.ipcThreads` for now. Decide on final config TODO key name and document somewhere in `Documentation/config/*.txt`. Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
…eleting is-active logic Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
… deleting is-active logic Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
… deleting is-active logic Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
…ompat/fsmonitor/macos.c Signed-off-by: Jeff Hostetler <jeffhost@microsoft.com>
fc1fe83 to
7660818
Compare
Author
|
Closing this obsolete FSMonitor PR in favor of #282. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR moves the FSMonitor branch thicket onto the multi-threaded simple-ipc code.
A single fixup commit on top was added to switch to the multi-threaded IPC API.
FSMonitor builds and tests pass on both Windows and Mac, but I haven't actually made
any changes to the existing branch thicket.
This PR should replace #273.
TODO
[x] I hard-coded the number of threads in the call in FSMonitor to
ipc_server_run(). This should be a config setting or something. -- I addedfsmonitor.ipcthreadsand--ipcthreads.Depends on #281.