Skip to content

Merge milestones/M151 to releases/shipped#1087

Merged
jrbriggs merged 177 commits intoreleases/shippedfrom
milestones/M151
May 6, 2019
Merged

Merge milestones/M151 to releases/shipped#1087
jrbriggs merged 177 commits intoreleases/shippedfrom
milestones/M151

Conversation

@jrbriggs
Copy link
Member

@jrbriggs jrbriggs commented May 1, 2019

Release notes

  • Large improvements to git checkout performance when there are many folder placeholders.
  • A background task will now compress and deduplicate the pack-file data in the shared object cache, resulting in more sustainable disk usage.
  • Improvements to the installation to more gracefully recover from an uninstallation of VFS for Git and subsequent reinstallation.
  • Upgrade supports an anonymous endpoint to decouple rollouts to different engineering systems.
  • Other bug fixes, engineering improvements, and test reliability fixes.

For full details, see Milestone M151

glensc and others added 30 commits January 13, 2019 21:17
- Moved ProjFS to Managed and Native Version
For ease of comparsion with the original Mac-only source files,
we retain the line endings on the refactored POSIX common files to
exactly match the original ones.
The OSAtomic APIs are not available in that form in user space, and so caused errors when linking the testable kext. The similar user space equivalents are already deprecated in any case, so replacing their use with the standardised C11 operations is the sensible choice.
Some constants differ per @chrisd8088's comment here:

#926 (comment)

When we add the Linux project, we should factor *out* of
POSIXFileBasedLock what's Mac-specific.
Mac kext: Replaces use of OSAtomic with standard C11 atomic operations
This change removes the relative path calculation from the kext and changes the native ProjFS library to expect absolute paths in messages from the kernel. The absolute path is passed through to event handlers where in many cases the logic for re-combining the relative path into an absolute one could be removed as well.
wilbaker and others added 22 commits April 24, 2019 06:13
When someone unmounts, it triggers a Stop() on the maintenance
step that is currently running. This can interrupt longer-running Git
commands more often than other actions, so check Stopping before
checking the error code for a response.
… to error code

When someone unmounts, it triggers a Stop() on the maintenance
step that is currently running. This can interrupt longer-running Git
commands more often than other actions, so check Stopping before
checking the error code for a response.

This is more likely a reason for people to hit this error than any
data-shape issues that we were expecting to cause a problem.

While I am updating the check around the `commit-graph verify`
command, I am not un-commenting it as we don't have the full
fix for the "too many pack-files" case.
The GitMaintenanceStep throws a StoppingException if the
RunGitCommand() method is run after a Stop() call. However,
it doesn't throw when the Stop() call happens during the
Git process. This requires extra handling of the Stopping
boolean after receiving a response from RunGitCommand().
The better thing to do is to throw the StoppingException
after the Git process completes.

Add a unit test that verifies we automatically skip the
remainder of a PerformMaintenance() method in a subclass.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
The GitMaintenanceStepTests have three subclasses of
GitMaintenanceStep to invoke the Stop() method at different
times and report which blocks were reached.

Instead, use an enum to specify when we should call
Stop() and use the same subclass in all methods.

Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
…Git is running

The GitMaintenanceStep throws a StoppingException if the
RunGitCommand() method is run after a Stop() call. However,
it doesn't throw when the Stop() call happens during the
Git process. This requires extra handling of the Stopping
boolean after receiving a response from RunGitCommand().
Such extra handling was added in #1064, but would not
be necessary with this change.

The better thing to do is to throw the StoppingException
after the Git process completes.

Add a unit test that verifies we automatically skip the
remainder of a PerformMaintenance() method in a subclass.

Looking through the error logs, all logged instances of a
failure during 'commit-graph verify' or 'multi-pack-index verify'
are actually due to this issue. Otherwise we would see errors
from GitMaintenanceStep saying "Git process {command} failed with errors:"
and those messages do no appear in the logs.
Ensure that modifications and enumeration of the `JsonTracer::listeners`
collection is safe for being accesses from multiple threads.

Also create a background thread to write to `TelemetryDaemonEventListener`
pipe, freeing up the trace message entry thread to continue.

Since message send/failure now happens after the call to write an event
message the mechanism for reporting `EventListener` failure and recovery
has been changed from a method return value approach to an event sink/
callback model to allow communication back to the `JsonTracer` when a
listener fails in the background.
…async

Ensure that modifications and enumeration of the JsonTracer::listeners collection is safe with multiple threads adding/removing listeners at the same time as we're writing messages. A ConcurrentBag is used to achieve this aim.

Also ensure the TelemetryDaemonEventListener does not hold up calling threads posting messages over the pipe by queuing up messages to be sent on a single background thread.
This will enable the Enlistment.GetNewGVFSLogFileName method to interact with
the file system through the file system abstraction layer, enabling us to unit
test this method and methods that call this method.

Instead of updating all call sites to pass in a physical file system,
this is an optional argument. If one is passed in, it will be used. If
no physical file system argument is provided, a default physical file
system object will be created.
Teach Enlistment.GetNewGVFSLogFileName to take FileSystem parameter
Fix Markdown formatting and some old references.
Fix Markdown formatting and some old references.
…when it's missing and the installed ProjFS version matches the packaged version

GVFS.Service should copy ProjectedFSLib.dll when it's missing and the installed ProjFS version matches the packaged version
@derrickstolee
Copy link
Contributor

I installed this version on my machine. I'll use it a bit and track the background maintenance by watching the logs.

string gitBinPath = GVFSPlatform.Instance.GitInstallation.GetInstalledGitBinPath();
if (string.IsNullOrEmpty(gitBinPath))
{
errorMessage = $"nameof(this.TryInitialize): Unable to locate git installation. Ensure git is installed and try again.";
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

$"nameof(this.TryInitialize): should be $"{nameof(this.TryInitialize)}:

Guid enumerationId,
string filterFileName,
bool restartScan,
IDirectoryEnumerationResults results)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: These parameters should be indented another level

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like several of the updated callbacks in this file have this issue.

string gitBinPath = GVFSPlatform.Instance.GitInstallation.GetInstalledGitBinPath();
if (string.IsNullOrEmpty(gitBinPath))
{
error = $"nameof(this.TryInitializeUpgrader): Unable to locate git installation. Ensure git is installed and try again.";
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's another improperly interpolated string. Thanks @wilbaker for finding the first one.

Copy link
Member

@kewillford kewillford left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.


if (randomValue <= reminderFrequency && ProductUpgraderInfo.IsLocalUpgradeAvailable(tracer: null))
if (randomValue <= reminderFrequency &&
ProductUpgraderInfo.IsLocalUpgradeAvailable(tracer: null, gvfsDataRoot: GVFSHooksPlatform.GetDataRootForGVFS()))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a search and this method is only called here and is passing null for the tracer so it really doesn't need to take the tracer if it is never going to be non-null

@jrbriggs jrbriggs merged commit 56e557b into releases/shipped May 6, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.