Skip to content

[Linux] Add GVFS platform and build configuration for Linux#1125

Merged
chrisd8088 merged 24 commits intomicrosoft:features/linuxprototypefrom
github:linux-gvfs-provider
Jun 1, 2019
Merged

[Linux] Add GVFS platform and build configuration for Linux#1125
chrisd8088 merged 24 commits intomicrosoft:features/linuxprototypefrom
github:linux-gvfs-provider

Conversation

@chrisd8088
Copy link
Contributor

@chrisd8088 chrisd8088 commented May 8, 2019

We add the core GVFS.Platform.Linux classes, derived from their GVFS.Platform.Mac equivalents but with appropriate changes for syscall argument signatures and flag values, type definitions, and structure fields (e.g., mode_t is a uint, and the layout of struct stat is quite different).

In order to avoid conflicting with the GNOME Virtual file system (GVfs), we use .vfsforgit as the name of our per-repository private directories, instead of .gvfs, as is used on Windows and Mac.

The build tooling is largely a duplicate of the corresponding Mac configuration, with some changes to ensure we build successfully on Linux (e.g., testing for != Windows is no longer sufficient to determine that the platform is OS X, so we add distinct IsLinux and IsOSX build conditions to the .csproj file for GVFS.Upgrader, among other subprojects).

The GVFS.sln configuration changes should ensure that the ProjFS.Linux.Managed and GVFS.Platform.Linux subprojects are built on all platforms (and their Mac and Windows equivalents
are built on Linux), and that the GVFS.Service.Mac classes are built on Linux, so as to allow the GVFS.UnitTests subproject to build and run successfully on all three platforms.

We also build the GVFS.Hooks.Linux and GVFS.Mount.Linux subprojects on both Linux and Windows, just as their Mac counterparts are built on both Mac and Windows.

The GVFS.Service binary is, for the time being, only a stub on Linux, as we have yet to determine how (or if) we should handle auto-mounting repositories.

Note that for now the BuildGVFSForLinux.sh script does not perform any installation or packaging.

/cc @jrbriggs, @wilbaker

@chrisd8088 chrisd8088 force-pushed the features/linuxprototype branch from 0fa217b to 5288077 Compare May 8, 2019 23:28
@chrisd8088 chrisd8088 force-pushed the linux-gvfs-provider branch from deacb1d to 02fc9d4 Compare May 8, 2019 23:48
@chrisd8088 chrisd8088 added the WIP label May 8, 2019
@chrisd8088 chrisd8088 changed the title Add Linux GVFS platform [WIP] Add Linux GVFS platform May 8, 2019
@chrisd8088 chrisd8088 removed the WIP label May 9, 2019
@chrisd8088 chrisd8088 changed the title [WIP] Add Linux GVFS platform [Linux] Add Linux GVFS platform and build configuration May 9, 2019
@chrisd8088 chrisd8088 changed the title [Linux] Add Linux GVFS platform and build configuration [Linux] Add GVFS platform and build configuration for Linux May 9, 2019
@chrisd8088 chrisd8088 removed the request for review from wilbaker May 9, 2019 20:19
@chrisd8088 chrisd8088 force-pushed the features/linuxprototype branch from 5288077 to 47fb515 Compare May 9, 2019 21:43
@chrisd8088 chrisd8088 force-pushed the linux-gvfs-provider branch from 02fc9d4 to 6e457bc Compare May 9, 2019 21:46
@chrisd8088 chrisd8088 force-pushed the features/linuxprototype branch from 47fb515 to 972f05a Compare May 11, 2019 05:14
@chrisd8088 chrisd8088 force-pushed the linux-gvfs-provider branch from 6e457bc to af62389 Compare May 12, 2019 23:44
Copy link
Contributor

@kivikakk kivikakk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@chrisd8088 chrisd8088 force-pushed the linux-gvfs-provider branch 2 times, most recently from 6923bb4 to 54dc071 Compare May 13, 2019 07:50
@kivikakk kivikakk mentioned this pull request May 15, 2019
@chrisd8088
Copy link
Contributor Author

/azp run PR - Linux - Build and Unit Test

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@chrisd8088 chrisd8088 force-pushed the linux-gvfs-provider branch from e898f42 to aa19474 Compare May 17, 2019 18:52
@chrisd8088 chrisd8088 force-pushed the features/linuxprototype branch from 9f017e7 to e966c32 Compare May 21, 2019 21:32
@chrisd8088
Copy link
Contributor Author

@wilbaker -- I realize there's a lot of fresh code here, since we're adding a whole new platform to the GVFS provider. To that end we've tried to keep the sequence of commits as clean and logical as possible (hence all the rebasing), so you might find it easier to review by looking at the commit sequence, at least in part.

There are places where we have some code duplication between Mac and Linux, and that'll be the focus of another PR based on posix-projfs-virt-instance, which is still WIP. But our hope is that this PR is relatively coherent now as just adding what's required to get the GVFS provider to build and run on Linux.

@chrisd8088 chrisd8088 force-pushed the linux-gvfs-provider branch from 96512e8 to e33d831 Compare May 29, 2019 22:10
kivikakk and others added 7 commits May 29, 2019 15:17
A directory containing only ".git" and ".gitattributes" will be
an uninitialized GVFS clone, but MirrorProvider will have an empty
initial directory without any Git artifacts.
In order to prevent exceptions triggered by file projection
requests arriving prior to our exit from our wait loop in
our ProjFS.Linux StartVirtualizationInstance() method,
we only set the projfs variable to non-null at the tail end of
the startup sequence, and then in the HandleProjEvent() callback
(as well as several the placeholder update methods), we always
acquire a local copy and return immediately ENODEV if it is
null.

The placeholder update methods should, in theory, never be
called with a null this.projfs, because they are only executed
by threads started after StartVirtualizationInstance() and
stopped before StopVirtualizationInstance().  However, we
defensely check for a null projfs to be on the safe side.

Similarly, using a local copy ensures we can not trigger a
NullReferenceException should projfs happen to become
null after our check, although we do not expect this to be
possible, since the libprojfs/libfuse threads which execute
projection requests should all be stopped before our
StopVirtualizationInstance() method sets projfs to null.

Also remove a probably-impossible-but-who-knows race if
StopVirtualizationInstance was called twice concurrently.

Co-authored-by: Ashe Connor <ashe@kivikakk.ee>
We split the "Unix" methods into both Mac and Linux (as their magic
consts differ), and change the build process so we can rely on
conditionally-included projects, viz. the respective GVFS.Platform.*
projects.
These are unused, and aren't portable between macOS/Linux and so
shouldn't be used.
On both Linux and Mac/Darwin, the uid_t type is an
unsigned int, not a signed int.
Remove Windows builds of Linux projects where we don't
build them for Mac, and equivalently remove some Windows
builds of the corresponding Mac projects, specifically:

GVFS.Linux and GVFS.Mac
GVFS.Hooks.Linux and GVFS.Hooks.Mac
GVFS.Mount.Linux and GVFS.Mount.Mac

Co-authored-by: Chris Darroch <chrisd8088@github.com>
@chrisd8088 chrisd8088 force-pushed the linux-gvfs-provider branch from e33d831 to b8b89b5 Compare May 29, 2019 22:21
Copy link
Member

@wilbaker wilbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved with suggestions

{
excludeCategories.Add(Categories.LinuxOnly);
}

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • On Linux should we also be excluding Categories.MacOnly?
  • Could you incorporate these new exclusions in this if/else block just below?
if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
{
    excludeCategories.Add(Categories.MacTODO.FailsOnBuildAgent);
    excludeCategories.Add(Categories.MacTODO.NeedsNewFolderCreateNotification);
    excludeCategories.Add(Categories.MacTODO.NeedsGVFSConfig);
    excludeCategories.Add(Categories.MacTODO.NeedsDehydrate);
    excludeCategories.Add(Categories.MacTODO.NeedsServiceVerb);
    excludeCategories.Add(Categories.MacTODO.NeedsStatusCache);
    excludeCategories.Add(Categories.MacTODO.NeedsCorruptObjectFix);
    excludeCategories.Add(Categories.MacTODO.TestNeedsToLockFile);
    excludeCategories.Add(Categories.WindowsOnly);
}
else
{
    excludeCategories.Add(Categories.MacOnly);
}

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • We already exclude MacOnly because of that same following block, which adds MacOnly to the exclusion list if the platform is not OSPlatform.OSX.
  • I'm unsure on what you're asking with the second question here. Incorporate what new exclusions where?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • I'm unsure on what you're asking with the second question here. Incorporate what new exclusions where?

(To be clear, not trying to be obtuse! Just not parsing the question well.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My reading of @wilbaker's suggestion here is that because many of these functional tests have not yet been ported to the Mac (and are therefore Windows-only not by design, but only because of lack of time), we will want to, for now, also exclude these on Linux. I'll make that change shortly!

Copy link
Contributor Author

@chrisd8088 chrisd8088 May 31, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually think I'm going to leave this as-is for now, because we haven't yet run the functional tests on Linux (see github#16) and so we may find that we need a different, likely larger, set of LinuxTODO tests. Until we investigate, it's a little hard to know which ones from the MacTODO category will really be POSIX-TODO and apply to Linux as well, and which are Mac-specific.

But as I'm 100% certain we'll be revisiting these tests and categoroes very soon, I think I'll go ahead and merge with this as it stands, as a placeholder (projecting filesystem pun intended! :-)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kivikakk @chrisd8088 apologies for the confusion.

The idea in the second point was to combine the new if/else with the existing one below so that it would look something like:

if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
{
    excludeCategories.Add(... stuff excluded on Mac ...);
}
else if (RuntimeInformation.IsOSPlatform(Linux))
{
    excludeCategories.Add(... stuff excluded on Linux ...);
}
else
{
   excludeCategories.Add(... stuff excluded on Windows ...);
}

It can get tricky to keep track of what we're excluding or including, and it might be easier to keep track if we set them all in one spot.

Copy link
Member

@wilbaker wilbaker Jun 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To clarify, when I wrote excludeCategories.Add(... stuff excluded on <Platform> ...); I meant call excludeCategories.Add as many times as needed in each block to exclude what should be excluded on each platform

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, makes perfect sense. I'll do a PR for this -- thanks!

EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug.Linux|x64 = Debug.Linux|x64
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The good news is I was able to build your branch on Mac with Scripts/Mac/BuildGVFSForMac.sh. The bad news is that something is configured in a way that VS for Mac doesn't like, but that can get sorted out before this is merged to master.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, can you describe it at all? I just tried a reasonably fresh build with VS for Mac of this tree and everything worked OK.

I trashed BuildOutput, packages, Publish, ran enough of BuildGVFSForMac.sh to generate the constants and assembly version files, then built the solution. Everything looks fine.


I will note that I encounter issues regularly if I try to build with VS and then build script (or vice versa) without cleaning everything in between — VS gets confused, red-underlines everything, says System is undefined, etc. etc. Just to see if I could trigger it in this case, after building with VS per above, I ran the full BuildGVFSForMac.sh, and then again in VS, and this one time it actually works.

(Well, actually, I get the incompatible SDK error thing on two projects, but that's not what I'm talking about and is already known!)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, these are the errors I'm seeing:

Error: NuGet packages need to be restored before building. NuGet MSBuild targets are missing and are needed for building. The NuGet MSBuild targets are generated when the NuGet packages are restored. (GVFS.Common)

Error: NuGet packages need to be restored before building. NuGet MSBuild targets are missing and are needed for building. The NuGet MSBuild targets are generated when the NuGet packages are restored. (GVFS.Hooks.Mac)

Error: NuGet packages need to be restored before building. NuGet MSBuild targets are missing and are needed for building. The NuGet MSBuild targets are generated when the NuGet packages are restored. (GVFS.Tests)

I tried deleting BuildOutput, packages, and Publish but it didn't help.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to make my steps reproducible:

  • Start with VS closed.
  • Deleted BuildOutput, packages, Publish. The clone is at ~/microsoft/VFSForGit/src, and the VFSForGit directory is empty except for the clone at src.
  • Performed git clean -fdx in the clone.
  • Opened GVFS.sln in Visual Studio for Mac (Community, 8.0.4 build 0).
  • It automatically restored packages when opening the solution. (I'll omit the entire Package Console log, but the last line reads Packages successfully restored.)
  • Tried to build the solution. Failed because I have Debug.Windows selected and a bunch of "This project type is not supported by Visual Studio Community 2019 for Mac." messages appeared.
  • Switched to Debug.Mac. Failed because GVFSConstants.GitVersion.cs is not found.
  • Made these changes to BuildGVFSForMac.sh and ran the script to create the two generated .cs files.
  • Built the solution again.
  • ========== Build: 16 succeeded, 0 failed, 2 up-to-date, 7 skipped ==========
    
    Build successful.
    

It's possible the git clean step was necessary? I'm not sure.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh thanks, I did forget to run git clean, let me give that a try.

@chrisd8088 chrisd8088 merged commit 2063bf3 into microsoft:features/linuxprototype Jun 1, 2019
@chrisd8088 chrisd8088 deleted the linux-gvfs-provider branch June 1, 2019 01:37
chrisd8088 added a commit that referenced this pull request Jul 20, 2019
[Linux] Add GVFS platform and build configuration for Linux

We add the core GVFS.Platform.Linux classes, derived from their
GVFS.Platform.Mac equivalents but with appropriate changes for
syscall argument signatures and flag values, type definitions, and
structure fields (e.g., mode_t is a uint, and the layout of
struct stat is quite different).

In order to avoid conflicting with the GNOME Virtual file system
(GVfs), we use .vfsforgit as the name of our per-repository private
directories, instead of .gvfs, as is used on Windows and Mac.

The build tooling is largely a duplicate of the corresponding
Mac configuration, with some changes to ensure we build
successfully on Linux (e.g., testing for != Windows is no
longer sufficient to determine that the platform is OS X, so
we add distinct IsLinux and IsOSX build conditions to the
.csproj file for GVFS.Upgrader, among other subprojects).

The GVFS.sln configuration changes should ensure that
the ProjFS.Linux.Managed and GVFS.Platform.Linux
subprojects are built on all platforms (and their Mac and
Windows equivalents are built on Linux), and that the
GVFS.Service.Mac classes are built on Linux, so as to allow
the GVFS.UnitTests subproject to build and run successfully
on all three platforms.

We also build the GVFS.Hooks.Linux and GVFS.Mount.Linux
subprojects on both Linux and Windows, just as their Mac
counterparts are built on both Mac and Windows.

The GVFS.Service binary is, for the time being, only a stub
on Linux, as we have yet to determine how (or if) we should
handle auto-mounting repositories.

Note that for now the BuildGVFSForLinux.sh script does not
perform any installation or packaging.
chrisd8088 added a commit that referenced this pull request Jul 26, 2019
[Linux] Add GVFS platform and build configuration for Linux

We add the core GVFS.Platform.Linux classes, derived from their
GVFS.Platform.Mac equivalents but with appropriate changes for
syscall argument signatures and flag values, type definitions, and
structure fields (e.g., mode_t is a uint, and the layout of
struct stat is quite different).

In order to avoid conflicting with the GNOME Virtual file system
(GVfs), we use .vfsforgit as the name of our per-repository private
directories, instead of .gvfs, as is used on Windows and Mac.

The build tooling is largely a duplicate of the corresponding
Mac configuration, with some changes to ensure we build
successfully on Linux (e.g., testing for != Windows is no
longer sufficient to determine that the platform is OS X, so
we add distinct IsLinux and IsOSX build conditions to the
.csproj file for GVFS.Upgrader, among other subprojects).

The GVFS.sln configuration changes should ensure that
the ProjFS.Linux.Managed and GVFS.Platform.Linux
subprojects are built on all platforms (and their Mac and
Windows equivalents are built on Linux), and that the
GVFS.Service.Mac classes are built on Linux, so as to allow
the GVFS.UnitTests subproject to build and run successfully
on all three platforms.

We also build the GVFS.Hooks.Linux and GVFS.Mount.Linux
subprojects on both Linux and Windows, just as their Mac
counterparts are built on both Mac and Windows.

The GVFS.Service binary is, for the time being, only a stub
on Linux, as we have yet to determine how (or if) we should
handle auto-mounting repositories.

Note that for now the BuildGVFSForLinux.sh script does not
perform any installation or packaging.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants