Skip to content

[WIP] First things for maturity and CD setup; IDE, Paket, FAKE#103

Closed
tomzo wants to merge 4 commits intoloic-sharma:masterfrom
ai-traders:ci
Closed

[WIP] First things for maturity and CD setup; IDE, Paket, FAKE#103
tomzo wants to merge 4 commits intoloic-sharma:masterfrom
ai-traders:ci

Conversation

@tomzo
Copy link
Contributor

@tomzo tomzo commented Sep 29, 2018

What does this PR do?

3 things:

  1. Adds reference to docker IDE image, which I have created earlier today. If you have local docker daemon then you can run any of the build tasks like this now ide <command>. E.g. ide ./build.sh will run msbuild publish and run tests with xunit.
  2. Moves dependency management to paket.
  3. Adds FAKE and draft of running tests CD-style.

Closes Issue(s)

Motivation and why

@loic-sharma I briefly spoke to you about some of this. You were asking what are benefits of paket and fake considering that msbuild is in place already.

Why paket:

  • paket gets consistent package versions across all projects and it can support modules for FAKE.
  • paket.dependencies gives you an quick overview of all packages used by application.
  • paket.lock gives you detailed info about each version of all direct and transitive dependencies.
  • I tried to install xunit.runner.console without using paket. But there is no such command on dotnet SDK. There is no nuget CLI for linux with works without mono, and I managed to keep away from mono so far.

Why FAKE:

  • we get consistent AssemblyVersion for all projects, from top of changelog file.
  • it provides single source of truth on what the release cycle and tasks involved.
  • fake is cross platform and can do much more than just msbuild.
  • programming tasks in xml is not easy and error prone. In F# FAKE you get as much as type safety in build scripts.
  • for all other task, which cannot be covered by msbuild or dotnet cli, we need a cross platform language. Otherwise windows guys start to code in powershell and linux guys in bash. We need something friendly to both platforms.
  • F# is cool

Why I am running tests with dotnet ...xunit.console.dll <assemblies-from-publish-dir>?
The main point in CD is that artifacts which constitute the application should be produced just once and then ran through all QA stages. We really need to be testing exactly the same files at each point until final release to ensure quality. In dotnet world that means using dotnet publish to force outputting application and all all its dependencies.
One of the common issues I had with dotnet assemlies is that running with dotnet test adapter uses additional assemblies from nuget caches. So I've seen tests passing when ran with dotnet test or dotnet xunit, but when same test is executed with dotnet ...xunit.console.dll <assemblies-from-publish-dir> it fails. Naturally same load errors happen in deployed application (after all tests have passed earlier...). So running tests with publish reproduces those errors and makes sure that we get load errors early.
PS: I actually ran into this already in BaGet and fixed it. The Microsoft.AspNetCore.App in BaGet.Tests was causing problems like this:

xUnit.net Console Runner v2.4.0 (64-bit .NET Core 4.6.26814.03)
System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Extensions.Logging.Abstractions, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Extensions.Logging.Abstractions, Version=2.1.1.0, Culture=neutral, PublicKeyToken=adb9793829ddae60'. The system cannot find the file specified.

Additional Notes

On the bad side, it runs on our private infrastructure and feedback will not be available to other contributors until we setup something. E.g. maybe server could post failures to the PR? @loic-sharma what do think?
I am not stressing that we (AIT) build it but I see no good CI provider which can give me a local docker host, which is necessary for e2e tests which I plan to add later.

Tests on debian stretch are fine after modifying LD_LIBRARY_PATH to include the directory with native extensions tests/BaGet.Tests/bin/Release/netcoreapp2.1/publish/runtimes/linux-x64/native/. Otherwise they fail with

System.DllNotFoundException : Unable to load shared library 'e_sqlite3' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libe_sqlite3: cannot open shared object file: No such file or directory
      Stack Trace:
           at Microsoft.Data.Sqlite.SqliteConnection..ctor(String connectionString)
...

Tests on alpine fail with a core dump, I couldn't fix it yet

xUnit.net Console Runner v2.4.0 (64-bit .NET Core 4.6.26814.03)
  Discovering: BaGet.Tests
  Discovered:  BaGet.Tests
  Starting:    BaGet.Tests
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForWellConfiguredDatabaseOptions [STARTING]
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForWellConfiguredDatabaseOptions [FINISHED] Time: 0.3717334s
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForNotConfiguredDatabaseOptions [STARTING]
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForNotConfiguredDatabaseOptions [FINISHED] Time: 0.0061826s
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForInvalidDatabaseType(databaseType: "<invalid>") [STARTING]
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForInvalidDatabaseType(databaseType: "<invalid>") [FINISHED] Time: 0.0032833s
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForInvalidDatabaseType(databaseType: "") [STARTING]
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForInvalidDatabaseType(databaseType: "") [FINISHED] Time: 0.0005202s
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForInvalidDatabaseType(databaseType: " ") [STARTING]
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForInvalidDatabaseType(databaseType: " ") [FINISHED] Time: 0.0004201s
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForWellConfiguredSqliteContext [STARTING]
    BaGet.Tests.ServiceCollectionExtensionsTest.AskServiceProviderForWellConfiguredSqliteContext [FINISHED] Time: 0.0025742s
    BaGet.Tests.PackageControllerTest.AskEmptyServerForNotExistingPackageID(packageID: "id01") [STARTING]
./tasks: line 8:  1601 Segmentation fault      (core dumped) dotnet $xunit_console tests/BaGet.Tests/bin/Release/netcoreapp2.1/publish/BaGet.Tests.dll -parallel none -maxthreads 1 -verbose

@loic-sharma can you check if you can run build.cmd on windows? I don't have any windows hosts and I just guessed how the script should look like.

This should provide sufficient environment for building and testing on
linux. Includes debian and alpine based images
For usage see https://github.com/ai-traders/docker-dotnet-ide
@tomzo tomzo force-pushed the ci branch 2 times, most recently from da9671a to f2d5867 Compare September 30, 2018 08:46
@tomzo tomzo changed the title First things for maturity and CD setup; IDE, Paket, FAKE [WIP] First things for maturity and CD setup; IDE, Paket, FAKE Sep 30, 2018
@tomzo
Copy link
Contributor Author

tomzo commented Oct 4, 2018

Opened #108 instead

@tomzo tomzo closed this Oct 4, 2018
0xced pushed a commit to 0xced/BaGet that referenced this pull request Mar 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Improve the docker build to use the release configuration and allow splitting off the database/web front end to allow scaling

1 participant