Issue description

While convenient, pkgs.nixosTests has surprising behavior. For example, this should fail, but it doesn't:

(import ./. { overlays = [ (self: super: { nginxStable = null; }) ]; }).nixosTests.nginx

So what is wrong here, is that the NixOS tests do not use the pkgs that they are passed. The NixOS tests assume that they can call Nixpkgs however they want. Although not a requirement for all tests, this means that in general the tests must reinvoke Nixpkgs, thus ignoring the Nixpkgs where .nixosTest was defined.

The documentation doesn't explain this ("Push NixOS tests inside the fixed point", which is misleading at best) and the implementation of nixosTests looks just fine, passing pkgs along as one would expect when reusing Nixpkgs for some tests. However, that is not what happens. pkgs will only be used to get qemu, lib, probably perl and perhaps some other things.

Why this is a problem: someone will use it to test whether NixOS, as built with their company overlay, still works. They will get false positives.

Steps to reproduce

nix-build --expr '(import ./. { overlays = [ (self: super: { nginxStable = null; }) ]; }).nixosTests.nginx'

Watch it succeed. It shouldn't.

Solutions

Ideally, we would pass pkgs to the NixOS machines. This can be done for some tests, but not for all the NixOS tests, because many depend on config.nixpkgs.config, which is unsupported by config.nixpkgs.pkgs (which is how you inject an existing Nixpkgs invocation).

So I don't see a way to make this work as one might predict. I suggest deleting pkgs.nixosTests.

@Ekleog do we really need this? It seems like just a convenience attribute.