Skip to content

Module system improvements#6794

Merged
nbp merged 18 commits into
NixOS:masterfrom
nbp:module-system-tweaks
Apr 3, 2015
Merged

Module system improvements#6794
nbp merged 18 commits into
NixOS:masterfrom
nbp:module-system-tweaks

Conversation

@nbp

@nbp nbp commented Mar 13, 2015

Copy link
Copy Markdown
Member

This branch change mainly 2 things, the way composite types are processed, and the way internal processing of modules is done.

  • Property evaluation within composite types: This change is a highly demanded feature, which tend to confuse everybody who used properties with submodules in the past. Composite types such as types.attrsOf, types.loaOf and types.listOf are now able to have properties within their bodies. The most interesting use-case involve submodule, such as jobs or file systems options:

    {config, lib, ...}:
    
    with lib;
    
    {
      options = { … };
    
      config = {
        jobs.ircSession = mkIf config.….enable {
          …
        };
      };
    }
    

    This example used to evaluate a submodule with an empty attribute set for unique submodule. With this change to the evaluation of types, which evaluate properties while merging the types, this problem no longer exists as it is equivalent to having no value to be merged. This implies that no submodule would be evaluated, and no empty job would be created.

  • Argument unification between modules and submodules: This change is a long standing sanity issue which mostly prevented one person to use modules as submodules. Arguments are now processed by an internal option _module.args, which is given to each module. This implies that any set of modules which can be build in a standalone manner (such as maybe NixUP one day) can also be included as a submodule even if it relies on extra arguments, as long as these arguments are used for the computation of the definitions only.

    This open the door for a feature which I have not yet heard many people mention, but that I wanted to add for a while, which is to share library functions to build the content of option definitions. For example, the activation script logic can move to the activation script module instead of being part of nixpkgs.

    {config, lib, ...}:
    
    {
      _module.args.activationLib = {
        mkActivationWithDep = deps: str: …;
      };
    }
    

    In addition, to this change, the computation of the pkgs argument of NixOS modules moved from being computed out-side the module system to be computed within the module system. This implies that nixos/lib/eval-config.nix no longer has to execute the module system twice. The first one with a dummy pkgs argument, and the second time with the one computed from the first iteration. This approach is better as it avoid this work-around.

Sadly this last change is coming with a few non-backward compatible changes, which are either not well known, or easy to fix:

  • mkOption, mkIf: These functions are part of the engine of the module system. As the pkgs argument is no longer a special argument, using pkgs.lib.mkOption will result in an infinite loop during the evaluation. Thus module should be written as { config, pkgs, lib, ... }: instead of { config, pkgs, ... }:, pkgs can still be kept for finding packages, but functions should use the lib argument. Thus pkgs.lib.mk… should be replaced by lib.mk…, and identically with pkgs.lib; should be replaced by with lib;.

  • Deprecate the modulePath argument: This attribute is not well known, nor used. It was introduced before the addition of the NIX_PATH environment variable as far as I can recall. From what I can tell, the only purpose of it was to refer to nixos modules which are not included by default within your own the configuration files. This can be done as follow:

    {
       imports = [ <nixos/modules/profiles/headless.nix> ];
    }
    
  • Explicit argument of modules: Due to the strictness of the deconstruction of attribute set, modules arguments are computed in a way which is similar to the callPackage function. This implies that one cannot write a module which expects any argument custom argument — other than config, lib and options — which is not explicitly named.

    { ... }@args:
    
    {
      foo.description = "${args.name} thing";
    }
    

    This example lead to the fact that name is not found on the attribute set args. On the other hand, if the attribute is made explicit, then it would be visible. This is counter intuitive, but hopefully this is not a common pattern.

    { name, ... }@args:
    
    {
      foo.description = "${args.name} thing";
    }
    

This work is based on the work made by @shlevy in pull request #2563. Reviewed carefully, rebased, and fixed.

shlevy and others added 15 commits March 12, 2015 23:42
This makes the relationship between property types clearer, and more
importantly will let option types parameterized by other option types
reuse the code for delegated type checking and merging.
This simplifies typechecking and allows properties to be used inside the lists
This simplifes typechecking and allows properties to be used inside of
the attribute sets.

This fixes the empty synergy-client and synergy-server services
previously generated on systems with synergy disabled.
This symplifies typechecking and allows properties to be used inside the
function body. It also makes possible checking the type of the result.
This allows for module arguments to be handled modularly, in particular
allowing the nixpkgs module to handle the nixpkgs import internally.
This creates the __internal option namespace, which should only be added
to by the module system itself.
Ideally the module system could be configured pretty much completely by
the contents of the modules themselves, so add comments about avoiding
complicating it further and possibly removing now-redundant
configurability from the existing interface.
With the new evaluation of arguments, pkgs is now defined by the
configuration, which implies that option declaration with pkgs.lib
will cause an infinite loop.
…he arguments.

The current implementation of the ApplyIfFunction is looking at the
arguments of a module to decide which arguments should be given to each
module.  This patch make sure that we do not wrap a submodule function in
order to keep functionArgs working as expected.
This move idioms which were used in `evalOptionValue` and in the `merge`
functions of `listOf` and `attrsOf` types, such that we can use a names such
as `isDefined` and `optionalValue` instead or repeating identical
comparisons of `defsFinal == []`.
@nbp nbp mentioned this pull request Mar 14, 2015
Comment thread lib/modules.nix Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should be _module now

@shlevy

shlevy commented Mar 14, 2015

Copy link
Copy Markdown
Member

👍

@cstrahan

Copy link
Copy Markdown
Contributor

Wow, this looks pretty nice! 👍

@offlinehacker

Copy link
Copy Markdown
Contributor

+1

Did i read correctly, that with this patch you can use modules as submodules? Can you please give an example config for that?

@nbp

nbp commented Mar 15, 2015

Copy link
Copy Markdown
Member Author

@offlinehacker , Indeed I have not tried it yet, but something like the following module will let you embed NixOS modules in something which let you enumerate a set of computer configuration as NixOps does:

{ lib, ... }:

with lib;

{
  options.machines = mkOption {
    type = types.attrsOf (types.submodules {
      imports = import <nixos/modules/module_list.nix>;
      options = {};
      config = {
         _module.args.system = mkDefault builtins.currentSystem;
      };
    });
    default = {};
    description = "List of NixOS configurations.";
  };
}

Even if this is one step in the right direction, this set of patches lacks one feature to make the services option an alias of the service abstraction layer, the same problem exists for the multi-instances issues.

@nbp nbp force-pushed the module-system-tweaks branch from 06be675 to 05e8a48 Compare March 15, 2015 13:46
@thoughtpolice thoughtpolice added 0.kind: enhancement Add something new or improve an existing system. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS labels Mar 15, 2015
@nbp

nbp commented Mar 16, 2015

Copy link
Copy Markdown
Member Author

The module system test cases can be tested by using the following command:

$ nix-build ./pkgs/top-level/release.nix -A lib.tests --no-out-link

which prints the following:

these derivations will be built:
  /nix/store/1cy0mxcnriwi6mam565v8k8j6d62lfmh-nixpkgs-lib-tests.drv
building path(s) ‘/nix/store/vm40zm9mb8019y18ggrly32lfrg2rlsw-nixpkgs-lib-tests’
====== module tests ======
31 Pass
0 Fail
/nix/store/vm40zm9mb8019y18ggrly32lfrg2rlsw-nixpkgs-lib-tests

@nbp nbp self-assigned this Mar 16, 2015
@edolstra

Copy link
Copy Markdown
Member

I'll try to have a look at this later this week.

@offlinehacker

Copy link
Copy Markdown
Contributor

@nbp nice, thanks :)

coreyoconnor added a commit to coreyoconnor/nix_configs that referenced this pull request May 3, 2015
@nbp nbp added the 9.needs: changelog This PR needs a changelog entry label May 12, 2015
offlinehacker added a commit to xtruder/nixpkgs that referenced this pull request Jun 8, 2015
This reverts commit 889f72b, reversing
changes made to a8d0614.

Conflicts:
	nixos/modules/misc/nixpkgs.nix
@domenkozar

Copy link
Copy Markdown
Member

@nbp will you add these changes to the changelog?

@nbp

nbp commented Sep 30, 2015

Copy link
Copy Markdown
Member Author

I think I already made a branch for adding these to the release notes … I will try to find it again.

@nbp

nbp commented Sep 30, 2015

Copy link
Copy Markdown
Member Author

Indeed I had a branch, but I might have forgot to merge it:
master...nbp:doc-6794

I will open a pull request and cc you.

@domenkozar

Copy link
Copy Markdown
Member

Thanks (would be awesome if you can do that now, I want to release today)

On Wed, Sep 30, 2015 at 9:07 PM, Nicolas B. Pierron <
notifications@github.com> wrote:

Indeed I had a branch, but I might have forgot to merge it:
master...nbp:doc-6794
master...nbp:doc-6794

I will open a pull request and cc you.


Reply to this email directly or view it on GitHub
#6794 (comment).

domenkozar added a commit that referenced this pull request Sep 30, 2015
Add pkgs module argument documentation for #6794 incompatible change.
nbp added a commit that referenced this pull request Sep 30, 2015
(cherry picked from commit 50146ce)
Signed-off-by: Domen Kožar <domen@dev.si>
Shados added a commit to Shados/nix-config-shared that referenced this pull request Aug 28, 2017
wizeman added a commit that referenced this pull request Oct 30, 2017
The latest stable version was failing with infinite recursion due to #6794.
I've updated the unstable version to the latest commit to catch recent fixes.
@ncfavier

Copy link
Copy Markdown
Member

This makes (listOf (attrsOf str)).check [ "foo" ] evaluate to true, was that intended? This is a problem because then check-meta.nix fails to fail when meta.maintainers is not in the right format (see NixOS/nixos-search#391 for discussion).

Would it be too much of a performance impact to reintroduce the correct checks for listOf and attrsOf? In this case we might simply make check-meta.nix aware of this by performing deeper checks.

cc @sternenseemann

Comment thread lib/types.nix
listOf = elemType: mkOptionType {
name = "list of ${elemType.name}s";
check = value: isList value && all elemType.check value;
check = isList;

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, but why the hell was this done? Please explain otherwise I am going to revert this. (Same for all other changed check functions)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think one reason might be that, with this change, we get more localised errors, for example

A definition for option `foo.[definition 1-entry 2]' is not of type `bar'.

instead of

A definition for option `foo' is not of type `list of bar'.

and similarly for attrsOf. I think we want to keep this behaviour.

We should also be careful not to duplicate work, as type checking is currently done by mergeDefinitions.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So perhaps the solution is to stop using type.check externally, instead call mergeDefinitions with a single definition (maybe make a checkType wrapper around that).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(And whatever we do, this really needs more documentation, for example the manual doesn't mention that type.check only checks one "layer" of types.)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the input. How about type checking the meta attribute as an entire module system like in NixOS, instead of doing things manually?

      attrs = (lib.evalModules {
        modules = [ {
          options = metaTypes;
          config = meta;
        } ];
      }).config;

This allows us to leverage all of the existing infrastructure, and the only thing that needs to change for that is to modify metaTypes to become a "module" instead of just a type declaration.

(Alternatively, how would checkType look like? How do I even use mergeDefinitions?)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm sorry, but why the hell was this done? Please explain otherwise I am going to revert this. (Same for all other changed check functions)

Previous attempts at fixing this:

I think there are several others.

@sternenseemann

Copy link
Copy Markdown
Member

In this case we might simply make check-meta.nix aware of this by performing deeper checks.

I think doing the check manually (and correctly) in check-meta.nix is the way to go. Reverting this change 7 years later is not a good idea: I know from experience that properly type checking in Nix incurs a significant performance penalty, so I'd expect a revert to cause a very noticeable performance regression in the already slow module system. The typechecking limitation should probably be communicated better, but having parametrized types makes sense for documentation purposes.

adrianpk added a commit to adrianpk/nixpkgs that referenced this pull request May 31, 2024
…ange.

(cherry picked from commit 50146ce)
Signed-off-by: Domen Kožar <domen@dev.si>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

0.kind: enhancement Add something new or improve an existing system. 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 9.needs: changelog This PR needs a changelog entry

Projects

None yet

Development

Successfully merging this pull request may close these issues.