Support Hexadecimal, Octal and Binary syntax by shikanime · Pull Request #7695 · NixOS/nix

shikanime · 2023-01-25T22:27:25Z

Motivation

Add support for integer literal syntax such as

0bXXXX for binary;
0xXXXX for hexadecimal;
0oXXXX for octal;

Context

         🌸＞ ' ' フ
         | . _  _l     < I'm just a regular person on the internet who doesn't have a
        ／` ミ＿xノ         lot of expertise in the Nix source code, but I saw a post about
       / .     |            someone wanting this certain feature and thought I'd chime in! 🌸
      /   ヽ . ﾉ
     │ . |  |  |
 ／￣| .  |  |  |
 | (￣ヽ＿_ヽ_)__)
 ＼二つ

fixes #7578

I don't know where to write the tests for the parser/lexer, feel free to comment on this!

Checklist for maintainers

Maintainers: tick if completed or explain if not relevant

agreed on idea
agreed on implementation strategy
tests, as appropriate
- functional tests - tests/**.sh
- unit tests - src/*/tests
- integration tests
documentation in the manual
code and comments are self-explanatory
commit message explains why the change was made
new feature or bug fix: updated release notes

sivizius · 2023-01-25T22:48:02Z

src/libexpr/lexer.l

 ID          [a-zA-Z\_][a-zA-Z0-9\_\'\-]*
 INT         [0-9]+
+BIN         0b[01]+
+OCT         0o[0-7]+


I suggest to change INT to [1-9][0-9]* and OCT to 0[0-7]*, so octal numbers start with 0 which is IMHO more common. On the other hand, this would break padding with leading zeros, but this looks better with spaces anyway IMHO. This would make 0 an octal 0, but it is still zero. The letter o is quite similar to 0, so 0o77 and 0077 look very similar and one might expect them to be both octal values, so at least add 0[0-7]+ as an alternative to 0o[0-7]+.

I don't think we should break 010 == 10, as Nix is meant to be a very stable language. It does seem like a good idea to warn about zero-padded integers, as the existence of octal support makes them quite ambiguous.
Aesthetics are of secondary importance, and explicit is good.

We should definitely deprecate 010 == 10, as numbers starting with 0 are usually read as octals for historic reasons. Perhaps a warning like »Deprecated syntax: If you intended to zero-pad a decimal value, consider to use spaces instead. If you meant to write an octal value, write 0o<value>. Values starting with 0 are ambiguous.«.

Another issue: Binary values tend to be quite long, so like in python, allowing _ to group e.g. bytes might be beneficial for readability, for decimals (e.g. multiple of 1000) or hexadecimals (e.g. words/dwords) and perhaps for floating point literals as well.

tests/eval.nix

src/libexpr/lexer.l

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

tests/eval.sh

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

roberth · 2023-02-21T15:16:17Z

I think we can potentially unblock this PR by running an experiment:

Add a deprecation warning to the syntax we'd like to steal ([ <number><identifier> ] - a list with at least these two consecutive items, without a separating space)
The warning refers to the issue, and asks users to report back if the change is a problem for them.

I think this should be discussed with the Nix team.
Question for the team: are we ok with running a deprecation experiment like this?

Click for possible future steps (not necessary to discuss yet; overly detailed)

Follow-up steps:

Change this PR to put the syntax change behind a feature flag
- If the feature flag is enabled, use the new syntax, as simple as that
- If the feature flag is not enabled, use the old syntax, but print a deprecation warning such as:
  Future ambiguous syntax found in file.nix:123, 0x100 We plan to reinterpret this syntax as a hexadecimal number in an upcoming release. We've estimated that this character sequence was highly unlikely to occur in practice, so if you would like for this file to be reproducible in future versions of Nix, please report back to us at .
Wait for reports. 1 year?
If nothing is reported, just steal the syntax. If something is reported, see if we can work around the issue by checking whether such a variable is in scope. Otherwise, this is blocked by a language version declaration feature, or we'd have to reconsider our backward compatibility policy

fricklerhandwerk · 2023-02-24T12:39:05Z

Triaged in the Nix team meeting:

@thufschmitt: technically this is a breaking change
- incidentally @yorickvP tried to address a very similar problem and stopped because of that, then started to work on an RFC to deal with breaking changes
- @Ericson2314: if we had proper language versioning it wouldn't be so much of an issue
- there would be some use cases for this syntax, seen this multiple times
@edolstra: this is nice to have, but not sure if there is a huge demand for non-decimal notation
- @fricklerhandwerk: depends on the audience you ask
@Ericson2314: intermediate goals (figuring out the breaking change story, and gaining experience as a team) are probably more valuable than the feature itself
agreement; postponed to after we deal with the overarching issue of breaking changes

nixos-discourse · 2023-02-24T13:02:04Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/2023-02-24-nix-team-meeting-minutes-35/25757/1

nyabinary · 2023-12-22T13:43:18Z

Triaged in the Nix team meeting:

* @thufschmitt: technically this is a breaking change
  
  * incidentally @yorickvP tried to address a very similar problem and stopped because of that, then started to work on an RFC to deal with breaking changes
  * @Ericson2314: if we had proper language versioning it wouldn't be so much of an issue
  * there would be some use cases for this syntax, seen this multiple times

* @edolstra: this is nice to have, but not sure if there is a huge demand for non-decimal notation
  
  * @fricklerhandwerk: depends on the audience you ask

* @Ericson2314: intermediate goals (figuring out the breaking change story, and gaining experience as a team) are probably more valuable than the feature itself

* agreement; postponed to after we deal with the overarching issue of breaking changes

Any update about this?

shikanime added 5 commits January 25, 2023 13:44

Add hexdecimal, octal and binary int syntax

a7c293c

Fix parser ordering

7f26fd5

Fix lexing

260cf24

Move type parsing to lexer

e301be5

Remove unused old tokens

6e4d10e

sivizius reviewed Jan 25, 2023

View reviewed changes

roberth reviewed Jan 26, 2023

View reviewed changes

tests/eval.nix Show resolved Hide resolved

roberth reviewed Jan 26, 2023

View reviewed changes

src/libexpr/lexer.l Show resolved Hide resolved

Rename bin to int2

4e06300

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

roberth reviewed Jan 26, 2023

View reviewed changes

tests/eval.sh Show resolved Hide resolved

shikanime and others added 3 commits January 26, 2023 21:35

Fix rename bin to int2

09b7f6c

Co-authored-by: Robert Hensing <roberth@users.noreply.github.com>

Add int literal overflow tests

3c014b7

Rename overflow with integer literal prefix

4994cca

fricklerhandwerk added language The Nix expression language; parser, interpreter, primops, evaluation, etc feature Feature request or proposal labels Feb 24, 2023

roberth added the breaking Changes we can't make without breaking old expressions, changing hashes, etc label Jun 1, 2023

roberth mentioned this pull request Jun 30, 2023

Make sure we can fix the parser's bad list/number interactions #8613

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support Hexadecimal, Octal and Binary syntax#7695

Support Hexadecimal, Octal and Binary syntax#7695
shikanime wants to merge 9 commits intoNixOS:masterfrom
shikanime:feat/support-hexa-octal-syntax

shikanime commented Jan 25, 2023 •

edited

Loading

Uh oh!

sivizius Jan 25, 2023

Uh oh!

roberth Jan 26, 2023

Uh oh!

sivizius Jan 26, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roberth commented Feb 21, 2023

Uh oh!

fricklerhandwerk commented Feb 24, 2023 •

edited

Loading

Uh oh!

nixos-discourse commented Feb 24, 2023

Uh oh!

nyabinary commented Dec 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

shikanime commented Jan 25, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Context

Checklist for maintainers

Uh oh!

sivizius Jan 25, 2023

Choose a reason for hiding this comment

Uh oh!

roberth Jan 26, 2023

Choose a reason for hiding this comment

Uh oh!

sivizius Jan 26, 2023

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

roberth commented Feb 21, 2023

Uh oh!

fricklerhandwerk commented Feb 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nixos-discourse commented Feb 24, 2023

Uh oh!

nyabinary commented Dec 22, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

shikanime commented Jan 25, 2023 •

edited

Loading

fricklerhandwerk commented Feb 24, 2023 •

edited

Loading