Skip to content

Add script for auto-formatting and validating board lists in Makefiles#10111

Closed
x3ro wants to merge 7 commits intoRIOT-OS:masterfrom
x3ro:pr/format-board-lists
Closed

Add script for auto-formatting and validating board lists in Makefiles#10111
x3ro wants to merge 7 commits intoRIOT-OS:masterfrom
x3ro:pr/format-board-lists

Conversation

@x3ro
Copy link
Copy Markdown
Contributor

@x3ro x3ro commented Oct 4, 2018

Contribution description

Add script for auto-formatting BOARD_INSUFFICIENT_MEMORY lists, as proposed in #9965. I'll add the finishing touches to this once there is a consensus that this is the desired format. A follow-up PR could then format all boards and add a check to Murdock.

I did not end up using @vincent-d's solution proposed in #6818 because I'm terrified of awk scripts 😅

Testing procedure

Run the script with either validate or format as parameter.

Todo

  • BOARD_INSUFFICIENT_MEMORY
  • BOARD_BLACKLIST
  • BOARD_WHITELIST
  • How to handle special cases?

Issues/PRs references

Fixes #9965
Fixes #6818
Related to #9983

@x3ro x3ro added State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet Area: build system Area: Build system Type: cleanup The issue proposes a clean-up / The PR cleans-up parts of the codebase / documentation Area: tools Area: Supplementary tools labels Oct 4, 2018
@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Oct 4, 2018

@MrKevinWeiss @kaspar030 In regards to the "one board per line" vs. "80 characters per line" question, I favor one board per line because it results in much cleaner diffs. For example, in the 80 characters per line case, if I wanted to change the first board in a list, this would likely change all following lines of the list in the diff.

@jnohlgard
Copy link
Copy Markdown
Member

I agree with one line per list item in general for the improved readability in the diffs.

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

Should we first merge #9965? In master not everything is structured in the same way so I had to manually edit some things. I only stalled because nobody seemed to want one item per line.
Just pay attention to cbor BLACKLIST, libfixmath_unittests BLACKLIST and some other ones that are not alphabetical.

@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Oct 5, 2018

Hmm, #9965 doesn't seem to be a PR. Did you mean #9983?

@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Oct 5, 2018

I ran my script against the PR with your changes from #9983, and my script would make some additional changes to board lists that fit on one line. You've left them as one line entries, whereas my script is strict in that it only allows one board per line no matter the length.

Either way I see no harm in merging that PR first :D

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

#9983 is right, don't know where I got that number from...

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

Ya a few that I missed (one liners). Hopefully I can get this pushed through if this is the way we want to go.

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

Looks like we should merge your script with all the changes that it does. I can do a review when you are ready!

@jnohlgard
Copy link
Copy Markdown
Member

Tiny formatting changes that I would like to discuss:
Putting the first item on a separate line from the variable, ending the last line with a backslash, and adding an empty comment line below would make the whole list homogeneous, no special cases for any item in the list.

so instead of:

BOARD_INSUFFICIENT_MEMORY := arduino-duemilanove \
                             arduino-uno \
                             nucleo-f031k6

we would get:

BOARD_INSUFFICIENT_MEMORY := \
  arduino-duemilanove \
  arduino-uno \
  nucleo-f031k6 \
# end BOARD_INSUFFICIENT_MEMORY

@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Oct 8, 2018

Okay, the script now formats all three variables. A couple of corner cases popped up, however:

Handling comments after a specific board would make the script quite a bit more complex, so I'm kinda hoping there's a way around it :D In such a case I suppose it'd be easy to just put it on the preceding line:

# tests/socket_zep/Makefile:

BOARD_WHITELIST = native    # socket_zep is only available on native

These here are a tad more complicated I suppose:

# tests/mpu_stack_guard/Makefile:

BOARD_WHITELIST += arduino-due      # cortex-m3
BOARD_WHITELIST += arduino-zero     # cortex-m0plus
BOARD_WHITELIST += cc2538dk         # cortex-m3
BOARD_WHITELIST += cc2650stk        # cortex-m3
BOARD_WHITELIST += ek-lm4f120xl     # cortex-m4f
BOARD_WHITELIST += f4vi1            # cortex-m4f

But could maybe be formatted as:

 # cortex-m3
BOARD_WHITELIST += arduino-due \
                   cc2538dk \
                   cc2650stk

# cortex-m4f
BOARD_WHITELIST += ek-lm4f120xl \
                   f4vi

Finally, there are cases where the list is terminated with a line such as the following, presumably to have a cleaner diff in case the last board is modified. Here the question is if we'd want to introduce this everywhere, or remove it where we're currently doing it:

# examples/filesystem/Makefile:

BOARD_BLACKLIST :=  chronos \
                    msb-430 \
                    msb-430h \
                    telosb \
                    wsn430-v1_3b \
                    wsn430-v1_4 \
                    z1 \
                    #

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

If we manually fix the corner cases would that make things easier. From what I saw there aren't that many of them and hopefully once they are fixed they stay fixed. At least that is what the review process should enforce.

@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Oct 8, 2018

True, my question was also along the lines of "do we need these corner cases". But I suppose that is a "no", meaning the script would simply not allow comments on board list lines?

@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Oct 8, 2018

@gebart I'm fine with that, I just tried to adapt the script to the most common way currently in use. Who would need to be involved in a discussion to make more serious modifications to the current format?

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

I also found this one BOARD_PROVIDES_NETIF
same issues

@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Oct 13, 2018

Yep, I've put some more work into this, but moving the variables around in the Makefile has a bunch of annoying corner cases that I need to deal with, so its not yet ready for prime time. I'll try to get it done next week once I'm back from vacation :D

Copy link
Copy Markdown
Contributor

@jcarrano jcarrano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not too familiar with how the static test scrips work, so I'll leave the review as a comment.


def chdir_to_riot_root():
SCRIPT_DIR = os.path.dirname(os.path.realpath(__file__))
RIOT_DIR = os.path.realpath(SCRIPT_DIR + "/../../..")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd prefer a script that does not need to live in a specific location in order to work properly.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The thought behind this is that I the script is working-directory agnostic. I'm happy with adding an argument that specifies the target directory if that seems useful to anyone, but the default mode of operation, in my opinion, should be "just run it and it will do the correct thing". Given that the script is pretty quick, it's also not a problem to always run it over the entirety of the RIOT code base.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To expand a bit more: in the case of a general-purpose script, i.e. one that is not intended to live in a particular directory in the RIOT source tree, I'd agree with you. In this case, however, I personally value "it does the right thing automatically" higher than the potential flexibility of being able to pass the target directory.

Furthermore, being able to pass the target directory would make the script potentially more complicated with questions such as "does it also have to work in a directory that is not part of a git repository" or, if the answer is no, checking if I'm in a git repository, ...

return prefix + (" \\\n" + spacing).join(self.boards)


def get_affected_filenames(target_variable):
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it acceptable to have both the formatting logic and the file selection logic here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think I understand the comment, could you elaborate?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if I want to auto-format a single Makefile? For example, I want to fix a file before committing it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I see. I've been simply running it over the entire code base. It's not too slow, so it hasn't been bothering me. Also has the added benefit of me not being able to forget any Makefile I might've touched 😅

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are several static checks with the same issue, so it's not a huuuge problem if this script does the same, though at some point it would be nice to have them fixed, but that's an issue for another PR.

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

@x3ro Anything I can do to help the progress?

@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Nov 6, 2018

@x3ro Anything I can do to help the progress?

It's mostly done, I need to find time to clean it up. Should be feasible sometime this week.

@x3ro x3ro force-pushed the pr/format-board-lists branch 2 times, most recently from 022aaf5 to 7837b27 Compare November 11, 2018 10:08
@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Nov 11, 2018

@MrKevinWeiss @jcarrano changes made and applied to #10130

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

I like the output, if the inline comment and multiple += instead of := / are handled manually (as there are only two cases) then I think at least everyone can agree on the output of it.

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

@x3ro ping

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

As a worst case I would like #10130 fixed and merged.

@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Jan 7, 2019

Apologies for the delay, I haven't had time and motivation to pick this up in the last weeks, but I still want to get it merged. I'm aiming for before / at the next Hack 'n' Ack.

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

@x3ro Yup, just let me know if you need something from me!

@stale
Copy link
Copy Markdown

stale bot commented Aug 10, 2019

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

@stale stale bot added the State: stale State: The issue / PR has no activity for >185 days label Aug 10, 2019
@MrKevinWeiss
Copy link
Copy Markdown
Contributor

I would still like to see this or something like this in. If you don't want it I can take over?

@stale stale bot removed the State: stale State: The issue / PR has no activity for >185 days label Aug 12, 2019
@MrKevinWeiss MrKevinWeiss self-assigned this Aug 12, 2019
@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Sep 11, 2019

Yep I haven't been able to work on it in quite a while. I'll be at Hack'n'Ack this month, and will do my best to get it merged on that day.

@x3ro x3ro force-pushed the pr/format-board-lists branch from 7837b27 to 4f1110d Compare September 25, 2019 20:34
@x3ro
Copy link
Copy Markdown
Contributor Author

x3ro commented Sep 25, 2019

So I've rebased and gone over the output again. The following seem to be the only files that are not formatted correctly yet:

  • tests/cortexm_common_ldscript/Makefile (many += to append to list)
  • tests/driver_cc110x/Makefile (many += to append to list)
  • tests/mpu_stack_guard/Makefile (many += to append to list, comments)
  • tests/sys_crypto/Makefile (many += to append to list)
  • tests/unittests/Makefile (many += to append to list)

I'll try to come up with a good fix for inline comments and propose to then go ahead and merge this PR, subsequently update #10130 and get that merged as well.

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

Very nice, thanks. This is back on my radar!

@stale
Copy link
Copy Markdown

stale bot commented Apr 28, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

@stale stale bot added the State: stale State: The issue / PR has no activity for >185 days label Apr 28, 2020
@stale
Copy link
Copy Markdown

stale bot commented Apr 28, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want me to ignore this issue, please mark it with the "State: don't stale" label. Thank you for your contributions.

@stale stale bot closed this May 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: build system Area: Build system Area: tools Area: Supplementary tools State: stale State: The issue / PR has no activity for >185 days State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet Type: cleanup The issue proposes a clean-up / The PR cleans-up parts of the codebase / documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BOARD_INSUFFICIENT_MEMORY alignment

4 participants