Skip to content

dist/pythonlibs/testrunner: add reset option, reset before cleanterm#12520

Closed
fjmolinas wants to merge 8 commits intoRIOT-OS:masterfrom
fjmolinas:pr_dont_reset
Closed

dist/pythonlibs/testrunner: add reset option, reset before cleanterm#12520
fjmolinas wants to merge 8 commits intoRIOT-OS:masterfrom
fjmolinas:pr_dont_reset

Conversation

@fjmolinas
Copy link
Copy Markdown
Contributor

@fjmolinas fjmolinas commented Oct 21, 2019

Contribution description

For some board make reset is only possible if a serial connection is not already open or its execution might disrupt it. This causes some tests to fail since before running a test the board
is reset.

This PR introduces two changes to address this:

  • reset before make cleanterm to ensure the application is always reset so one could eventually call make test multiple times.
  • make resetting after make cleanterm (which used as a sync feature) optional with TESTRUNNER_RESET_SYNC

I include both changes in the same PR because I think it is important to always rest the board before flashing.

This is not enough to run all tests with boards like remote-revb since in absence of a reset there is no way to sync the test and the device. If also using #12461, tests failing go down to 14 (from 100%).

Testing procedure

Since the default behavior is the same testing can be performed on boards that do need this feature, e.g.: remote-revb:

  • In master or setting TESTRUNNER_RESET_SYNC=1
TESTRUNNER_RESET_SYNC=1 make -C tests/xtimer_usleep BOARD=remote-revb flash test
make: Entering directory '/home/francisco/workspace/RIOT/tests/xtimer_usleep'
Building application "tests_xtimer_usleep" for "remote-revb" with MCU "cc2538".

"make" -C /home/francisco/workspace/RIOT/boards/remote-revb
"make" -C /home/francisco/workspace/RIOT/boards/common/remote
"make" -C /home/francisco/workspace/RIOT/core
"make" -C /home/francisco/workspace/RIOT/cpu/cc2538
"make" -C /home/francisco/workspace/RIOT/cpu/cc2538/periph
"make" -C /home/francisco/workspace/RIOT/cpu/cortexm_common
"make" -C /home/francisco/workspace/RIOT/cpu/cortexm_common/periph
"make" -C /home/francisco/workspace/RIOT/drivers
"make" -C /home/francisco/workspace/RIOT/drivers/periph_common
"make" -C /home/francisco/workspace/RIOT/sys
"make" -C /home/francisco/workspace/RIOT/sys/auto_init
"make" -C /home/francisco/workspace/RIOT/sys/div
"make" -C /home/francisco/workspace/RIOT/sys/isrpipe
"make" -C /home/francisco/workspace/RIOT/sys/newlib_syscalls_default
"make" -C /home/francisco/workspace/RIOT/sys/stdio_uart
"make" -C /home/francisco/workspace/RIOT/sys/test_utils/interactive_sync
"make" -C /home/francisco/workspace/RIOT/sys/tsrb
"make" -C /home/francisco/workspace/RIOT/sys/xtimer
   text	   data	    bss	    dec	    hex	filename
  12176	    136	   2680	  14992	   3a90	/home/francisco/workspace/RIOT/tests/xtimer_usleep/bin/remote-revb/tests_xtimer_usleep.elf
/home/francisco/workspace/RIOT/dist/tools/cc2538-bsl/cc2538-bsl.py -p "/dev/ttyUSB0" -e -w -v -b 115200 /home/francisco/workspace/RIOT/tests/xtimer_usleep/bin/remote-revb/tests_xtimer_usleep.bin
Opening port /dev/ttyUSB0, baud 115200
Reading data from /home/francisco/workspace/RIOT/tests/xtimer_usleep/bin/remote-revb/tests_xtimer_usleep.bin
Cannot auto-detect firmware filetype: Assuming .bin
Connecting to target...
CC2538 PG2.0: 512KB Flash, 32KB SRAM, CCFG at 0x0027FFD4
Primary IEEE Address: 00:12:4B:00:14:D5:2D:91
Erasing 524288 bytes starting at address 0x00200000
    Erase done
Writing 524288 bytes starting at address 0x00200000
Write 16 bytes at 0x0027FFF0F8
    Write done                                
Verifying by comparing CRC32 calculations.
    Verified (match: 0xf2c214f2)
/home/francisco/workspace/RIOT/dist/tools/pyterm/pyterm -p "/dev/riot/tty-remote-revb" -b "115200" --noprefix --no-repeat-command-on-empty-line
Twisted not available, please install it if you want to use pyterm's JSON capabilities
Connect to serial port /dev/riot/tty-remote-revb
Welcome to pyterm!
Type '/exit' to exit.
main(): This is RIOT! (Version: 2020.01-devel-177-g3802d-pr_dont_reset)
Running test 5 times with 7 distinct sleep times
Help: Press s to start test, r to print it is ready
r
r
r
  • This PR:
make -C tests/xtimer_usleep BOARD=remote-revb flash test
make: Entering directory '/home/francisco/workspace/RIOT/tests/xtimer_usleep'
Building application "tests_xtimer_usleep" for "remote-revb" with MCU "cc2538".

"make" -C /home/francisco/workspace/RIOT/boards/remote-revb
"make" -C /home/francisco/workspace/RIOT/boards/common/remote
"make" -C /home/francisco/workspace/RIOT/core
"make" -C /home/francisco/workspace/RIOT/cpu/cc2538
"make" -C /home/francisco/workspace/RIOT/cpu/cc2538/periph
"make" -C /home/francisco/workspace/RIOT/cpu/cortexm_common
"make" -C /home/francisco/workspace/RIOT/cpu/cortexm_common/periph
"make" -C /home/francisco/workspace/RIOT/drivers
"make" -C /home/francisco/workspace/RIOT/drivers/periph_common
"make" -C /home/francisco/workspace/RIOT/sys
"make" -C /home/francisco/workspace/RIOT/sys/auto_init
"make" -C /home/francisco/workspace/RIOT/sys/div
"make" -C /home/francisco/workspace/RIOT/sys/isrpipe
"make" -C /home/francisco/workspace/RIOT/sys/newlib_syscalls_default
"make" -C /home/francisco/workspace/RIOT/sys/stdio_uart
"make" -C /home/francisco/workspace/RIOT/sys/test_utils/interactive_sync
"make" -C /home/francisco/workspace/RIOT/sys/tsrb
"make" -C /home/francisco/workspace/RIOT/sys/xtimer
   text	   data	    bss	    dec	    hex	filename
  12176	    136	   2680	  14992	   3a90	/home/francisco/workspace/RIOT/tests/xtimer_usleep/bin/remote-revb/tests_xtimer_usleep.elf
/home/francisco/workspace/RIOT/dist/tools/cc2538-bsl/cc2538-bsl.py -p "/dev/ttyUSB0" -e -w -v -b 115200 /home/francisco/workspace/RIOT/tests/xtimer_usleep/bin/remote-revb/tests_xtimer_usleep.bin
Opening port /dev/ttyUSB0, baud 115200
Reading data from /home/francisco/workspace/RIOT/tests/xtimer_usleep/bin/remote-revb/tests_xtimer_usleep.bin
Cannot auto-detect firmware filetype: Assuming .bin
Connecting to target...
CC2538 PG2.0: 512KB Flash, 32KB SRAM, CCFG at 0x0027FFD4
Primary IEEE Address: 00:12:4B:00:14:D5:2D:91
Erasing 524288 bytes starting at address 0x00200000
    Erase done
Writing 524288 bytes starting at address 0x00200000
Write 16 bytes at 0x0027FFF0F8
    Write done                                
Verifying by comparing CRC32 calculations.
    Verified (match: 0xeafee2be)
/home/francisco/workspace/RIOT/dist/tools/pyterm/pyterm -p "/dev/riot/tty-remote-revb" -b "115200" --noprefix --no-repeat-command-on-empty-line
Twisted not available, please install it if you want to use pyterm's JSON capabilities
Connect to serial port /dev/riot/tty-remote-revb
Welcome to pyterm!
Type '/exit' to exit.
main(): This is RIOT! (Version: 2020.01-devel-178-g98935-pr_dont_reset)
Running test 5 times with 7 distinct sleep times
Help: Press s to start test, r to print it is ready
r
READY
s
START
Slept for 10064 us (expected: 10000 us) Offset: 64 us
Slept for 50064 us (expected: 50000 us) Offset: 64 us
Slept for 10298 us (expected: 10234 us) Offset: 64 us
Slept for 56844 us (expected: 56780 us) Offset: 64 us
Slept for 12186 us (expected: 12122 us) Offset: 64 us
Slept for 98829 us (expected: 98765 us) Offset: 64 us
Slept for 75063 us (expected: 75000 us) Offset: 63 us
Slept for 10063 us (expected: 10000 us) Offset: 63 us
Slept for 50064 us (expected: 50000 us) Offset: 64 us
Slept for 10298 us (expected: 10234 us) Offset: 64 us
Slept for 56844 us (expected: 56780 us) Offset: 64 us
Slept for 12185 us (expected: 12122 us) Offset: 63 us
Slept for 98828 us (expected: 98765 us) Offset: 63 us
Slept for 75064 us (expected: 75000 us) Offset: 64 us
Slept for 10063 us (expected: 10000 us) Offset: 63 us
Slept for 50064 us (expected: 50000 us) Offset: 64 us
Slept for 10298 us (expected: 10234 us) Offset: 64 us
Slept for 56843 us (expected: 56780 us) Offset: 63 us
Slept for 12185 us (expected: 12122 us) Offset: 63 us
Slept for 98828 us (expected: 98765 us) Offset: 63 us
Slept for 75063 us (expected: 75000 us) Offset: 63 us
Slept for 10063 us (expected: 10000 us) Offset: 63 us
Slept for 50064 us (expected: 50000 us) Offset: 64 us
Slept for 10298 us (expected: 10234 us) Offset: 64 us
Slept for 56844 us (expected: 56780 us) Offset: 64 us
Slept for 12185 us (expected: 12122 us) Offset: 63 us
Slept for 98828 us (expected: 98765 us) Offset: 63 us
Slept for 75064 us (expected: 75000 us) Offset: 64 us
Slept for 10063 us (expected: 10000 us) Offset: 63 us
Slept for 50064 us (expected: 50000 us) Offset: 64 us
Slept for 10298 us (expected: 10234 us) Offset: 64 us
Slept for 56843 us (expected: 56780 us) Offset: 63 us
Slept for 12185 us (expected: 12122 us) Offset: 63 us
Slept for 98829 us (expected: 98765 us) Offset: 64 us
Slept for 75063 us (expected: 75000 us) Offset: 63 us
Test ran for 1691591 us

make: Leaving directory '/home/francisco/workspace/RIOT/tests/xtimer_usleep'

Issues/PRs references

Part of #12448

For some board `make reset` is only possible if a serial connection
is not already open or its execution might disrupt it. This
casuses some tests to fail since before running a test the board
is reset. run(...) now has a reset option to disable this.

- The default value is `MAKE_RESET_WITH_TERM` env variable value
  which itself defaults to True.
- Some boards can't be resetted when the serial port is busy, e.g.
  when `make term` was called, since `make test` resets the device
  after opening a terminal this should be avoided.
@fjmolinas fjmolinas added Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation Area: tests Area: tests and testing framework labels Oct 21, 2019
@MrKevinWeiss
Copy link
Copy Markdown
Contributor

Another good idea. I think also some esps can benefit from this.

@MrKevinWeiss MrKevinWeiss added Reviewed: 1-fundamentals The fundamentals of the PR were reviewed according to the maintainer guidelines Reviewed: 2-code-design The code design of the PR was reviewed according to the maintainer guidelines labels Oct 21, 2019

# Some boards can't be resetted during test since this resets the serial
# connection. Default to 1 for those boards.
MAKE_RESET_WITH_TERM = int(os.environ.get('MAKE_RESET_WITH_TERM', 1))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about we turn this into a variable holding the whole command?
TESTRUNNER_RESET_CMD = "make reset"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I pushed changes adding make reset before clean term. I changed the variable name to TESTRUNNER_RESET_SYNC I think the expected behavior is more explicit with this name

I can still change it to TESTRUNNER_RESET_SYNC_CMD if you prefer.

@fjmolinas
Copy link
Copy Markdown
Contributor Author

Another good idea. I think also some esps can benefit from this.

Could you give me more details about the issues esp have, I don't have any but I'm interested in knowing all the corner cases we have :)

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

I think there was a problem just resetting while connected to the terminal. This would solve it. Of course @cladmi knows more but now that he is gone I can try to dig into it.

- Some boards can't be resetted when the serial port is busy, e.g.
  when `make term` was called, since `make test` resets the device
  after opening a terminal this should be avoided. Export a variable
  to `test` target alowing to disable reseting after the terminal is
  opened.
For some board `make reset` is only possible if a serial connection
is not already open or its execution might disrupt it. This
casuses some tests to fail since before running a test the board
is reset. The reset is not performed is TESTRUNNER_RESET_SYNC env
variable is set.
@fjmolinas fjmolinas changed the title dist/pythonlibs/testrunner: add reset option dist/pythonlibs/testrunner: add reset option, reset before cleanterm Oct 21, 2019
@fjmolinas
Copy link
Copy Markdown
Contributor Author

@MrKevinWeiss @kaspar030 I made some changes to the implementation after @kaspar030 comment. I updated the PR description accordingly. I basically also added a reset before cleanterm
that way even if the sync reset is disabled, the board is still reset before calling make test.

@fjmolinas
Copy link
Copy Markdown
Contributor Author

I basically also added a reset before cleanterm
that way even if the sync reset is disabled, the board is still reset before calling make test.

Note, keeping the second reset is temporary and only while tests are still depending on resetting after cleanterm.

@MrKevinWeiss
Copy link
Copy Markdown
Contributor

If I don't specify TESTRUNNER_RESET_SYNC=1 then it doesn't work

BOARD=nucleo-l433rc make flash test -C tests/pthread_tls/
Traceback (most recent call last):
  File "/home/kevinweiss/WorkingDirectory/RIOT/tests/pthread_tls/tests/01-run.py", line 4, in <module>
    from testrunner import run
  File "/home/kevinweiss/WorkingDirectory/RIOT/dist/pythonlibs/testrunner/__init__.py", line 15, in <module>
    from .spawn import find_exc_origin, setup_child, teardown_child
  File "/home/kevinweiss/WorkingDirectory/RIOT/dist/pythonlibs/testrunner/spawn.py", line 29, in <module>
    TESTRUNNER_RESET_SYNC = int(os.environ.get('TESTRUNNER_RESET_SYNC', 1))
ValueError: invalid literal for int() with base 10: ''
/home/kevinweiss/WorkingDirectory/RIOT/tests/pthread_tls/../../Makefile.include:711: recipe for target 'test' failed
make: *** [test] Error 1

@fjmolinas
Copy link
Copy Markdown
Contributor Author

If I don't specify TESTRUNNER_RESET_SYNC=1 then it doesn't work

Fixed, it wasn't handling an empty value properly.

Copy link
Copy Markdown
Contributor

@aabadie aabadie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments but needs more testing.

# default value (10)
TIMEOUT = int(os.environ.get('RIOT_TEST_TIMEOUT') or 10)


Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unrelated change (which introduces a pep8 btw)

DEBUGGER_FLAGS = $(BINDIR) $(ELFFILE)
OBJDUMPFLAGS += --disassemble --source --disassembler-options=force-thumb

# remote use cc2538-bsl attaching to the serial PORT to reset the board. If
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This comment is hard to read. Maybe (with typos fixed):
remote uses cc2538-bsl attached to the serial PORT to reset the board. If 'make reset' is called when the terminal is opened, the device switches to bootloader mode and the next commands will fail, leaving the device in bootloader mode. To avoid this problem, the use of reset as sync method is disabled.


# Resetting the board after the terminal is opened is used by the testrunner
# to sync the device (application under test), and the test script. This can't
# be done for some boards so provide a way of disabling the behavior.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a way to disable this behavior

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe also add: "This behavior is enabled by default" ?

# need to reset after opening the terminal because most tests use this as a
# way to sync the device and the test. As long as this is still the behavior
# we keep both resets.
try:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe move the try/except logic in a separate private function outside setup_child ? This would avoid some code duplication.

Why not only reset here if TESTRUNNER_RESET_SYNC is disabled ? Otherwise the device under test is reset twice is TESTRUNNER_RESET_SYNC is enabled.

@fjmolinas
Copy link
Copy Markdown
Contributor Author

Went with #12862 instead.

@fjmolinas fjmolinas closed this Jan 3, 2020
@fjmolinas fjmolinas deleted the pr_dont_reset branch June 9, 2022 08:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: tests Area: tests and testing framework Reviewed: 1-fundamentals The fundamentals of the PR were reviewed according to the maintainer guidelines Reviewed: 2-code-design The code design of the PR was reviewed according to the maintainer guidelines Type: enhancement The issue suggests enhanceable parts / The PR enhances parts of the codebase / documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants