tests: various improvements for Octoprobe (WIP) #17782

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Draft

dpgeorge wants to merge 22 commits into micropython:master from dpgeorge:tests-improvements-for-octoprobe

+205 −57

Member

dpgeorge commented Jul 29, 2025 •

edited

Loading

Summary

This is a collection of various small improvements to the test suite to get Octoprobe to pass more of the tests.

Once fully working, these changes will be made into separate PRs.

Testing

To be tested by Octoprobe: https://reports.octoprobe.org/

Test report prior to this PR: https://reports.octoprobe.org/github_selfhosted_testrun_194/octoprobe_summary_report.html

Test report with this PR: https://reports.octoprobe.org/github_selfhosted_testrun_199/octoprobe_summary_report.html

dpgeorge added the tests label

codecov bot commented Jul 29, 2025 •

edited

Loading

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.38%. Comparing base (b87d73f) to head (fa51543).

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #17782   +/-   ##
=======================================
  Coverage   98.38%   98.38%           
=======================================
  Files         171      171           
  Lines       22298    22299    +1     
=======================================
+ Hits        21937    21938    +1     
  Misses        361      361

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from 18ad8c0 to 5958518 Compare

July 31, 2025 02:27

hmaerki mentioned this pull request

LOLIN_C3_MINI / RUN-TESTS_STANDARD / invalid pin octoprobe/testbed_micropython#34

Open

github-actions bot commented Jul 31, 2025 •

edited

Loading

Code size report:

Reference:  tools/mpy_ld.py: Write architecture flags to output natmod if needed. [b87d73f]
Comparison: tests/micropython/ringio_big.py: Print results at end of test. [merge of fa51543]
  mpy-cross:    +0 +0.000% 
   bare-arm:    +8 +0.014% 
minimal x86:   +14 +0.007% 
   unix x64:   +24 +0.003% standard
      stm32:    +8 +0.002% PYBV10
      esp32:  +104 +0.006% ESP32_GENERIC[incl +16(data)]
     mimxrt:    +8 +0.002% TEENSY40
        rp2:    +8 +0.001% RPI_PICO_W
       samd:    +8 +0.003% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:   +16 +0.004% VIRT_RV32

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from c399b93 to ca374fa Compare

July 31, 2025 13:16

Contributor

hmaerki commented Jul 31, 2025 •

edited

Loading

@dpgeorge: Two easy to reproduce bugs which could be added to ths PR

dpgeorge force-pushed the tests-improvements-for-octoprobe branch 2 times, most recently from aee37bb to 8920560 Compare

August 1, 2025 04:05

hmaerki mentioned this pull request

LOLIN_D1_MINI: net_hosted/ssl_verify_callback.py octoprobe/testbed_micropython#49

Open

dpgeorge force-pushed the tests-improvements-for-octoprobe branch 3 times, most recently from 66db214 to 3993063 Compare

August 10, 2025 02:13

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from ec94811 to f7dc897 Compare

August 13, 2025 09:22

This was referenced Aug 14, 2025

run-tests.py: returncode after error: could not enter raw REPL octoprobe/testbed_micropython#63

Closed

run-natmodtests.py: octoprobe/testbed_micropython#64

Open

ESP32_S3_DEVKIT: Failed to flash the firmware. Is USB connected octoprobe/testbed_micropython#65

Open

dpgeorge force-pushed the tests-improvements-for-octoprobe branch 2 times, most recently from 941611d to 4d60e1c Compare

August 16, 2025 14:53

dpgeorge force-pushed the tests-improvements-for-octoprobe branch 2 times, most recently from 96642d5 to 87300a8 Compare

August 26, 2025 07:17

This was referenced Aug 26, 2025

nrf/modules/machine/spi: Print SPI baudrate, polarity and phase. #17984

Merged

esp8266/main: Decrease usable stack by 64 bytes. #17986

Merged

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from 87300a8 to 35f3163 Compare

September 8, 2025 03:01

hmaerki mentioned this pull request

Use new --test-instance argument consistently in all test runners octoprobe/testbed_micropython#68

Merged

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from 35f3163 to a82b761 Compare

September 22, 2025 00:33

This was referenced Sep 23, 2025

ADA_ITSYBITSY_M0: misc/rge_sm.py: MemoryError: memory allocation failed octoprobe/testbed_micropython#69

Open

RUN-TESTS_STANDARD: ARDUINO_NANO_33: extmod/machine_soft_timer.py: TypeError: extra keyword arguments given octoprobe/testbed_micropython#70

Open

RUN-TESTS_STANDARD: ARDUINO_NANO_33: extmod/time_res.py: TypeError: 'module' object isn't callable octoprobe/testbed_micropython#71

Open

RUN-TESTS_STANDARD: LOLIN_D1_MINI: stress/list_sort.py: ... octoprobe/testbed_micropython#72

Open

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from 011aa7c to 64c8c24 Compare

September 26, 2025 05:21

This was referenced Oct 7, 2025

tests/extmod/time_res.py: Properly skip functions not in time module. #18219

Merged

tests/run-tests.py: Allow port:<dev> specification in run-multitests.py. #18220

Merged

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from 62f5dcf to 39ba8fe Compare

November 18, 2025 14:17

dpgeorge mentioned this pull request

Adjust Pins used to allow tests on ESP32-C3. #18386

Merged

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from 39ba8fe to e143b45 Compare

November 19, 2025 02:25

dpgeorge added 22 commits

January 13, 2026 11:35


          tests/extmod/machine_spi_rate.py: Support samd with default SPI.

af70616

Signed-off-by: Damien George <damien@micropython.org>


          tests/extmod: Support ESP32-C3 targets.

204449a

Some boards define ESP32C3, some define ESP32-C3.

Probably it should just use default pins, eg SPI(1).

Signed-off-by: Damien George <damien@micropython.org>


          tests/extmod/machine_spi_rate.py: Use default pins on ESP32.

c195eab

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-perfbench.py: Skip bm_hexiom on small targets.

ab5eb0b

Signed-off-by: Damien George <damien@micropython.org>


          tests/perf_bench: Skip tests when vfs doesn't exist.

02c453b

Eg for ESP8266-512K.

Signed-off-by: Damien George <damien@micropython.org>


          tests/float/string_format_modulo2_intbig.py: Get working on esp8266.

974dd97

In --via-mpy --emit native mode.

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-perfbench.py: Skip tests that have a MemoryError.

5615dde

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-tests.py: Skip various tests on small SAMD targets.

2410dbf

Tested on ADAFRUIT_ITSYBITSY_M0_EXPRESS.

Signed-off-by: Damien George <damien@micropython.org>


          tests/multi_net: Skip tests on axTLS when needed.

86a249b

Signed-off-by: Damien George <damien@micropython.org>


          tests/multi_net/udp_data_multi.py: Reduce number of UDP groups.

62094f1

On bare-metal lwIP targets, there's only enough buffer space for 4 groups.

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-perfbench.py: Skip misc_mandel if target doesn't have complex.

1dae802

Signed-off-by: Damien George <damien@micropython.org>


          esp32/modules: Convert AttributeError to ImportError.

b5ecd8a

Signed-off-by: Damien George <damien@micropython.org>


          tests/ports/stm32/adcall.py: Print values if test ADC fails.

2ce2eb0

Signed-off-by: Damien George <damien@micropython.org>


          tests/extmod/machine_spi_rate.py: Support mimxrt and nrf ports.

ff66f5c

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-tests.py: Abort test if serial port fails at start.

86459b8

The test runner will abort with error code 2 if:
- the target could not be detected (enter_raw_repl failed at the
  detection stage)
- unittest is not installed but is needed

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-tests.py: Abort test run if enter_raw_repl fails many times.

2032cbf

Now, if a test fails and crashes a board to the point it no longer responds
(but the serial connection is still alive and able to read/write), it only
takes a maximum of 30 + 5 * 4 = 50 seconds to detect that state and exit
the test run (previously it would be 30 + 10 seconds * number of remaining
tests = potentially large).

Can test this by adding the following to a test:

    import time, micropython
    micropython.kbd_intr(-1)
    while 1:
        time.sleep(1)

Signed-off-by: Damien George <damien@micropython.org>


          esp8266/boards: Include unittest in 512k board variant.

17226df

Signed-off-by: Damien George <damien@micropython.org>


          tests/run-natmodtests.py: Abort test run if target fails.

dec4717

Signed-off-by: Damien George <damien@micropython.org>


          tests/misc/rge_sm.py: Use list.append instead of list += [...].

3d2e50a

To reduce heap memory use.

Signed-off-by: Damien George <damien@micropython.org>


          py/objlist: Make list append allocation more gentle.

6b32fd7

Not sure if this is actually better, may lead to more fragmentation.

Probably needs some tests.

See micropython#4087

Really need to retry alloc with smaller amount if it fails, but then
really we need a lazy malloc, per the TODO in py/misc.h.

Signed-off-by: Damien George <damien@micropython.org>


          tests/extmod/machine_soft_timer.py: Skip on nrf boards.

e0500ee

They don't have the `freq` argument.

Signed-off-by: Damien George <damien@micropython.org>


          tests/micropython/ringio_big.py: Print results at end of test.

fa51543

So it can be properly skipped.

Signed-off-by: Damien George <damien@micropython.org>

dpgeorge force-pushed the tests-improvements-for-octoprobe branch from e143b45 to fa51543 Compare

January 13, 2026 00:41

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

tests