Fix `NoMemoryError` problem with RBS tests + MSVC Ruby by kou · Pull Request #363 · test-unit/test-unit

kou · 2025-12-25T02:36:30Z

GitHub: Fix GH-358

If we refer @suite from @options, @suite may not be GC-ed because @options may be live longer than a TestRunner. For example, AutoRunner's @runner_options may be @options and AutoRunner may be live longer than TestRunner.

kou · 2025-12-25T02:45:23Z

Hmm... This is needed for parallel execution...

tikkss · 2025-12-25T12:19:18Z

Thanks for your investigation.
Hmm... Let’s think about how we can solve this even with parallel execution.

tikkss · 2025-12-27T00:27:30Z

CI shows that passing the full test suite in @options causes NoMemoryError on Windows.

`NoMemoryError` has occurred at master (`9c6fce7`)

https://github.com/tikkss/test-unit/actions/runs/20529781836/job/58979317764#step:11:2970

failed to allocate memory (NoMemoryError)

`NoMemoryError` has not occurred with the following diff

diff --git a/lib/test/unit/ui/testrunnermediator.rb b/lib/test/unit/ui/testrunnermediator.rb
index d17ac15..7cc0967 100644
--- a/lib/test/unit/ui/testrunnermediator.rb
+++ b/lib/test/unit/ui/testrunnermediator.rb
@@ -45,7 +45,6 @@ module Test
                 notify_listeners(channel, value)
               end
               @options[:event_listener] = event_listener
-              @options[:test_suite] = @suite
               @test_suite_runner_class.run_all_tests(result, @options) do |run_context|
                 catch do |stop_tag|
                   result.stop_tag = stop_tag

https://github.com/tikkss/test-unit/actions/runs/20530199053/job/58980369410

`NoMemoryError` has occurred with the following diff

diff --git a/lib/test/unit/ui/testrunnermediator.rb b/lib/test/unit/ui/testrunnermediator.rb
index d17ac15..85c406e 100644
--- a/lib/test/unit/ui/testrunnermediator.rb
+++ b/lib/test/unit/ui/testrunnermediator.rb
@@ -41,10 +41,6 @@ module Test
           start_time = Time.now
           begin
             with_listener(result) do
-              event_listener = lambda do |channel, value|
-                notify_listeners(channel, value)
-              end
-              @options[:event_listener] = event_listener
               @options[:test_suite] = @suite
               @test_suite_runner_class.run_all_tests(result, @options) do |run_context|
                 catch do |stop_tag|
@@ -76,7 +72,9 @@ module Test
             run
           else
             worker_context = WorkerContext.new(nil, run_context, result)
-            @suite.run(worker_context, &@options[:event_listener])
+            @suite.run(worker_context) do |channel, value|
+              notify_listeners(channel, value)
+            end
           end
         end

https://github.com/tikkss/test-unit/actions/runs/20530390681/job/58980819540#step:11:2608

failed to allocate memory (NoMemoryError)

tikkss

How about this?

NoMemoryError has not occurred:

https://github.com/tikkss/test-unit/actions/runs/20531829390/job/58984227952

@options[:test_suite] is no longer used except to emit events for the top-level test suite.
The code that is called when an event is emitted is shown below.

In this code, the @tests instance variable is not used, so I decided it is safe to remove it.
@tests stores all tests, so it uses a large amount of memory.

test-unit/lib/test/unit/ui/console/testrunner.rb

Lines 495 to 518 in 5914658

    
           def test_suite_started(suite) 
        
             last_test_suite = @test_suites.last 
        
             @test_suites << suite 
        
             if @top_level 
        
               @top_level = false 
        
               return 
        
             end 
        
             output_single(indent, nil, VERBOSE) 
        
             if suite.test_case.nil? 
        
               _color = color("suite") 
        
             else 
        
               _color = color("case") 
        
             end 
        
             prefix = "#{last_test_suite.name}::" 
        
             output_single(suite_name(prefix, suite), _color, VERBOSE) 
        
             output(": ", nil, VERBOSE) 
        
             @indent += 2 
        
           end 
        
           def test_suite_finished(suite) 
        
             @indent -= 2 
        
             @test_suites.pop 
        
           end

test-unit/lib/test/unit/ui/console/testrunner.rb

Lines 486 to 493 in 5914658

    
           def suite_name(prefix, suite) 
        
             name = suite.name 
        
             if name.nil? 
        
               "(anonymous)" 
        
             else 
        
               name.sub(/\A#{Regexp.escape(prefix)}/, "") 
        
             end 
        
           end

tikkss · 2025-12-27T00:33:13Z

lib/test/unit/ui/testrunnermediator.rb

-              end
-              @options[:event_listener] = event_listener
-              @options[:test_suite] = @suite
              @test_suite_runner_class.run_all_tests(result, @options) do |run_context|


Suggested change

@test_suite_runner_class.run_all_tests(result, @options) do |run_context|

event_listener = lambda do |channel, value|

notify_listeners(channel, value)

end

@options[:event_listener] = event_listener

# Large test suites (e.g. ruby/rbs) can cause

# `NoMemoryError` only on MSVC Ruby. Clears a

# needless instance variable to save memory

# while keeping metadata for UI events.

notifiable_suite = @suite.dup

notifiable_suite.instance_variable_set(:@tests, [])

@options[:test_suite] = notifiable_suite

@test_suite_runner_class.run_all_tests(result, @options) do |run_context|

lib/test/unit/ui/testrunnermediator.rb

GitHub: Fix test-unitGH-358 I don't know why this fixes the `NoMemoryError` problem but this removes needless references. So GC will collect more objects. This change is included in 691a96e but this isn't needed for parallel test execution. Either way, we don't need this.

kou · 2025-12-30T06:46:23Z

Thanks for looking into this! I didn't notice that @suite is the problem.

How about not changing @options directly?

tikkss

Sounds nice! NoMemoryError has not occurred:

https://github.com/tikkss/test-unit/actions/runs/20595170524/job/59148224585

How about this to a bit of DRY? This changes has not occurred too:

https://github.com/tikkss/test-unit/actions/runs/20595436660/job/59148989342

lib/test/unit/ui/testrunnermediator.rb

kou · 2025-12-31T03:26:57Z

We can do it for DRY but I don't want to change @options entirely.

:event_listener will not cause NoMemoryError but it will keep referring the test runner after the test runner is used. And I don't want to maintain what option values can be added to @options.

tikkss · 2025-12-31T11:01:10Z

Thanks for your polite reply. I agreed. Let's merge this PR.

kou · 2025-12-31T11:48:24Z

Let's release a new version and try it in ruby/ruby.

tikkss · 2025-12-31T12:04:35Z

OK. I'll release a new version. And I'll create a PR in ruby/ruby.

tikkss · 2026-01-02T11:16:39Z

I've created a PR in ruby/ruby#15786 .

kou force-pushed the fix-no-memory-error-on-windows branch from 6c84c6d to a2e58ef Compare December 25, 2025 02:37

tikkss reviewed Dec 27, 2025

View reviewed changes

kou force-pushed the fix-no-memory-error-on-windows branch from a2e58ef to 4ec23a0 Compare December 30, 2025 06:02

kou force-pushed the fix-no-memory-error-on-windows branch from 4ec23a0 to b7c364a Compare December 30, 2025 06:25

tikkss approved these changes Dec 30, 2025

View reviewed changes

lib/test/unit/ui/testrunnermediator.rb Show resolved Hide resolved

tikkss reviewed Dec 30, 2025

View reviewed changes

lib/test/unit/ui/testrunnermediator.rb Show resolved Hide resolved

kou merged commit 813ea12 into test-unit:master Dec 31, 2025
39 checks passed

kou deleted the fix-no-memory-error-on-windows branch December 31, 2025 11:47

	def test_suite_started(suite)
	last_test_suite = @test_suites.last
	@test_suites << suite
	if @top_level
	@top_level = false
	return
	end

	output_single(indent, nil, VERBOSE)
	if suite.test_case.nil?
	_color = color("suite")
	else
	_color = color("case")
	end
	prefix = "#{last_test_suite.name}::"
	output_single(suite_name(prefix, suite), _color, VERBOSE)
	output(": ", nil, VERBOSE)
	@indent += 2
	end

	def test_suite_finished(suite)
	@indent -= 2
	@test_suites.pop
	end

	def suite_name(prefix, suite)
	name = suite.name
	if name.nil?
	"(anonymous)"
	else
	name.sub(/\A#{Regexp.escape(prefix)}/, "")
	end
	end

-              @test_suite_runner_class.run_all_tests(result, @options) do |run_context|
+              event_listener = lambda do |channel, value|
+                notify_listeners(channel, value)
+              end
+              @options[:event_listener] = event_listener
+              # Large test suites (e.g. ruby/rbs) can cause
+              # `NoMemoryError` only on MSVC Ruby. Clears a
+              # needless instance variable to save memory
+              # while keeping metadata for UI events.
+              notifiable_suite = @suite.dup
+              notifiable_suite.instance_variable_set(:@tests, [])
+              @options[:test_suite] = notifiable_suite
+              @test_suite_runner_class.run_all_tests(result, @options) do |run_context|

Conversation

kou commented Dec 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kou commented Dec 25, 2025

Uh oh!

tikkss commented Dec 25, 2025

Uh oh!

tikkss commented Dec 27, 2025

NoMemoryError has occurred at master (9c6fce7)

NoMemoryError has not occurred with the following diff

NoMemoryError has occurred with the following diff

Uh oh!

tikkss left a comment

Choose a reason for hiding this comment

Uh oh!

tikkss Dec 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kou commented Dec 30, 2025

Uh oh!

tikkss left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kou commented Dec 31, 2025

Uh oh!

tikkss commented Dec 31, 2025

Uh oh!

Uh oh!

kou commented Dec 31, 2025

Uh oh!

tikkss commented Dec 31, 2025

Uh oh!

tikkss commented Jan 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kou commented Dec 25, 2025 •

edited

Loading

`NoMemoryError` has occurred at master (`9c6fce7`)

`NoMemoryError` has not occurred with the following diff

`NoMemoryError` has occurred with the following diff

tikkss Dec 27, 2025 •

edited

Loading