Skip to content

Dont use the ThreadDeathWatcher to release memory back to PoolArena a…#7446

Closed
normanmaurer wants to merge 2 commits intono_thread_death_watcher_fast_thread_local_threadfrom
finalizer_thread_cache
Closed

Dont use the ThreadDeathWatcher to release memory back to PoolArena a…#7446
normanmaurer wants to merge 2 commits intono_thread_death_watcher_fast_thread_local_threadfrom
finalizer_thread_cache

Conversation

@normanmaurer
Copy link
Copy Markdown
Member

…fter PoolThreadCache is not used anymore

Motivation:

At the moment we use the ThreadDeathWatcher to release memory back to the PoolArena after the PoolThreadCache is not used anymore if the cache is used by a non-FastThreadLocalThread. This will need to spawn on extra thread and iterate over all the Threads each second.

Modifications:

Better just use a finalizer to return the memory back to the PoolArena after the cache is not used anymore and the cache belonged to a non-FastThreadLocalThread.

Result:

Less running threads / overhead.

…ocalThread with wrapped Runnable is used

Motivation:

We dont need to use the ThreadDeathWatcher if we use a FastThreadLocalThread for which we wrap the Runnable and ensure we call FastThreadLocal.removeAll() once the Runnable completes.

Modifications:

- Dont use a ThreadDeathWatcher if we are sure we will call FastThreadLocal.removeAll()
- Add unit test.

Result:

Less overhead / running theads if you only allocate / deallocate from FastThreadLocalThreads.
…fter PoolThreadCache is not used anymore

Motivation:

At the moment we use the ThreadDeathWatcher to release memory back to the PoolArena after the PoolThreadCache is not used anymore if the cache is used by a non-FastThreadLocalThread. This will need to spawn on extra thread and iterate over all the Threads each second.

Modifications:

Better just use a finalizer to return the memory back to the PoolArena after the cache is not used anymore and the cache belonged to a non-FastThreadLocalThread.

Result:

Less running threads / overhead.
@normanmaurer
Copy link
Copy Markdown
Member Author

PTAL... This depends on #7445.

@ejona86
Copy link
Copy Markdown
Member

ejona86 commented Nov 27, 2017

Hmm... since finalize is deprecated in Java 9, do we want to do something else? We could keep the cleaner thread around and use WeakReference+ReferenceQueue. The thread would sit on remove(), although it will probably still need to do periodic wakeups. I don't know the code well enough to know if we could avoid the thread altogether.

@normanmaurer
Copy link
Copy Markdown
Member Author

@ejona86 hmm yeah we could have some kind of "ReaperThread" that will cleanup this stuff... Even this would be better then the ThreadDeatchWatcher imho...

So I think a finalize() may be ok for now as we will need to do another PR anyway to get rid of all finalize overrides in all the classes. So for now I think this PR is at least an improvement.

WDYT ?

@ejona86
Copy link
Copy Markdown
Member

ejona86 commented Nov 27, 2017

I've found adding threads within a library to be difficult. If we remove it now, then depending how much time passes I'd anticipate re-adding it to cause breakage.

Using ReferenceQueue is much closer to the current code than finalize, although it is likely true ThreadDeathWatcher is the wrong API. And just because there are (a few) other finalize() usages doesn't mean we should paint ourselves into a corner and make the problem worse.

Is there a reason to try hard to kill ThreadDeathWatcher now at the expense of dealing with the migration twice and any thread-creation fallout?

@normanmaurer
Copy link
Copy Markdown
Member Author

@ejona86 ThreadDeathWatcher caused class loader leaks etc. That said I agree with you. Let me see if I can make it work with a ReferenceQueue and a Thread. I will open another PR so we can compare the different approach more easily.

@normanmaurer
Copy link
Copy Markdown
Member Author

@ejona86 actually I would argue that if we would run on java9 only we should use the Cleaner for it which we not do atm: https://docs.oracle.com/javase/9/docs/api/java/lang/ref/Cleaner.html

WYTD ?

@ejona86
Copy link
Copy Markdown
Member

ejona86 commented Nov 27, 2017

"on java9 only", I'd agree with that, but given support is currently Java 6, it's a pretty moot. Cleaner when on Java 9 sounds good, but we will need a fallback. If we chose finalize since it's easy, that could be fine. Note that Cleaner does create a thread (albeit, in a very specific manner).

Hmm... classloader leak... I guess the failure case is the ThreadLocal instance could be thrown away, and let the storage evaporate (since Thread uses WeakReference for the mapping), but the Thread itself remain running. So ThreadDeathWatcher continues running unawares, and in fact maybe even transitively retains the ThreadLocal because it retains the ClassLoader.

So it seems the problem is the false assumption that the ThreadLocal storage's life ends at the same time as the thread. I guess that's why Cleaner has "The context class loader of the thread is set to the system class loader." Aaaand... we couldn't do the same approach, because the Cleaner trick relies on the fact that Cleaner is in the system class loader.

Does my analysis feel right? If so, then it seems finalize is likely the only option, short of disabling caching on the other threads to begin with.

};
}

// We create an instanceof this if we need to call free() as part of the GC of the cache.
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/instanceof/instance of/

@ejona86
Copy link
Copy Markdown
Member

ejona86 commented Nov 27, 2017

Wait... if my analysis is correct, then even Cleaner wouldn't work, since it maintains strong references to Runnables retains the ClassLoader (which retains UnpaddedInternalThreadLocalMap.class which has the static ThreadLocal). Cleaner would need the references to be non-static (or properly cleared). My head hurts a little thinking about it (mostly because of normal "there's still a reference, in the finalize"), but it does seem finalize would work. But it also seems we're backing ourselves into the corner.

I feel like something else needs to give. Either the caching needs to go, or an API to clear the cache, or some slow caching path, or something. For the slow caching path, maybe we could force all non-FastThreadLocalThreads to share a single (possibly-sharded) cache and just throw a lock on it, but we would still need to figure out a good time to clear it.

// Wait for the ThreadDeathWatcher to have destroyed all thread caches
// Wait for the GC to have destroyed all thread caches
while (allocator.metric().numThreadLocalCaches() > 0) {
System.gc();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also System.runFinalization()

@Override
protected void finalize() throws Throwable {
try {
free();
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is it safe to call free multiple times?

@normanmaurer normanmaurer force-pushed the no_thread_death_watcher_fast_thread_local_thread branch from 132758b to 7e131b8 Compare November 28, 2017 12:43
@normanmaurer
Copy link
Copy Markdown
Member Author

normanmaurer commented Dec 1, 2017

@carl-mastrangelo @ejona86 WDYT of the approach: #7463 ?

@normanmaurer
Copy link
Copy Markdown
Member Author

Let me close this.. as we should better use #7463

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants