You tell users, that asSet returns an immutable Set.

@return a type-safe immutable {@code Set} containing the specified collection elements.

This is not true, when you pass in a mutable Set. In this case the Collection is only cast to a Set. You probably want to change the documentation or use java.util.Collections.unmodifiableSet