Synchronous Messages

December 1, 2011 is the first JDK8 Warnings Cleanup Day (WCD).

This is an OpenJDK community effort to stamp out warnings in the JDK build. You mean there are warnings when you build the JDK? There certainly are, but they’re fairly well hidden in the log files. In many cases you need to add build options to javac such as -Xlint:all to see them. If you do this, you’ll see that there are over 10,000 warnings emitted by javac during the course of the build. How did they all get there? There are many reasons, but there are two main ones.

Prior to Java SE 5.0, a Java compiler was required to either compile a program without error, or fail to compile a program due to errors. There was no official concept of “warning.” The introduction of generics in Java SE 5.0, however, required a Java compiler to produce unchecked warnings for potentially dangerous uses of raw types. Older code that hadn’t been updated to use generics suddenly compiled with many “Recompile with -Xlint:unchecked” warning messages. There’s still a lot of this obsolescent code in the JDK. In fact, these warning messages account for the majority of warning messages that occur in builds of the JDK.

In addition to these mandatory warnings, javac in JDK1.5 was enhanced to produce warnings about “legal, but suspect and often problematic, program constructs.” (See Java SE 5.0 New Features and Enhancements.) An example of such a construct is a switch statement that “falls through.” The scope of these warnings was expanded significantly in javac in JDK7. The kinds of warnings and options for controlling them are detailed in the JDK7 tool documentation for javac. In particular, javac can produce warnings about undesirable, but relatively harmless, uses of raw types. These “harmless” uses are described in Maurizio Cimadamore’s article, Diagnosing Raw Types.

In the Summer of 2011, Sasha Boulgakov, a summer intern, did some analysis about build warnings and how to reduce them. He discovered that there were over 13,500 warnings issued during the build of the jdk repository alone, that is, not including corba, hotspot, jaxp, jaxws, and langtools repositories. (A notable point is that the langtools team has been quite diligent about warnings, having kept their build at zero warnings for quite some time. Kudos to the langtools team and especially Jon Gibbons for their efforts.) Although Sasha had only a short summer with us, he managed to knock off around 2,800 warnings from the jdk repository’s build. Further work by Oracle engineer Kurchi Hazra has brought this down further to around 10,000 which is where we stand today.

Why are warnings important? Most of them are probably noise and don’t represent real problems. Since the compiler is doing us the favor of pointing out “suspect and often problematic” code, a certain portion of these warnings indicate actual bugs. However, this message is drowned out in all the noise. If we could get rid of all the noise, we’d be able to see the real bugs more clearly.

That brings us to Warnings Cleanup Day.

This is a collaborative effort among the OpenJDK community members (inside and outside of Oracle) to clean up the build warnings. For this first event, we’re focusing on warnings from javac emitted during the build of the jdk repository. (There are also warnings emitted during native code compilation, but that’s a totally different subject.) We don’t have any specific numeric goals, but we’re going to try to knock off as many as we can. You can see the initial announcement here. The main WCD page is here, and the division of labor and status updates will be posted here and updated during the day. Email discussion will occur on jdk8-dev@openjdk.java.net and some of us will be hanging out on the #openjdk IRC as well.

Let’s clean up those warnings!

Updated 9 Oct 2012 with clarifications on Java language and javac history from Alex Buckley.

Java 7 includes a couple new features that change exception handling. Specifically, these features are try-with-resources and multi-catch with more precise rethrow. These are the first changes to the exception facility since chained exceptions were added in 1.4. Exception chaining was an important improvement, but it didn’t really change anything fundamental about the way you had to write exception handling code. Indeed, exception handling code is pretty much unchanged since the beginning of Java.

Java 7 changes that.

Consider this common exception handing task. In a two-step operation, if an exception is thrown in the second step, you need to clean up anything left from the first step.

For example, suppose you want to create and return a Foo, which in turns needs a Bar to have been created first. If creating the Foo fails for some reason, you want to make sure that the Bar is cleaned up. But if creating the Foo succeeds, you don’t clean up the Bar. You could start off with some code that looks like this:

Foo createFoo() throws FooException {
    Bar bar = new Bar();
    try {
        return new Foo(bar);
    } catch (FooException fe) {
        bar.cleanup();
        throw fe;
    }
}

(This assumes that the Foo constructor has been declared with throws FooException, a checked exception.)

The problem here is that if creation of the Foo (or something else within the try block) throws something other than FooException. The bar.cleanup() code gets skipped, and you’re screwed. In some sense what you want to do is to catch and rethrow a Throwable, but you can’t do this because you’d then be forced to change the declaration of createFoo to have throws Throwable which you really don’t want to do either.

A more effective approach is to use a finally clause for this, since it executes your code and then lets the exception propagate unchanged. Unfortunately, the finally clause also executes even if no exception is thrown, so you have to add some conditionals to make sure cleanup doesn’t occur in the successful case. This usually involves creating a local, initializing it to null, and then checking for null in the finally clause. Your code ends up looking something like this:

Foo createFoo() throws FooException {
    Foo foo = null;
    Bar bar = new Bar();
    try {
        foo = new Foo(bar);
    } finally {
        if (foo == null) bar.cleanup();
    }
    return foo;
}

The reasoning is as follows. The foo local variable is initialized to null and is assigned at the very end of the try block. If it’s null at the time the finally block executes, an exception must have been thrown from somewhere in the try block, and so the cleanup code needs to get run. This is a bit ugly, but it works, and I see it often enough to recognize it as an idiom.

Turns out that in Java 7 we don’t need to write this anymore. Instead, we can do this:

Foo createFoo() throws FooException {
    Bar bar = new Bar();
    try {
        return new Foo(bar);
    } catch (Throwable t) {
        try {
            bar.cleanup();
        } catch (Throwable t2) {
            t.addSuppressed(t2);
        }
        throw t;
    }
}

(Update 2017-11-22: added suppressed exception handling in outer catch-block. If bar.cleanup() throws an exception, especially a checked exception, this is caught and added to the original Throwable’s suppressed exception list.)

That is, we can catch and rethrow Throwable, just like we wanted to originally, but we don’t have to change the method declaration to throws Throwable. How can this work?

This is the “more precise rethrow” portion of the multi-catch with more precise rethrow feature, part of Java 7’s Project Coin. This feature is often overlooked, since it’s combined with multi-catch. Most treatments of this feature focus on multi-catch and barely talk about “more precise rethrow.” But as we can see from the code above, this feature can be used independently of multi-catch, and quite effectively as well.

To understand what’s going on here, we need to revisit some fundamentals of checked exceptions.

All Java code must conform to the “catch or specify” requirement. Any checked exceptions that can be thrown by a piece of code must either be caught by a try-catch statement that contains the code, or the checked exceptions must be specified in the throws clause of the method that contains the code. In Java 7, fundamentally this is still true, but the change is that the compiler now does additional analysis of what exceptions can be thrown by a throw statement inside a catch clause.

Prior to Java 7, the analysis was as follows. What exceptions can the “throw t” statement throw? The t variable is declared as type Throwable. Therefore, “throw t” can throw Throwable. This must be caught or specified, which in turn requires the method declaration to be changed to throws Throwable.

In Java 7, the analysis has been made deeper. Instead of just looking at the declared type of t, the compiler looks at where t came from. First, it makes sure that t points to the same object as it did at the beginning of the catch clause. (That is, t must be “effectively final”.) If so, the compiler then looks at the checked exceptions that can be thrown from the try block. In this case, we’ve called the Foo constructor, which can throw FooException. That’s the only checked exception that can arise from the try block, therefore the only checked exception type that t could have in the catch block is FooException. (It could potentially be an unchecked exception, but we don’t care about that, since unchecked exception types don’t participate in the catch-or-specify rule.) Now we get to the “throw t” statement; since we know t is either unchecked or is a FooException, we need to make sure that the enclosing method declares “throws FooException.” Indeed it does, and we’re done.

Now, if this chain of reasoning doesn’t hold up — for example, t is assigned to something else in the catch block — we revert to the old analysis, and t is treated as its declared type, just as before. Fortunately, most exception catching code doesn’t assign to the exception variable, so the new analysis will usually apply. Note that the exception itself can be modified, for example with the addSuppressed or initCause methods, and the new analysis will still apply. You just can’t assign to t and still expect this to work.

Anyway, where are we after all that?

It used to be that “catch Throwable” was a no-no. It usually meant you were going to so something stupid like swallow the exception. You couldn’t rethrow the throwable, since you’d be forced to redeclare your method with “throws Throwable” and nobody wants to do that. Now, in Java 7, it makes perfect sense to catch Throwable, do some cleanup, and then rethrow, without garbaging up the throws clause of your method declaration.

Will this make your exception handling code better? Is this a new Java exception handling idiom?

* * *

I’ll be talking about this more at my OSCON/Java talk next week in Portland.

I’ve written an article on the new Java SE 7 try-with-resources feature for the JavaTech Journal. This is a free PDF download. (You have to register with the site in order to get the download link.) This issue has several articles about Java SE 7, which will be released in the summer.

The article covers exception handling in Java SE 6 and earlier and what made it tedious and error-prone, the new try-with-resources feature and what it does, and the multiple resources variant and how it solves exception handling issues with wrapped resources.

This article was informed by the “Coinification” work I had done in JDK 7. I had previously converted some JDK 7 code to use diamond, and I had blogged about it here and here. I had also converted some code to use try-with-resources and the more precise rethrow features of Project Coin, but I haven’t blogged about it, mainly because most of my writing effort was focused on the JTJ article. Now that it’s been published, I’ll be writing more here about these features, and I’ll cover some of the finer points that didn’t make it into the article. Meanwhile, download and read the latest JTJ issue. You can also download preview binaries, sources, and documentation for JDK7 here.

Enjoy!

In my earlier post on Diamond, I described four places where the diamond operator can be used:

1. field or local variable initializer (61%)
2. right-hand side of assignment statement (33%)
3. method argument (4%)
4. return statement (2%)

(The percentages refer to the frequency of occurrence in code found by the automated “diamond finder.”)

Now, just because you can use diamond in some code doesn’t mean that you should. So, when should you use it?

An observation before we begin. If you have some code that provides generic type arguments, and you use diamond, the compiler will almost always infer the type arguments that you would have put there anyway. Using diamond will only rarely change the semantics of your program. The decision about whether or not to use diamond is entirely driven by things like style, readability, and clarity.

Let’s step through the different use sites for diamond and see how it affects these aspects of the code.

1. Field or Local Variable Initializer

An example of this usage is as follows. Without diamond, the code looks like this:

private static Map<ClassLoader, Map<List<String>, Object>> loaderToCache
    = new WeakHashMap<ClassLoader, Map<List<String>, Object>>();

And with diamond, it looks like this:

private static Map<ClassLoader, Map<List<String>, Object>> loaderToCache
    = new WeakHashMap<>();

If you want to know what the compiler will fill into the diamond for you, it’s right there in the declaration on the left-hand side. The code without diamond is redundant, and the code with diamond is much shorter and loses no information compared to before. This is pretty clearly a win. Since this is the most common usage (over 60%) even using diamond only in this situation helps quite a bit.

2. Right-hand Side of Assignment Statement

Now consider this code:

perms = new ArrayList<>();

What type arguments are inferred here? Well, you have to hunt down the declaration of the perms variable. But, no big deal. If you want to use the perms variable, such as to call a method on it, you have to know what type it is. And if you know that it’s a List or a Collection and you want to add something to it, you have to know what the type argument is. So not having the type arguments in the constructor doesn’t really hurt anything. Plus, if you want to see the type, you can usually just ask your IDE.

(On NetBeans 7, the gesture that works for me is a bit odd. First, hold down Control and click on some whitespace in your code — not on a symbol, since that will navigate to its declaration. Then, still holding Control, hover over the constructor that uses diamond. The full declaration of the constructor being called, including type arguments, should be displayed in a tooltip. Of course, you can always ask to navigate to the declaration of the variable.)

This is a point on which reasonable people can disagree. My diamond conversion changeset for the core libraries included diamond in assignment statements. The primary reviewer was Joe Darcy, who as Coin project lead is of course quite familiar with diamond. Subsequent diamond conversion changesets were for the security libraries, and the security team was uncomfortable using diamond in assignment statements. Given that they’re the ones who have to maintain this code in the long term, I decided it was best to conform to their wishes and not do diamond conversion for assignment statements in their area.

Personally I’m fine with using diamond in assignment statements, but your opinion might differ.

3. Method Argument

It’s possible to use diamond in the argument for a method call. This is fairly rare, occurring in only 4% of the cases examined. There are some complications with type inference, so it cannot be used in certain cases. But it’s hard to tell which cases these are. Consider for example the Collections.synchronizedSet method. Its declaration is as follows:

<T> Set<T> synchronizedSet(Set<T> s)

That is, it’s a generic method that has a type parameter T; it takes an argument of type Set of T and returns a value that’s also Set of T. It’s possible, though rare, to specify the generic type argument explicitly. Here’s how it’s done:

  Set<Number> set0 = Collections.<Number>synchronizedSet(new HashSet<Number>());

It’s not necessary to specify the generic type argument explicitly in this case. The following code works just as well (and is much more common):

Set<Number> set1 = Collections.synchronizedSet(new HashSet<Number>());

Where does the generic type argument come from? In this case the compiler infers it from the type argument from the constructor, and the result of the inference is that T is Number. Now let’s try to use diamond:

Set<Number> set2 = Collections.synchronizedSet(new HashSet<>());

Unfortunately, this does not work. Why not? In the previous case, the compiler used the type argument Number from the HashSet constructor call to determine the value of the type argument T. When diamond is used, the compiler needs first to fill in the diamond before it can establish the value of T. But in order to fill in the diamond, the compiler needs to know what type is expected as the argument for synchronizedSet(), and this type includes the type parameter T. So we have a circular dependency. The compiler can’t infer the right type, so the compilation fails. The compiler doesn’t flag the circularity as an error, though. What it does is start off by attempting to use Object in the diamond and then proceeding from there. The actual error message is as follows:

error: incompatible types
Set<Number> set2 = Collections.synchronizedSet(new HashSet<>());
    required: Set<Number>
    found:    Set<Object>

This is a bit odd, but that’s where the Set<Object> comes from. But note that the following statement does work:

Set<Object> set3 = Collections.synchronizedSet(new HashSet<>());

Now this is weird. The compiler attempts to fill in the diamond with Object, and this ends up working, so the statement compiles successfully.

The compiler doesn’t always start with Object. It will use additional type information if it’s available, such as from other method arguments. Consider this example:

Set<Number> set4 = Collections.synchronizedSet(new HashSet<>(set1));

This also works. The difference here is that we’re passing set1 as an argument to the HashSet constructor. This supplies enough information to the compiler about the type that should go into the diamond, and this in turn supplies the information to establish the value of the T type parameter of the synchronizedSet() method.

What’s the point of all this? The type inference that goes on works pretty well for variable initializers and assignments, but it can break down in some obscure ways for method arguments, especially for generic methods. This can make the code obscure, too. In the set4 example above, what type is inferred for the diamond? One might think that it has to be Number, but that’s not always the case. It really depends on the type of set1, which is declared elsewhere. We know that set1 is HashSet<Number>, whose type argument Number happens to match the type argument on the left-hand side. It looks like this type is inferred from the left-hand side, but in fact it isn’t. The following example illustrates this.

  Set<? extends Number> set6 = Collections.synchronizedSet(new HashSet<>(set5));

What gets inferred for the diamond? We can’t tell, other than that it must be a Number or a subclass of Number. It really depends on the type of set5. It might be Number or it might be AtomicLong. Indeed, since I haven’t provided the declaration for set5, it goes to show that we can’t tell how the diamond gets filled in simply by looking at this statement. We have to go look elsewhere.

As you can see, there’s considerable subtlety surrounding the use of diamond in method arguments, and there are some obscure interactions with the compiler’s type inference algorithm. It might be quite difficult for a programmer to figure out what type gets filled into the diamond. In some cases diamond can’t be used, and it results in a compiler error that might be puzzling.

For these reasons we’ve concluded that it’s probably not a good style to use diamond within method arguments even when it would be legal to do so. In doing my diamond conversion over the core libraries, I made sure to avoid using diamond in method arguments. In fact I had put some diamonds in method arguments in early on, and when we decided to avoid using diamond in these cases, I went back in and pulled them out.

4. Return Statement

Finally, it’s possible to use diamond in the return statement. In this case the diamond is inferred from the return type of the method. This is similar to an assignment statement, since the type of the expression in the return statement must be assignment compatible to the return type of the method. The method’s return type might be declared fairly far away from the return statement, but this isn’t necessarily any worse than assigning to an object’s field, which might also be declared far away. For short methods, like the following,

List<Foo> makeFooList() {
    int capacity = ...; // compute initial capacity
    return new ArrayList<>(capacity);
}

it’s pretty much a no-brainer to use diamond. But, overall, the opportunity to use diamond in return statements is quite rare, so it’s probably not worth worrying too much about whether or not to use diamond for these cases.

Summary

The first two cases — field or variable initializers, and assignment statements — together comprise well over 90% of the opportunities for using diamond. I’d recommend using diamond in those cases. Even if you’re not comfortable using diamond in assignment statements, you can still get a lot of benefit using diamond just in initializers. Diamond in return statements are rare enough not to worry about very much, though it’s probably reasonable to use diamond in the obvious cases. Finally, it’s probably a good idea to stay away from diamond in method arguments. When it’s legal to use, it might be hard to see how the diamond is filled in, and when it’s not legal, it might be hard to figure out why.

My colleague Joe Darcy has recently posted the Early Draft Review for JSR 334, Project Coin. (Joe is the spec lead for this JSR.) Let’s see what the EDR has to say about the diamond operator. First, there are some changes to the grammar that allow an empty type argument list. Not too surprising. Then, it continues:

Let K be the class type being instantiated. To create an instance of K, i, a constructor of C is chosen at compile-time by the following rules, where C is chosen as follows:

• If the type denoted by K has an empty type argument list (diamond operator) and K is a classtype of the kind G<X1, X2… Xn>, then C is a type constructed from K, where all constructors c1(), c2() … cn() are obtained from corresponding constructors in G by prepending additional type-arguments T1, T2 … Tn, where the declared bounds of T1, T2 … Tn are obtained from the declared bounds of X1, X2 … Xn, where all occurrences of X1, X2… Xn are replaced by T1, T2 … Tn. Furthermore, all occurrences of types X1, X2 … Xn in the obtained constructor signatures are replaced by T1, T2 … Tn.
• otherwise C == K

Uhhm, I’m not entirely sure, but I think this means that the compiler just does the right thing. 🙂

I should mention that, just above this, the draft states:

Note to readers: the description of diamond semantics below will be reworked in future drafts of the specification.

Whew! I know that sometimes specifications are opaque — and I’ve written some pretty opaque specifications in my day — but this one really takes the cake. Fortunately, this is only a draft, so this can get fixed before the final version of the specification.

But what does this mean, really? Instead of trying to explain the draft specification, I’ll instead pursue a more intuitive approach. Basically, here is what happens, as I understand things. When you use the diamond operator, the compiler will infer from context a set of type arguments that make the expression work. This is a only little less abstract, so let’s look at some examples.

List<Number> list = new ArrayList<>();

What type argument does the compiler infer for the ArrayList? The only argument that will work is Number, so that’s what will get used. No other type will work in the diamond, not even a subclass of Number, such as Integer, since the resulting type ArrayList<Integer> would have no type relationship with List<Number>. (For further explanation, see Angelika Langer’s Generics FAQ #102, or Josh Bloch’s Effective Java, Second Edition, Item 25.)

In this case the context is pretty clear. In what other contexts can one use the diamond operator?

As I mentioned in my previous post, what I’ve done is to run a Jackpot-based diamond converter over a bunch of source files in the JDK. This applies the diamond operator everywhere it can possibly be used. It identified several hundred potential use sites for diamond. These cases broke down as follows:

1. field or local variable initializer (61%)
2. right-hand side of assignment statement (33%)
3. method argument (4%)
4. return statement (2%)

There were also a tiny number of obscure edge cases that I’ll ignore for now. Let’s look at each of the four major cases in turn.

1. Field or Local Variable Initializer

This case is by far the most common. This is what’s shown in my one-line example above. It’s also probably the most common usage cited in toy examples given in talks or articles about Coin and the diamond operator. At least, in Joe’s talks at JavaOne and Devoxx last year, the one-liner examples he showed were all field or local variable initializers. Here’s an example of an actual change I pushed recently, to src/share/classes/java/lang/reflect/Proxy.java. The original code was:

private static Map<ClassLoader, Map<List<String>, Object>> loaderToCache
    = new WeakHashMap<ClassLoader, Map<List<String>, Object>>();

and the replacement code is:

private static Map<ClassLoader, Map<List<String>, Object>> loaderToCache
    = new WeakHashMap<>();

Now even though this is a pretty complex generic type, figuring out what gets inferred inside the diamond is pretty easy. It’s right there in the declaration on the left-hand side. Having the type arguments repeated on the right-hand side is noisy and redundant.

2. Right-hand Side of Assignment Statement

Consider the following statement:

perms = new ArrayList<>();

What does this do? Well, you have to look for the declaration of the “perms” variable in order to figure it out, but that’s not too hard. In decent code it should be obvious anyway. In this case (src/share/classes/java/io/FilePermission.java) the declaration is

List<Permission> perms;

so this case is also pretty straightforward. It’s true that you can’t tell from the assignment statement by itself what the instantiated type will be. But in order to use the perms variable, you have to know what type it is anyway (or you have to ask your IDE) so this isn’t too much of a burden.

3. Method Argument

Here’s an example from src/share/classes/com/sun/java/util/jar/pack/BandStructure.java, around line 1700:

List<List<Attribute.Layout>> attrDefs = ...;
...
attrDefs.set(i, new ArrayList<>(...));

What gets filled into the diamond here? This is where it gets a bit tricky. Looking at the snippets above, it’s pretty obvious that the type argument has to be Attribute.Layout. And indeed it is. But the compiler has to go through a couple different steps to get here.

1. The “set” method of attrDefs takes parameters (int, E).
2. The attrDefs variable is declared List<E> where E is List<Attribute.Layout>.
3. ArrayList<> needs to match List<Attribute.Layout>; therefore, what goes in the diamond is Attribute.Layout.

This isn’t exactly the analysis the compiler goes through, but you can see this case has many more moving parts than the others. This is actually a pretty simple case of a method argument as well: we didn’t have to do method overload resolution. If the set() method had been overloaded, the compiler would use the types of the arguments to figure out which method to use. But with diamond we’re trying to infer the argument type based on the type the method accepts… from what I understand this isn’t actually a circularity, but it greatly complicates the compiler’s analysis.

4. Return Statement

Here’s an example from src/share/classes/java/lang/ClassLoader.java:

public Enumeration<URL> getResources(String name) throws IOException {
    ...
    return new CompoundEnumeration<>(tmp);
}

This isn’t too different from the assignment statement. The type required is determined based on the return type of the method, and in this case it’s URL.

Summary

The four cases described here account for virtually all of the cases where the diamond operator can be applied in real code. Of these four cases, one case — the initializer case — is the most common in real code and also the most common in toy examples and in slideware. This is a good validation of the designs that have been presented over the past couple years. How many times have we seen toy examples used to justify features that turn to be hardly useful in practice?

The two most common cases (initializers and assignment statements) comprise over 90% of actual uses found of the diamond operator. Furthermore, their type inference is straightforward and they’re fairly easy for programmers to understand. Other cases may involve much more complex type inference, and are potentially harder to understand (hm, maybe there’s a correlation there), but they also occur much less frequently.