Funny compile error using Collectors.maxBy and Comparator.naturalOrder() (and using lambdas)

[Andrea Binello]

Hi, I am developing a small Maven-based project to provide some nice demos about the new Java 8 Stream API:

https://github.com/andbin/java8-streams-demos

Normally I use Eclipse for my developments and everything works fine as expected. Today I tried to build my project outside Eclipse, using just Maven and JDK 8u112 (the latest at this moment). I got a "funny" compile error in one of my demos: MaxOriginDistanceByQuadrant.java

To be more clear about the problem, I tried to re-create a simpler/single source to reproduce the problem.
And it is:

import java.util.Arrays;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.Optional;
import java.util.TreeMap;
import java.util.stream.Collectors;

public class StreamProblem {
    public static void main(String[] args) {
        List<MyObj> myObjs = Arrays.asList(
            new MyObj("A", 1),
            new MyObj("B", 3),
            new MyObj("A", 5),
            new MyObj("B", 7),
            new MyObj("B", 4),
            new MyObj("A", 2)
        );

        Map<String,Optional<Double>> map = myObjs.stream()
                .collect(Collectors.groupingBy(
                        obj -> obj.getStr(),
                        () -> new TreeMap<>(),
                        Collectors.mapping(
                                obj2 -> obj2.getNum(),
                                Collectors.maxBy(Comparator.naturalOrder())
                        )
                ));

        System.out.println(map);
    }
}

class MyObj {
    private String str;
    private double num;

    public MyObj(String str, double num) {
        this.str = str;
        this.num = num;
    }

    public String getStr() {
        return str;
    }

    public double getNum() {
        return num;
    }
}

A javac StreamProblem.java using directly the JDK 8u112 gives the following very "funny" error:

StreamProblem.java:21: error: no suitable method found for groupingBy((obj)->obj.getStr(),()->new TreeMap<>(),Collector<Object,CAP#1,Optional<T#1>>)
                .collect(Collectors.groupingBy(
                                   ^
    method Collectors.<T#3,K#1>groupingBy(Function<? super T#3,? extends K#1>) is not applicable
      (cannot infer type-variable(s) T#3,K#1
        (actual and formal argument lists differ in length))
    method Collectors.<T#4,K#2,A#1,D#1>groupingBy(Function<? super T#4,? extends K#2>,Collector<? super T#4,A#1,D#1>) is not applicable
      (cannot infer type-variable(s) T#4,K#2,A#1,D#1
        (actual and formal argument lists differ in length))
    method Collectors.<T#5,K#3,D#2,A#2,M>groupingBy(Function<? super T#5,? extends K#3>,Supplier<M>,Collector<? super T#5,A#2,D#2>) is not applicable
      (inferred type does not conform to upper bound(s)
        inferred: Object
        upper bound(s): T#6,Comparable<? super T#6>,Object)
  where T#1,T#2,T#3,K#1,T#4,K#2,A#1,D#1,T#5,K#3,D#2,A#2,M,T#6 are type-variables:
    T#1 extends T#2
    T#2 extends Comparable<? super T#2>
    T#3 extends Object declared in method <T#3,K#1>groupingBy(Function<? super T#3,? extends K#1>)
    K#1 extends Object declared in method <T#3,K#1>groupingBy(Function<? super T#3,? extends K#1>)
    T#4 extends Object declared in method <T#4,K#2,A#1,D#1>groupingBy(Function<? super T#4,? extends K#2>,Collector<? super T#4,A#1,D#1>)
    K#2 extends Object declared in method <T#4,K#2,A#1,D#1>groupingBy(Function<? super T#4,? extends K#2>,Collector<? super T#4,A#1,D#1>)
    A#1 extends Object declared in method <T#4,K#2,A#1,D#1>groupingBy(Function<? super T#4,? extends K#2>,Collector<? super T#4,A#1,D#1>)
    D#1 extends Object declared in method <T#4,K#2,A#1,D#1>groupingBy(Function<? super T#4,? extends K#2>,Collector<? super T#4,A#1,D#1>)
    T#5 extends Object declared in method <T#5,K#3,D#2,A#2,M>groupingBy(Function<? super T#5,? extends K#3>,Supplier<M>,Collector<? super T#5,A#2,D#2>)
    K#3 extends Object declared in method <T#5,K#3,D#2,A#2,M>groupingBy(Function<? super T#5,? extends K#3>,Supplier<M>,Collector<? super T#5,A#2,D#2>)
    D#2 extends Object declared in method <T#5,K#3,D#2,A#2,M>groupingBy(Function<? super T#5,? extends K#3>,Supplier<M>,Collector<? super T#5,A#2,D#2>)
    A#2 extends Object declared in method <T#5,K#3,D#2,A#2,M>groupingBy(Function<? super T#5,? extends K#3>,Supplier<M>,Collector<? super T#5,A#2,D#2>)
    M extends Map<K#3,D#2> declared in method <T#5,K#3,D#2,A#2,M>groupingBy(Function<? super T#5,? extends K#3>,Supplier<M>,Collector<? super T#5,A#2,D#2>)
    T#6 extends Comparable<? super T#6> declared in method <T#6>naturalOrder()
  where CAP#1 is a fresh type-variable:
    CAP#1 extends Object from capture of ?
StreamProblem.java:25: error: cannot find symbol
                                obj2 -> obj2.getNum(),
                                            ^
  symbol:   method getNum()
  location: variable obj2 of type Object
2 errors

Note that this exact code perfectly compiles and works in Eclipse (the latest Neon.2 Release (4.6.2) Build id: 20161208-0600).

I noticed that I can solve the problem using the so called type witness on the naturalOrder():

Collectors.maxBy(Comparator.<Double>naturalOrder())

However, there is still another funny thing. I have rewrote the code using method references only (no lambdas):

        Map<String,Optional<Double>> map = myObjs.stream()
                .collect(Collectors.groupingBy(
                        MyObj::getStr,
                        TreeMap::new,
                        Collectors.mapping(
                                MyObj::getNum,
                                Collectors.maxBy(Comparator.naturalOrder())
                        )
                ));

And it compiles and works on JDK 8u112 even without the type witness !

So I am wondering: is this a bug of JDK8? (I guess yes... and I hope it will be fixed!)
Is it related to maxBy? And/or lambdas?
Is the type witness on naturalOrder() the best solution at this moment?



By the way: if someone can give me some judgments on my project, I willingly appreciate.
Thanks.

[Piet Souris]

The problem me thinks, is that if you look at the API of collect, Collectors.groupingBy and Collectors.mapping, the class parameter <T> in the mapping is independant of the class parameter <T>  as used in the groupingBy. That is why java does not recognize the mehod getNum of obj2, ince it does not know what type obj2 is, This code works:

Map<String, Optional<Double>> map = myObjs.stream()
                .collect(Collectors.groupingBy(
                        obj -> obj.getStr(),
                        () -> new TreeMap<>(),
                        Collectors.mapping(
                                (MyObj obj2) -> obj2.getNum(),
                                Collectors.maxBy(Comparator.naturalOrder())
                        )
                ));

by helping java a little by revealing the type of obj2 that we are dealing with. That is why using 'MyObj::getNum' also suffices, and if you use Comparator<Double> it somehow also recognizes MyObj, although I do not completely understand what is happening in that case.

[Pierre-Yves Saumont]

The reason why it works using method reference is obvious: method references include both the parameters and return types, so it gives the compiler much more information than needed. As Piet said, specifying the argument type for the mapper function is enough to make type inference work.

You have in fact many possible solutions:

1) using a method reference MyObj::getNum

2) specifying the parameter type for the mapper: (MyObj obj2) -> obj2.getNum()

3) using a parameterized method call: Collectors.maxBy(Comparator.<Double>naturalOrder())

4) Using a named Function:

Function<MyObj, Double> f = obj -> obj.getNum();

Map<String,Optional<Double>> map = myObjs.stream()
                                       .collect(Collectors.groupingBy(
                                           obj -> obj.getStr(),
                                           () -> new TreeMap<>(),
                                           Collectors.mapping(
                                               f,
                                               Collectors.maxBy(Comparator.naturalOrder())
                                           )
                                       ));

5) Using an named Comparator for the downstream Collector:

Comparator<Double> comp = Comparator.naturalOrder();

Map<String,Optional<Double>> map = myObjs.stream()
                                         .collect(Collectors.groupingBy(
                                             obj -> obj.getStr(),
                                             () -> new TreeMap<>(),
                                             Collectors.mapping(
                                                 obj2 -> obj2.getNum(),
                                                 Collectors.maxBy(comp)
                                             )
                                         ));

Thee are probably many other possibilities to help the compiler inferring the Double type. The best one is obviously using method references.

This said, your very question was "is it a JDK bug?". The answer is no. It might be an Eclipse bug, depending on what were the requirements. Eclipse has its own compiler, and this compiler is (as you noted) not fully compatible with Java. So if the requirement for the Eclipse compiler was to be compatible with Java, it's an Eclipse bug. If the requirement was to be compatible with the Java 8 specification only, it's a feature.

Not everything is specified. For what is not specified, there is a "reference implementation". From Java Platform, Standard Edition 8 Reference Implementations :

The official Reference Implementations for Java SE 8 (JSR 337) are based solely upon open-source code available from the JDK 8 Project in the OpenJDK Community.

About one year ago, I submitted an issue to the Eclipse development team about the inverse problem: places were the Eclipse compiler was unable to infer types although Oracle JDK and OpenJDK could. The issue was rejected because they said it was not Eclipse which was not compatible, it was Oracle JDK that was not conform to the spec.

This obviously was not a big problem. An IDE with restrictions is fine. But an IDE that allows more that what the compiler used in production can accept is much more problematic. This shows that in any case, programmers should either avoid using Eclipse or compile with the JDK used in the build chain before submitting their work.

This, by the way, is not a problem specific to Eclipse. Programmers should develop with the same version of the JDK that the one used in production. Or at least, they should compile their code and run their tests locally using the correct version before submitting.