TestNG Version

7.5.1, but the behavior is reproducible on the latest release (7.8.0).

Background

I am using Jackson, a very popular library that helps support JSON in Java. As a result of an upgrade from 7.4.0 to 7.5.1 (for JDK 8 support), I ran into a set of test failures involving the JsonNode class from Jackson, which implements Iterable. It appears that Iterables and Collections now must be in the same order to be considered equivalent by assertEquals. However, by definition, JSON objects do not have to have elements in the same order.

The biggest inconsistency is that for two JsonNodes node1 and node2 that have elements in differing orders, TestNG declares that:

• assertEquals(node1, node2) is false
• assertTrue(node1.equals(node2)) is true

Some history on this shows that this behavior has changed a few times.

• Version 6.7: Ordering was not enforced. Example test below passes.
• 6.8.1: Ordering enforced. Example test fails.
• 7.3.0: Ordering was still enforced.
• 7.4.0: Comparing collection using equals method for assertEquals testng#2460 implemented since it was noted that Set comparisons fail.
• 7.5: changes in behavior with assertEquals(Collection) et al testng#2540 was raised, noting that the new behavior was at odds with the API documentation. The behavior was reverted to enforce ordering again.
• 7.5: assertEquals(set,set) now requires same ordering which it didn't before testng#2643 implemented to provide a separate assertEquals for Set, a subclass of Collection.

The concerns noted on testng-team/testng#2540 are fair and make sense. However, I'm unsure that the current behavior is appropriate. To me, having a set-specific implementation of assertEquals is the wrong approach to fix this problem when any subclass can decide whether ordering of the elements contained within the Iterable or Collection matters or not. But I recognize that this is nuanced.

I resolved this issue in my project by using the AssertJ library, which compares JsonNodes in a way that I expect. So this issue is not critical for me, but I think it's worth evaluating whether some of the inconsistencies mentioned here can potentially warrant a change in this behavior.

I've added a test case that defines my own subclass in an attempt to make a simple reproducible test. It involves a mathematical "unordered pair" object whose equivalency is evaluated incorrectly.

Expected behavior

assertEquals(one, two) should always have the same behavior as assertTrue(one.equals(two)).

Actual behavior

assertEquals(one, two) is not equivalent to assertTrue(one.equals(two)) in the example code provided below.

Is the issue reproducible on runner?

• Shell
• Maven
• Gradle
• Ant
• Eclipse
• IntelliJ
• NetBeans

Test case sample

import org.testng.annotations.Test;
import java.util.*;
import static org.testng.Assert.*;

public class TestCase {
  private static class UnorderedPair implements Iterable<Integer> {
    List<Integer> children;

    public UnorderedPair(int a, int b) {
      children = Arrays.asList(a, b);
    }

    public int getFirst() {
      return children.get(0);
    }

    public int getLast() {
      return children.get(1);
    }

    @Override
    public boolean equals(Object o) {
      if (!(o instanceof UnorderedPair)) {
        return false;
      }

      UnorderedPair other = (UnorderedPair) o;
      if (getFirst() == other.getFirst() && getLast() == other.getLast()) {
        return true;
      }
      return getFirst() == other.getLast() && getLast() == other.getFirst();
    }

    @Override
    public Iterator<Integer> iterator() {
      return children.iterator();
    }
  }

  @Test
  public void testEquals() {
    UnorderedPair ascendingOrder = new UnorderedPair(1, 2);
    UnorderedPair descendingOrder = new UnorderedPair(2, 1);

    // This assertion passes.
    assertTrue(ascendingOrder.equals(descendingOrder));

    // This assertion fails.
    assertEquals(ascendingOrder, descendingOrder);
  }
}

Contribution guidelines

In case you plan to raise a pull request to fix this issue, please make sure you refer our
Contributing section for detailed set of steps.