WARNING: This blog post might contain strong language
It all started yesterday evening, when I saw @mheusser Testing Challenge on Twitter.
As usual my first question started with “why”. I believe that to be able to develop/test any feature (and make sure it is useful), we need to understand the real business value that we are delivering, hence the question “why?”.
I was very disappointed when I saw @mheusser’s answer because he didn’t give me any context to understand the real business value gained by the customer when sorting by PO. So due to the fact that @mheusser wasn’t going to give me any relevant information and the fact that I couldn’t be bothered asking him again I decided to employ my favorite Product Owner: Justin McDoitnow.
First I found out from Justin that PO actually meant Product Order and not Polymorphic Ostrich like I initially thought.
Second, I asked him what he meant with sorting by PO and he goes “Are you stupid? I mean they are in alphabetical order for f***s sake, and get going that we are late for the release!”
Good old Justin, always charming.
Numbers, letters? “Only letters”
At this point I didn’t ask what alphabet was used or if a special collation had to be applied because I didn’t want him to beat me up right now and suggested as valid the English alphabet with 26 small characters [a to z] no spaces allowed; Justin seemed happy and also confirmed with me that sorting works only one way, ascending order [a before b].
I then called the lead developer Zack Hack, and asked him if they had reused any open source library for doing the sorting and he goes “No, Gus, we didn’t because we analysed what was available and they were all bad. But hey! We implemented this brand new algorithm that performs much better than anything that has been written before, it rocks! Let me tell you how we do it! So we take…”. I obviously put Zack on mute and let him reach developer climax on his own. The information was enough, as usual the wheel was reinvented yet again, and it had to be tested.
So I started testing the first scenario.
Scenario1: Let’s make sure Zack didn’t f**k up the alphabet
I populated the database (or whatever persistence layer you like) with the following POs
zzzzzzzzzz
yyyyyyyyyy
……….
aaaaaaaaaa
Logged on to the app, clicked on sort and verified that the order was, guess…
aaaaaaaaaa
bbbbbbbbbb
……….
zzzzzzzzzz
OK Zack didn’t screw the English alphabet, that’s a good start.
Now I only verified sorting based on the first letter, but that’s not enough, Zack and his new “very useful algorithm” could have screwed up the sorting of the second character or the third or the fourth, etc.
I need a new Scenario.
Scenario2: Let’s make sure Zack’s algorithm can sort POs with at least one common character in early position
That’s not as easy now, I make a quick calculation and discover that the possible combinations of 10 elements from a pool of 26 where the order is not relevant is 1.83579396e+8
I call Justin and tell him that I need 3 light years to complete testing and he threatens to fire me so I decide to try to identify a pattern that might help me reduce the amount of test cases to a small number. Pen and paper, try, fail, retry and eventually after 10 minutes I notice an interesting behaviour applied to certain set of characters, have a look:
Did you see anything? I knew you would, well done, yes the diagonal has the first 10 letters in order and under the diagonal the letters are repeated starting from the second to the last. Why did I do this?
I did it because it allows me to do something special: First the 10 POs (horizontal lines) are already in “order” according to our rules. Hence pushing the “sort” button would leave them as they are and I can assert that the first PO stays on top.
This first test does nothing more than Scenario1 didn’t do with all the a’s and the z’s, but watch what happens if I move the first character of the first PO to the last position and I push up the first character of the second PO to be the first character of the first PO:
At this point if I push the imaginary “sort” button the alphabetical order of the second letter will be compared and I expect the second PO to go to the top (c>b) hence the second PO to be the first in the order.
What if I now move the second character of the second PO to the last position and I push up the third character of the third PO to be the second character of second PO?
Ehm… I think I see a pattern… Every time we do our move and we click sort we compare a character at position n and the n(th) PO becomes the first in the order.
This could take me a couple of hours to test manually through the user interface or just about half an hour to automate it.
I have a quick peek into the code and to my horror I discover that the whole multi-tier application is a monster single Java class of 13,256,459 lines of unreadable code. This means that any change, even unrelated to “sort PO” might break it! Definitely automate now!
OK Eclipse up and write some really dirty code that does the job:
public class TestSort {
public static void main(String[] args) {
char[][] testData = {
{ 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j' },
{ 'b', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j' },
{ 'b', 'c', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j' },
{ 'b', 'c', 'd', 'd', 'e', 'f', 'g', 'h', 'i', 'j' },
{ 'b', 'c', 'd', 'e', 'e', 'f', 'g', 'h', 'i', 'j' },
{ 'b', 'c', 'd', 'e', 'f', 'f', 'g', 'h', 'i', 'j' },
{ 'b', 'c', 'd', 'e', 'f', 'g', 'g', 'h', 'i', 'j' },
{ 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'h', 'i', 'j' },
{ 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'i', 'j' },
{ 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'j' } };
for (int i = 0; i < 10; i++) {
if (PushSortAndReturnPoIndexOfTopRecord(testData, i) == i) {
System.out.println("PASS");
} else {
System.out.println("FAIL: PO at position " + i
+ " was ordered incorrectly");
}
char temp = testData[i][i];
testData[i][i] = testData[i][9 - i];
testData[i][9 - i] = temp;
}
}
private static int PushSortAndReturnPoIndexOfTopRecord(char[][] myData,
int rowNumber) {
int index = 0;
// Seed DB with 10 records
// Call the sort function
// returns the index of the row that is now at the top
return index;
}
}
I will look at refactoring this piece of sh** ehm, code and add it to our CI system once I have some time, for the moment it helped me prove what I wanted.
Of course my code is verifying the order only looking at the PO field and the other fields might have been messed up. I do a quick manual check and see that this doesn’t happen, for the moment this is enough for me, I will add this extra check to the automation while I re-factor the code.
While I was writing the test code I had a couple of thoughts:
- Can 2 POs be identical? Justin says “NO!” and calls me a moron because supposedly I was meant to know it.
- Are we enforcing this unique constraint on the database or we just hope it doesn’t happen?
I’m going to answer question 2 myself, just pull up a SQL client, and run an insert with a PO that already exists, luckily I get “ORA-00001: unique constraint (XYZ.PK_PurchaseOrder) violated”.
Good!
I had some strange feeling and I asked Justin what would be the worst thing that could happen if the customer couldn’t sort by product owner for some reason on in the worst scenario if the sort returned wrong results. Justin face turned white and he started rambling about “huge loss of revenue, loss of customers, loss of testers limbs etc…” Apparently the company relies on sorting by PO so much that any malfunction could be fatal.
So, I ask, if it is so important, say a customer has just sorted by PO and a new PO comes through soon after and it should be seen by the customer in the top 20 displayed on screen, what happens if he doesn’t see it? Justin goes “if it happens we’re f****d”.
How long is too long for a PO to be displayed after it has been added?
“5 seconds”.
Ok, new test. I seed the DB with 20 items and sort. All good, they are all sorted in right order, but we knew that already. Now I add a new PO seeding it so that it should be the first after sorting. I don’t sort and I wait for 5 seconds. It appears at the top on the screen! Yes, the guys have thought about this, but I am not sure how the system will support the load with many PO’s already present and many added every second, let me see…
Hey Justin, how many PO’s do we have in our DBs in production, and how many new PO’s do we expect per day? Justin thinks for a while and goes “At the moment about 100,000 but we need to be able to scale to up to a Billion and in 3 years we are projecting sales in excess of 10,000 items per day hence potentially 10,000 POs per day”.
I seed a db with ~1B POs (LOL), sort them by PO, go for a coffee, a spin on the bike then return and see if the POs are sorted. When I return, I write some code to add 20 new PO’s each second while sorting by PO, then I pack my stuff and go on a 3 weeks holiday, I’ll come back just in time for the test to be completed.
Thanks to @mheusser for the test challenge.
mysoftwarequality 