Dear Alex,

You know me. I am not an AI expert.

You are, and over the last couple of years, you’ve made increasingly bold statements about AI’s ability to change all things in our professional lives as well as in our everyday lives.

I trust your opinion, just as I trust that these statements come from a place of expertise and careful analysis of the facts. I do not question everything you say.

But Alex, my friend, you are greedy. You never have enough. Every day, you must conquer another area, disrupting our world even more than you already have (much to my despair at times: I see the inroads made by AI-generated music, crushing my hope that technology will someday wash the dishes and let me focus on poetry and music, while all I get instead are computers doing the art part while I am left with the chores.)

I digress.

The thing I know really well is programming languages – especially legacy programming languages. I know their behaviors, complexities and oddities. I know how they are used, how to compile them, how to analyze them. This is my world.

I delve in legacy modernization. To paraphrase a slogan we coined years ago, “Our software makes yours live longer”.

AI has already contributed a lot to the field, by automating tests, explaining what a piece of code does, synthesizing concise, human-readable information out of thousands of line of logs, and more.

But now, you are pushing for AI to be used not at the periphery, not for ancillary purposes, but at the core of Legacy Modernization, namely, to translate mainframe application code directly to Java, C# or even Python. These claims are made with a lot of spin, very impressive demos (always) and precious little details.

Now that you are on my turf, it is only fair that I get to ask you tough questions. You may be able to come up with burning zingers, ridiculing me in the process, but you should not expect me to take your claims at face value, just because they come with the AI label attached to them.

And what is a little disagreement among friends?

Because you see, Alex, at its core, legacy modernization is a numbers and precision game.

It deals with large systems, thousands of programs, tens of millions of lines of code. No matter where you stand on the spectrum of all the flavors of legacy modernization, scale is always a key concern.

This focus on large figures is not just an incidental side effect; it is intrinsic to the domain. If the software systems we were dealing with were not huge, if scale was an issue, one would just go about rewriting them, and that would be the end of it. There would be no modernization to speak of.

Scale is what makes simplistic approaches unrealistic. Scale fuels the entire legacy modernization world.

As a consequence, legacy modernization is also a precision game: the existing behavior must be reproduced as accurately as possible. One does not have the time and bandwidth to detect and investigate hundreds or thousands of possible differences of behaviors one by one. Accuracy allows this numbers game to play in your favor.

That means that in legacy modernization, when you say that something is close enough to the original behavior, you do so at your own peril.

(Circling back to the digression I started above: AI-generated music can work because music easily survives some level of imprecision. Details in music matter only to the extent that they can be heard. If you can’t hear them, they just don’t matter. In steep contrast, in software, there is no such thing as a detail too small to matter. A single misplaced comma can make or break your multi-billion dollars software system.)

At this point, my dear Alex, please let me make a bold statement of my own: when using AI to transform legacy systems, you are essentially giving up on precision and accuracy.

Because the exact, accurate, mechanical, non-AI version of this translation is a solved problem (To be clear, I have serious reservations about the true value of automated COBOL to Java translation when it comes to maintenance, integration, performance and staffing- but let’s leave that for another conversation with another imaginary person; the reservoir of gender-neutral first names is massive.) I have my doubts as per their value, but the fact remains that there have been deterministic tools to perform this kind of translation for decades.

Because the demos of these AI-powered tools repeatedly show that the resulting code is clean, well-structured and idiomatic. There is a catch though: there is no such thing as a free lunch. These otherwise very desirable properties come with a steep price, namely differences of behavior, some mundane, some crushing.

Because allowing for some inaccuracy in exchange for magically dealing with the most common cases is what the current breed of AI does.

My dear Alex, from a distance, you may think that this does not sound like a bad deal at all. 95% of your existing system in nicely structured Java or Python? That is worth something, right?

I would not be so sure.

What used to be a modernization project has now turned into a vast testing and remediation endeavor. You just don’t know where these 5% of differences or missing features are. And guess what: if your AI flavor of the month hasn’t been able to deal with it automatically, dealing with it post hoc will not be trivial. Remediating the remaining 5%, when (and if) you will have found where they are, will be far from a walk in the park. It will require eyeballs. It will require brains. And as if this was not enough, by its mere size and complexity, we both know as a statistical fact that this remediation effort will introduce new defects of its own.

(I consider these 95% as an optimistic estimate, but even if the final figure was closer to 99%, it would not change the argument much. Whether the gap is 1 or 5%, you have no idea where to look. Quite a few needles in a very large haystack, if you ask me. And the challenge is not to find just the proverbial one, but to ensure you’ve found them all, given that you don’t know beforehand how many there are.)

The big organizations that run large amounts of COBOL code are a serious and risk-averse bunch: banks, insurance companies, airlines, governments. You are now going to them with a pitch along the lines of: “Please pay us (tons of) money so that we can convert 95% of your code to Java, and trust us that we’ll find and correct all the bugs and shortcomings in a timely and cost-effective way. And no, we don’t even know where to look.” Good luck with that. It is not even the money part that would stress me, it’s the intrinsic uncertainty that come with any project of this magnitude.

And to be honest, if there is something that I just don’t buy, it is that yet another (as yet unspecified) piece of AI will magically address all the shortcomings of the existing one(s). Something that proved too hard to handle early on will be even harder to tackle once the artifacts have gone through one or more intermediate transformations. As much as I like you, Alex, AI just cannot always be the answer, even when AI is the problem.

The elephant in the room metaphor also show its limits here: even if you can deal with COBOL (which, as this post shows, is not exactly a slam dunk to start with), mainframe systems depend on an ecosystem of languages and tools, most of which you’ve never heard of (I am not holding this against you: I am already so pleased that you have at least an idea of what COBOL is).

What about PL/I? Assembler? REXX? Easytrieve? Pre-relational databases? Indexed files? Schedulers (of which there are multiple flavors)? JCLs and the attached utilities? Transaction processing infrastructure? If and when you decide to look into these technologies, you may realize that there are just not enough portfolios around to train an AI on some of these.

So, my dear Alex, tell me.

Did I get it wrong when assuming that you had given up on precision and accuracy?

Did I get it wrong when stating that adding yet another layer of AI cannot always be the answer?

Did I get it wrong in pointing out that dealing with COBOL alone won’t get you very far?

By now Alex, I’ve made it clear that I am not an AI expert, but you should not discount my questions as coming from someone who just does not get it. They deserve a serious answer.

What do you think?

Yours truly,

Darius.