Are you still panning? There is a better way to extract mainframe gold

With the emergence of AI tools to unlock the data in mainframe systems, we're seeing a shift from historical analysis to real-time insight. It's the difference between panning for gold and picking gold flecks straight out of a flowing stream.

Traditional practices

Long before the advent of AI, mainframes quickly found their place in enterprises as a high-capacity, high-performance transaction engine, able to process millions of transactions in milliseconds, which in turn generated masses of pure transactional data. That core, unadulterated data is the "gold truth" of the enterprise.

Traditionally, the analysis we were doing was on historical data.

That's in part because there was no need for enterprises to be working on data in real-time. We were doing what-if and trend analyses; there was no imperative for instant decision-making or real-time interventions in the business process.

Also, the mainframe processes that were running and the data it contained were too critical to our operations to risk trying to extract it with the tools we had available at the time. And it was very costly – anything that you tried to do with data on the mainframe would increase MSU usage_[1].

To optimise MSU usage and avoid any risk to the operation of the mainframe or the data itself, best practice was to set up end-of-day batch ETL processes and utilities to move information out into a data warehouse. The entire concept of analytics was backward-looking.

Pressure to respond quickly

Now, in a digital world, consumers have an increasing expectation that everything is instantaneous – think about government or banking transactions, or insurance underwriting, where an insurance policy is issued immediately following the application. To achieve this, our systems need access to reliable data in real-time so that we can leverage AI, machine learning and analytics to accelerate decision-making and return more accurate (read safe) outcomes.

That's critical for fraud detection or credit checks. When you are running an AI or ML model on a credit card purchase or a loan application, unless you have real-time data covering the most recent transactions, the algorithm will be fed old data.

Hallucinations typically occur when LLMs and AI models are making inferences from poor-quality, incomplete or stale data. If your AI model can't see the most recent mainframe transactions, its answers will often be wrong or misleading, even if the model itself is technically sound.

On top of this, leaders in financial services and banking (and, I'd argue, government) shouldn't trust AI in production without clear governance and guardrails. It has to operate within enforced policy, prove it works against real past cases, and pass the same security bar as any other system touching sensitive data.

Competitive advantage

More than just approving a loan application, with access to that real-time gold truth in the mainframe together with a range of other contextual data, the system can also decide on other factors, including the best rate of interest and equated monthly instalment (EMI) it can offer the borrower.

There's a massive competitive advantage for a company that can respond the quickest. Think about car insurance. It's standard practice in Australia to apply for your third-party or comprehensive vehicle insurance online, and consumers typically make multiple applications to compare prices.

If you receive an immediate quote and policy back from your application, rather than waiting for a phone call or an email to confirm details or send through the quote, you're more likely to go with the first one – especially if your insurance has just expired and you need cover quickly!

As an organisation, you can only have confidence in this sort of instant response or automated decision-making if your algorithms have access to the "gold" triumvirate: clean, contextual and real-time data.

A new way to mine data

We have the tools now to bring this gold together and apply an AI layer to make sense of it all.

Today, as a transaction happens on the mainframe, that can immediately be reflected in the enterprise's online data store, data warehouse or lakehouse like Rocket Vertica. And this could be on the cloud or in a sovereign environment, depending on the sensitivity and value of the data.

The key point here is that we can bring together information from every relevant enterprise system and source, but also while keeping the original data where it is – if we choose to. To do this you need to give your data context; which you can do by building a semantic layer to represent critical corporate data and develop a common dialect for that data.

Once we have that, with our AI layer like Rocket EVA we can start to apply different use cases and solve major business challenges.

The goal for any enterprise to leverage AI to its full effect is to have clean, contextual real-time data for the AI engine to run on. The bottleneck today is in making this gold data available.

I'll leave you with this question: if you have mainframe data, where millions of transactions are taking place on a daily basis, and you don't have access to this gold stream in real-time, how many thousands of recent transactions are your AI tools missing when they are analysing that data and making decisions for you?

[1]  MSU refers to a million service units, a measurement of the amount of processing work a mainframe can perform in one hour, which is also used to calculate ongoing software costs.