The AI black box problem: How financial organizations can ensure AI explainability and transparency

Now that we know why AI black box systems are not the best choice for your organization, let’s explain why working on AI explainability should be your most valuable product.

In regulated industries like finance, telecom, or healthcare, explainability is not just a nice-to-have – it’s a must-have. Think about it and ask yourself – would you trust an AI system that automatically approves your loan or one that diagnoses your health condition, without giving you a clear view into the process and how the decision was made? We doubt it.

Regulations like the EU AI Act and GDPR require that decisions made by AI systems are transparent. For example, under Article 22, the GDPR gives individuals the right to an explanation of automated decisions that affect them. If an AI system can’t provide that, not only are you violating regulations, but you’re also actively inviting legal trouble.

Legal issues aside, explainability builds trust. When customers understand how AI works, they are more likely to accept it, use it, and eventually recommend it to others.

Note: AI explainability is not the same as AI transparency. We wrote about this in our article “AI transparency vs. AI explainability: Where does the difference lie?” so make sure to check it out for a better understanding.

Let’s now take a closer look at the risks that black box AI systems bring to finance. Opaque algorithms in finance can easily reject clients for loans or credit without giving a clear explanation of why the decision was made. Or even worse, such systems can unintentionally discriminate against someone based on demographic characteristics or geographic location. Does that sound like a problem? – It definitely is.

Black box AI systems in finance (and other industries) can lead to:

• financial damage from lawsuits filed by clients of banks and other financial institutions
• legal trouble with regulators
• reputational damage depending on the level of the issue

It’s easy to see that AI black box models in finance are not just undesirable – they should be seen as a high-level threat that even large, well-established institutions like banks and other financial organizations cannot afford.

But are explainable AI systems accurate enough? That’s what we’ll discuss in the next section.

These days, we often hear that highly explainable AI systems are usually inaccurate – but is that really true? Absolutely not!

While it may be true that some very complex AI systems, like deep neural networks, are harder to interpret and explain, there’s always a way to find a balance between explainability and accuracy. Hybrid AI models, for example, combine simpler, more interpretable algorithms with complex ones to offer the best of both worlds.

So we can say that the key is choosing the right model for the right problem. In regulated industries like finance or telecom, explainability of AI models should definitely be prioritized. Why? Because, in the end, it’s better to have a slightly less accurate AI system that is fully explainable than a black box model that puts your entire business on thin ice, with risks for your clients, investors, or regulators. Wouldn’t you agree?

Below, we’ll list some examples where we believe AI should be fully explainable in the financial industry.

Explainability in Finance: Show Me the Money, but Show Me How!

As we’ve already mentioned, the financial industry is one of the most regulated industries in the world, along with telecom and healthcare – and for a good reason: money. When money is involved, explainability and transparency are non-negotiable.

Let’s take two examples:

1. Loan approval for bank clients – if an AI system is performing credit scoring to decide whether a person can get a loan, and it decides to deny it, the AI must explain why. Was it because of a poor credit history? Was it due to their income level? Or maybe because of the overall economic situation in the financial or lending market?
2. Fraud detection and prevention – as we’ve mentioned in other articles: where there’s money, there’s fraud. Banks, as huge institutions, process billions of transactions every day. Every transaction carries a certain level of risk, and as we wrote in our article “AI in financial industry: How banks and insurers can deploy AI without regulatory risks,” banks often use AI to detect and prevent fraud. However, if the AI blocks a transaction, it needs to give a reason – why was this transaction flagged? What parameters led it to believe it was fraud and make the decision to block it?

As we can see from both examples above – in each case, AI is making an autonomous decision. Without a clear explanation, everyone involved may think a mistake was made and will have to step in manually, wasting valuable time. But if the AI system clearly explains how it reached that decision, it will help financial institutions better understand the situation, act faster, and maintain better relationships with everyone involved.

As we’ve written before, the EU AI Act is a game changer for several reasons. The first is, of course, that it’s the first comprehensive regulation that governs the development, deployment, and use of AI systems. The second reason is that it doesn’t treat all AI systems the same – it classifies them based on the level of risk they pose to users. The third reason, and the most important for this article, is that it aims to bring transparency into AI systems, build trust, and ensure alignment with ethical principles.

You can read more about the EU AI Act in our guide “EU AI Act guide” here.

When it comes to the EU AI Act, the situation is very clear. Systems classified as high-risk AI – for example, an AI system used for credit scoring in the financial industry – must meet the strictest requirements. These include creating detailed technical documentation, listing the datasets and training data used, and explaining how the system makes decisions – all with the goal of making the AI system as explainable as possible.

You’ll probably agree that for enterprise-level companies like financial institutions, this is not just a challenge, as it may seem at first – it’s also an opportunity. Why an opportunity? Because by following these rules, they can strengthen their reputation, avoid the risks that come with using AI technology, and stand out from the competition.