Explainability & Wicked Problems — The 3rd Dimension to AI Hardware (Flux Capacitor) !!

9 min readSep 7, 2021

Next Wednesday (Sep 15, 2021), I have a talk at the AI Hardware Summit 2021. Should be interesting. You can download the slides (here) — this blog is an annotated version of my talk — helped to clarify the thoughts.

AI hardware has been evolving — from CPUs to GPUs to Tensor Cores. Time has come to add the Computational Model Factory to the Silicon ! Till now the focus was the computation of neural networks; this is a good time to think about accelerating the Building and Evolution of Explainable Models. Let us see if I can make a convincing case !!

The Flux Capacitor, of course, is a reference to “Back to the Future” movies ! It powers the DeLorean to time travel !!

We will cover the Where/Who, What and Why.
The How/Computability of Explainability will take longer — topic for a future discussion.
I had proposed the talk “Computability of Explainability” at Nvidia GTC2021, but couldn’t make it.

May be GTC2022 and/or the AI Hardware Summit’22 is a good venue for that discussion !

Let me start by thanking the giants whose shoulders I am standing on (and able to see much further)
A list of references are at the end — I have been diligent in collecting them as I develop the slides; but probably missing a couple. If so, apologies.

The What

An obviously wrong model and a good model. If we have a mechanism that creates the circles we see here, it would explain why the models behave the way they behave ! And, then we will have a chance to correct them.

And, we do - it is the explainability algorithm called LIME !

The algorithm perturbs the image with black pixels and inspects the predictions.

After a few computational cycles and perturbing all parts of the image, it creates a simple model that can explain the predictions.
From 1st impressions, this model looks correct. It looks fine as it made only 1 error.
Let us run it by the LIME Explainability algorithm

When we run through the LIME we can see the problem. Actually the algorithm found a shortcut — given the training images, wolf is associated with the snow background ! Remember, AI/ML algorithms are a lazy bunch — they will learn all kinds of shortcuts

We have built a snow detector !

Before you dismiss this as a simplistic case, may I point out that there is a legend, in the 60s. Which says the US Army built a brightness detector instead of tank classifier ! You might see other variations of this tale.
Of course, they didn’t have explainability or LIME then !

There are a few more canonical examples in my slides.
Remember, these are simpler examples — but, with 100s of features and many weak signals, the models will be a lot more difficult to explain.

This leads us to the complexities of explainability. NIST has 4 principles
#1 states that we should have explainability (but no qualification about what it should be !) - no need for hardware there.
But, #2 requires tailored explanations for different audience,
#3 expects accuracy and
#4 requires metacognition.

Suddenly the art of explainability becomes more compute intensive and now we can see a dim possibility that hardware acceleration might be a good idea. … Read on

The Where/Who

The most complex explanations are in the high-risk, high-complexity domains
They span financial, healthcare, autonomous vehicles and even social media like news and recommendations. We do need to understand why a news was recommended !

The Explainability 7 gives us a context in which to think about Explainability. One challenge is that there no consistent nomenclature; luckily there are some general understanding.

Interpretability is the mechanism, the algorithms, the how. It is a property of the model/algorithm
Explainability is the Why, an outcome that could be different depending on the audience.
Transparency is necessary, but not sufficient. Even if we know the underpinning of neural networks, that is not going to help us understand if a trillion parameters is a dog vs. cat classifier or if it understands traffic signs or an NLP model. Where is the Voight-Kampff test when we need one ?
I have written (Here) more notes on this line of thought

Explainability will become a policy for AI systems, soon
The combined RFI and the RFI from NIST are a prelude for the regulators to form an opinion and then formulate recommendations, guidelines and finally regulations.
So this is the right time for us

Lots of interesting things can be achieved with the right policy at the appropriate level. Thoughtful policy can guide and accelerate the adoption of AI — and hasty policies can create barriers for innovation

I like this slide as it illustrates the different aspects.
Trust : We need to ascertain that the models do what they say on the tin in which they came
There will be policies
And, we need tools for different audience at different levels of details. This is where the hardware acceleration is necessary

The Why

Now let us move on to thinking about the Why

Explainability is at an interesting juncture in the Gartner’s hyper cycle.
What it says is that people have certain expectations and the current explainability platforms won’t meet those expectations.
Which means we have enough data & demand to create multiple minimum viable products !

Moreover explainability also touches other trending domains like Governance, ModelOps and even the Feature Store !

Of course, it is best to use self explainable algorithms. But as you will see a little later, they are not as effective as the more complex algorithms. Even Linear Regression or Trees are not easily explainable if they have lots of features or trees with different sampling techniques and lots of trees.

Remember, model explainability still doesn’t mean causality ! And conceptual soundness and robustness to explainability attacks, are all still need to considered and dealt with !

This is an interesting slide — especially for us who are thinking about explainability silicon!
The X-axis has interpretability and Y-axis has performance.
Linear and Smooth models are less accurate but easy to compute
Highly accurate models have non-linear/non-smooth relationships and take longer time to compute !! I rest my case !!
Remember, these are not a blanket/universal statements — these are for high risk, complex models with large datasets

This paper makes the case for hardware acceleration very clear
The title says it all ! Either high interpretability-low accuracy or high accuracy but low interpretability
The top right quadrant, the ideal approaches, is empty !

The question for us — Are we going to make this a moot point by accelerating the explainability computation in hardware ? Are we ready for the Flux Capacitor ?

Yet another paper — saying Kernel SHAP is the best explainability method so far, but is quite slow especially with large datasets !

Talking about large datasets, a recent IMF blog proposes, quoting a recent study, that credit scores should be based on web history ! I have no clue what possessed them to this level of irrational exuberance and reckless abandonment ! Luckily this caught the attention of the EDPS (European Data Protection Supervisor) who nixed the idea.

Just think of counterfactual explanations on decisions based on web history ! That would be one for the record books !

In short, Hardware Acceleration of Explainability is a viable opportunity. The question, of course, is “Are we ready to explore the foundational computation for explainability with an eye toward embedding the most resource intensive ops in the silicon ?

Remember, building a neural network is not the ultimate goal. No one wants to build a churn model or a fraud model. They all want to stop churn and reduce fraud — which is what explainability does. It tells you how to stop churn or pointers to fraud behaviors

So, if we, as hardware builders, want to move towards a model factory centricity from the current compute-centricity, and accelerate explainability workloads, there is an opportunity !!

A quick detour — Wicked Problems

There are a class of problems called the Wicked Problems. very interesting to study — Explainability has some of the characteristics.

The How

The How is a topic for another talk.
We can’t solve this in 25 minutes — the how requires at least couple of hours.
As mentioned earlier, I had a proposal “The Computability Of Explainability” at the Nvidia GTC2021, but couldn’t.

I will propose a talk (or a tutorial) on “The Computability of Explainability” at the Nvidia GTC 2022 or the AI Hardware 2022 !

In the meantime, there are interesting papers to read and understand, algorithms to chase, …

And don’t forget to checkout the work by Lloyd Shapely. His original paper was written in 1951 and won the Nobel Prize in 2012 !
As a nostalgic trivia, I am sure the original 1951 paper was cyclostyled !
Probably none of you know what cyclostyling means !
I had my Masters thesis done that way — No laser printers then ;o)