This week, I also shared our vision on #digitaltwins at the MathWorks site Digital Twins and the Evolution of Model-based Design.

Last week, I discussed my misgivings about the idea of data-driven v.s. model-driven AI . This week, we extrapolate that question and ask a much broader question:   “Will AI breakthroughs this decade be based on last decade’s AI developments, OR will we see new directions of AI research in this decade?”

Undoubtedly, the last decade 2010 to 2020, was a game-changer for AI research. It was based on the current deep learning models characterised by parallelized networks comprising relatively simple neurons that use non-linear activation by adjusting the strengths of their connections. Using this model, we find the rules for function f(x), which maps domains x to y when the rules are hierarchical and (typically) the data is unstructured. To do this, we need a large number of labeled examples. The labels are at a higher level of abstraction (ex if the image is a cat or a dog). The algorithm can then discern the features that comprise the object(ex a cat has fur, whiskers, etc.). This is the essence of deep learning called representation learning and is common knowledge for data scientists.

Especially in the latter part of the last decade, three developments have accelerated this model:

a)    Transformer based models

b)    Reinforcement learning and

c)    Generative adversarial networks.

And they continue to amaze us daily – for example:

Deepmind - Alphafold – deep mind protein folding ; Deepmind - meta-algorithm creating the one algorithm to rule them all , i.e., a deep learning model that can learn how to emulate any algorithm, generating an algorithm-equivalent model that can work with real-world data. And of course, Deepmind’s AI predicts almost precisely when and where it is going to rain , and now megatron Turing NLG from Nvidia and Microsoft – a large language model that claims to exceed GPT-3

Impressive as these are, all the examples above share some crucial properties

• They suit narrowly defined problems.
• You can generate your own data (ex alphaGO) with practically endless experimentation
• The cost of experimentation is low.
• There is plenty of data available ex images, text, speech, etc
• The models are black box
• Risk of failure is low (ex: advertising)

 The other extreme is: rules are symbolic, i.e., decided by humans.

 That was the early days of AI which ultimately led to the AI winter.

 However, I believe that this decade will be all about techniques that can interject expert intelligence into the algorithm (but is not the same as symbolic as in the early days of AI)

•  Hybrid cognitive models CLARION cognitive architecture
• Causal models/ Bayesian models / Graph-based models from Judea Pearl, ex Judea Pearl – the foundations of causal inference TED talk.
• Simulations including physics-based models

 One such interesting case is from some work by Max Welling. I have referred to this work before, but in this case, I am referring to the use of generative models to build counterfactual worlds. This could work where we have a problem domain with too many exceptions, i.e., there is a very long tail of situations that do not show up in your dataset used to model the problem.

To put the problem in context