The journey towards ultimate efficiency

The path to AI maturity is not only long but also winding. Different applications need different algorithms and tools and unlike the other technologies (like cloud and analytics) which are more scalable, AI calls for a fresh look at all the different use cases.

Here you find a recap of the status today and glimpse into what is needed before AI can play a valuable role in process automation.


– a basic Machine Learning challenge

Machine learning models are typically designed to operate in the future by making decisions based on novel inputs. Even though these systems are meant to operate in the future; they are trained on samples of data with the assumption that the data represents an omniscient universe.

Unfortunately, this is not how intelligence works in the real world. Humans, no matter how intelligent, have to deal with unfamiliar situations all the time. This explains why even the smartest of us are often uncertain about many things. True intelligence therefore lies in recognizing your limitations.

Unknown unknowns usually surface when the data used to train the predictive model is not representative of the samples that were encountered during testing when the model is deployed. The mismatch can be blamed on unmodeled biases during the gathering of training data. To help make the point clear, think of an image classification program whose goal is to predict whether the given image is a cat or a dog.

Assume the training data is made of images of black dogs, and brown and white cats and the features of the animals include presence or absence of whiskers, color, shape of nose etc. such a model might learn to use color for recognition of the animals even though there are other attributes because the color can adequately separate the two classes of data presented. Consequently, on testing the system, the model might classify a white dog as a cat.

When developing intelligent systems, the developers should keep in mind that the system will have to deal with known-unknowns and unknown-unknowns once deployed.

Even though coding an algorithm that addresses this problem is a tall order, developers can at least aim to reduce misclassification of the training samples through empirical risk minimization. As Ethan Rudd observes, empirical risk is directly proportional to the number of misclassified training examples.

Training data

– Garbage in now means even more garbage out

Data collection is one of the huge challenges in machine learning. A lot of time is spent on collection, cleaning, understanding and feature engineering of the data. In smart factories, processes like quality control are automated.

Such applications depend on the data that was gathered at the observation stage and so there is often no training data for the detection of new defects or new products. Deep learning can be used for feature engineering but for it to be effective; lots of training data will be required. There is, therefore, a dire need for scalable and accurate data collection methods.

The purpose of data acquisition is to locate datasets that can be used for training machine learning models. The data acquisition process can be automated through three distinct approaches namely:

  • Data discovery
  • Data augmentation
  • Data generation

Data discovery is a two-step process. The first step is the indexing of generated data and then publishing the same for purposes of sharing. The second stage is where other users search for the data in something like a  Data lake.

Most collaborative systems have been designed to make this process easy but it is not uncommon to find systems that were not built with the intention of sharing their datasets. In such a case, a post hoc approach can be taken to generate metadata for the various datasets.

Aaron Edell attempted to use AI to predict the stock market and what made his experiment easy to set up was the availability of a reliable dataset which he could use to train his algorithm.

Another technique that can be used is to augment existing data sets with some external data. This is commonly used in machine learning for adding the features to train on. In some cases, datasets are not complete and will need to be completed by gathering more information.

A good example of how data can be augmented is when augmentation is used for the generation of embeddings that represent entities, words, or knowledge that can then be used to solve various problems in Natural Language Processing.

In cases where there is no data to train on, datasets must be generated manually or automatically. Manual generation of data is laborious and is, therefore, implemented through crowdsourcing. Individuals are tasked with generating bits of data which eventually ties together to become a dataset.

However, automatic techniques can be used to increase the efficiency of this process. Data augmentation can also be considered as data generation in situations where the datasets are incomplete and need some filling in.


– And other steps towards automation

The watershed moment for AI intelligent decision making was in 2011 when the IBM Watson computer system triumphed over two Jeopardy! game show champions (Brad Rutter and Ken Jennings).

This was followed by a sequence of breakthroughs in machine learning and today, computer algorithms easily beat humans in skill games and video games even without prior instruction. But AI is also used in other real-life and useful scenarios for recommending specific actions.

For instance, in 2016, Starbucks introduced an AI-based recommendation system across all their franchises all over the world. If you purchase a morning coffee at Starbucks, the system will automatically recommend a muffin. This is a good illustration of how predictive analytics can transform the retail industry.

Recommendation systems can use a collaborative filtering approach, a content-based approach or a hybrid approach. Collaborative filtering is the process of using peer opinions to recommend an action for others. A user is matched against a dataset that has users who have had a similar interest and then a recommendation is made based on what other users with similar interests liked.

A content-based approach focuses on user profiles derived from the data the user provides either explicitly (e.g. when they rate a service) or implicitly (e.g. when they click on native ads). The hybrid approach is a combination of the other two. Examples of how this is implemented in the real world include:

  • Amazon: customers who bought this also bought…
  • YouTube: recommended videos
  • Spotify: recommended songs
  • Facebook: people you may know and news you may like
  • Netflix: Other movies or TV series you may enjoy
  • LinkedIn: Jobs you may be interested in

Making decisions

Action possibilities can be structured, semi-structured or even unstructured. It all depends on the degree of certainty of a solution to a given problem. Structured decisions are deterministic with known solutions, unlike unstructured decisions which depend on the decision maker. In between the two is a spectrum of unique decisions which can best be described as semi-structured.

The decision maker is a very important component of system and should not be left out in the system design. Analytical models can be used to represent semi-structured action possibilities. As a consequence, they tend to attract the most attention from technology aiding. Technology can be leveraged to assist human behavior in many ways. For instance, technology can help the human user in choosing the best input or selecting the most relevant data.

Business intelligence and analytics can help in addressing problems that encompass huge datasets that are widely distributed. A combination of business intelligence and decision support techniques results in the development of a robust intelligent decision system. Such a system should be capable of doing the following:

  • Learn from past experience
  • Make sense out of contradictions and ambiguities
  • Respond very quickly to new situations
  • Solve problems through reasoning
  • Cope with perplexing scenarios
  • Make inferences in rational and ordinary ways
  • Manipulate the environment through the application of knowledge
  • Think and reason

Intelligent decision support systems are mostly based on neural networks, case-based reasoning or genetic algorithms. However, intelligent agents can also be very instrumental.

An intelligent agent is a computer system that is capable of acting autonomously in order to realize a desired objective. For even intelligent behavior, multi-agent systems can be used.

This is where multiple agents interact with each other to improve the quality of decisions made. The decision maker will therefore know that the output they get from the system was arrived at from the views of many agents.

Process automation

– with a human in the loop or not

To automate or not is no longer a question of if but a question of when. Organizations need to think of integrating IA for their own benefit and even survival. There are already an increasing number of enterprises that are adopting end-user automation via robotics processes.

According to a report by Ernst & Young, the rationale for this enthusiasm is the huge improvement in productivity that automation achieves. AI tools continue to evolve as they undertake more intelligent tasks.

Automation can, therefore, be achieved in areas never thought of before like budgeting, planning and analysis for decision making. Full process automation can be achieved through the following steps.

  • Structure data interaction: This refers to traditional systems where integration happens through the exchange of well-structured information. Examples of structured data interaction include Relational Database Management Systems (RDBMS), Application Programming Tools (APIs), and data transformation tools.
  • Robotic Process Automation (RPA): this entails the automation of rules-driven and standardized activities by the use of scripts and other techniques to achieve efficiency in a business process. RPA is ideal in situations where it would be too expensive or inefficient for a human user to execute the tasks.
  • Machine Learning (ML): ML involves systems that learn through the handling of variations that are may not be anticipated up front. The systems are trained on the go through the assimilation of learnings from decisions and data, and may also make some simple algorithmic predictions and classifications.
  • Natural language processing: NLP relies on learning algorithms and various statistical methods to analyze unstructured data in order to understand the meaning, intent and sentiment. A case in point is when a customer raises a support ticket, which is subsequently analyzed to establish how urgent it is or determine how frustrated the customer might be. This will then help the system to determine the severity and priority of the support ticket.
  • Natural Language Generation  (NLG): NLG is a technology for generating text as the user speaks or writes from structured information like fields and numbers. It is widely applied in the generation of financial reports and analyses e.g. the numbers that reflect the performance of a company.
  • Chatbots and virtual agents: these are agents that can interpret text/voice in free form to quickly respond with predefined answers. For instance, a customer service function can be used to respond to frequently asked questions. The chatbots can continuously learn and grow their vocabulary to get better at interpreting unstructured information.
  • AI Decision systems: AI Decisions systems rely on an array of algorithms, models and technologies to solve complicated problems. They are often driven by cognitive capabilities and deep learning systems to apply statistical models and recognize patterns that can help in making choices. These can be used to address several decision points, e.g., the determination of demand for a product in a given geographical location based on factors like weather. This can help the business to know how best to control their inventory.


The adoption of AI across different sectors continues to heighten by the day. AI is now as good as experts in various use cases. As machine learning continues to get more sophisticated, AI systems will continue to grow even smarter.

The best illustration of this is how computers are now successfully beating humans at games. Interestingly, this is happening even without prior training.

Intelligent systems will help companies to make fast and accurate decisions and they will ultimately improve efficiency and customer satisfaction.

But we are still in the early stages of AI adoption and there are a lot of changes and improvements to expect as we move towards intelligent systems. At the moment, the end-user is awed whenever they interact with even simple AI in their day to day lives, like say when ads are re targeted to them based on their browser history.

Currently it is complicated to get good value out of AI, especially for SMEs. Fortunately we – as well as numerous other interesting startups – are working hard to unlocking the benefits of AI by bridging the chasm between legacy IT systems and infrastructure and a beneficial use of AI for daily operations, customer support and effective process automation.

We will know we have achieved AI maturity when AI is fully integrated in every sector and when interacting with AI stops feeling like a bit of magic.