Anomaly Detection for the AI-Driven Business

Anomaly detection is an approach that can be useful across an array of industries and for a variety of purposes, including IT and DevOps, manufacturing, healthcare, banking and finance, and in the public sector.

Anomaly detection is all about finding patterns of interest (outliers, exceptions, peculiarities, etc.) that deviate from expected behavior within dataset(s). As with most data science projects, the ultimate end goal or output of anomaly detection is not just an algorithm or working model. Instead, it’s about the value of the insight that outliers provide. That is, for a business, money saved from preventing equipment damage, money lost on fraudulent transactions, etc. In health care, it can mean earlier detection or easier treatment. 

A non-exhaustive look at use cases for anomaly detection systems include:

  • IT, DevOps: Intrusion detection (system security, malware), production system monitoring, or monitoring for network traffic surges and drops. Challenges include the need for a real-time pipeline to react plus the huge volumes of data and unavailability of labeled data corresponding to intrusions, making it difficult to train/test. This industry usually has to adopt a semi-supervised or unsupervised approach.
  • Banking and insurance: Fraud detection (credit cards, insurance, etc.), stock market analysis, early detection of insider trading. However, financial anomaly detection is high risk, so this use case is challenging because it must be done truly real time so that it can be stopped as soon as it happens. Also, it’s more important perhaps than other use cases to be careful with false positives that may disrupt user experience.
Watch Video
  • Manufacturing and industry, construction, agriculture, etc.: For predictive maintenance or service fraud detection. Challenges are that industrial systems produce data from different sensors that varies immensely – different levels of noise, quality, frequency of measurement, which can make the data especially hard to work with.
  • Healthcare: Condition monitoring, including seizure or tumor detection. Challenges are that costs of misclassifying anomalies are very high; also, labeled data more often than not belongs to healthy patients, so usually have to adopt a semi-supervised or unsupervised approach.
  • Public sector: Used for detection of unusual images from surveillance. Challenges are that this use case requires deep learning techniques, making this type of anomaly detection more expensive.

Anomaly Detection in Dataiku

By far the most laborious step when it comes to anomaly detection is feature engineering. Continuing to iterate until false positives/negatives are reduced and the system is effective yet agile is a time consuming yet critical part of the process. 

What can be helpful is having a visual representation of the entire process so that iteration is simpler and faster every time, even once the model is in production – that’s where Dataiku comes in. Dataiku is the platform democratizing access to data and enabling enterprises to build their own path to AI, and it offers:

  • Robust AutoML features, including feature engineering, hyperparameter tuning, the ability to compare dozens of algorithms, and more.
  • One-click model deployment on the cloud with Kubernetes.
  • Model monitoring features to avoid model drift and actively monitor data changes over time.

Data Scientists and the AI Revolution

Data scientists are a critical piece of the AI puzzle, and they know what tools and technologies work best for any given task.

Read more

Go Further

Anomaly Detection for Retail

A step-by-step guide to incorporating machine learning-based anomaly detection techniques to a stock optimization use case in retail.

get the guidebook

Why Anomaly Detection Matters to Everyone

Across industries, there is no other data science strategy more important to understand and to leverage than anomaly detection.

learn more

Fraud Detection for Banks

A step-by-step guide to introducing machine learning-based anomaly detection techniques to fraud use cases in banking.

get the guidebook

Anomaly Detection for Healthcare Fraud

Given the immense cost, timely and effective fraud detection is imperative to improve the quality of care.

learn more