A step-by-step guide to incorporating machine learning-based anomaly detection techniques to a stock optimization use case in retail.get the guidebook
Anomaly detection is all about finding patterns of interest (outliers, exceptions, peculiarities, etc.) that deviate from expected behavior within dataset(s). As with most data science projects, the ultimate end goal or output of anomaly detection is not just an algorithm or working model. Instead, it’s about the value of the insight that outliers provide. That is, for a business, money saved from preventing equipment damage, money lost on fraudulent transactions, etc. In health care, it can mean earlier detection or easier treatment.
A non-exhaustive look at use cases for anomaly detection systems include:
- IT, DevOps: Intrusion detection (system security, malware), production system monitoring, or monitoring for network traffic surges and drops. Challenges include the need for a real-time pipeline to react plus the huge volumes of data and unavailability of labeled data corresponding to intrusions, making it difficult to train/test. This industry usually has to adopt a semi-supervised or unsupervised approach.
- Banking and insurance: Fraud detection (credit cards, insurance, etc.), stock market analysis, early detection of insider trading. However, financial anomaly detection is high risk, so this use case is challenging because it must be done truly real time so that it can be stopped as soon as it happens. Also, it’s more important perhaps than other use cases to be careful with false positives that may disrupt user experience.