FSRA: AI-Powered Risk Assessment for Financial Services

As a financial services regulator, The Financial Services Regulatory Authority of Ontario (FSRA) oversees mortgage brokers, life and health insurance agents, credit unions, pension plans, financial planners, financial advisors, and more. As a regulator, they are forward-looking, but always careful to ensure consumer protection is never jeopardized. 

FSRA aimed to democratize analytics and AI across the enterprise, implement an AI Governance framework, and develop an AI-powered business application, but they faced a myriad of obstacles along the way. 

They did not have:

• Governing mechanisms for AI
• Tools to integrate and analyze disparate data sources
• The ability to automate internet-based background checks to improve productivity and access critical data effectively 
• A centralized data science platform to do all of this work in

Enter: Dataiku. 

Overcoming the Roadblocks With Dataiku

FSRA wanted a comprehensive data science platform that could bring together professionals from the business, IT, data science, and other areas of the organization. They chose Dataiku for its comprehensive capabilities, flexibility, and user-friendly interface. 

The team at FSRA needed to build its first AI-powered business application to address several tactical challenges:

• The automation of internet-based background checks, which were manually conducted for thousands of applications and had become untenable due to the volume ramp-ups.
• The existing process was not sustainable for the long run, requiring automation, scalability, and the synthesis of background checks based on specific business criteria.
• There were numerous disparate and legacy databases, each with different schemas, formatting, and unstructured data inputs, which made it hard to integrate and analyze the data efficiently and effectively.
• The business had to manually search each database separately and then combine the search results, which led to a massive loss in productivity. 
• Unstructured data sources, often embedded in comment fields inside the databases, presented a known but virtually inaccessible source of valuable information, which limited the business’s ability to access and use that data effectively. 
• The team lacked a single view that encompassed all the risk signals from external and internal data sources, so the ability to assess the full complement of risks accurately and make informed decisions was severely hampered. 

With Dataiku in the picture, the team leveraged pre-trained AI models such as Spacy for NER and Word2Vec for embeddings to develop custom search capabilities for various watchlists and checks. The team also developed custom keyword dictionaries to train the models. By training the AI models to filter search results based on specific risk criteria, the team automated the process and significantly improved scalability.

Dataiku enabled users to leverage Large Language Models (LLMs) easily to address natural language understanding problems that were otherwise unattainable using traditional AI models. This capability empowered FSRA to make informed decisions based on natural language analysis.

Furthermore, Dataiku’s native support of Python and R enabled developers to rapidly write code, leverage third-party APIs, and build custom recipes and functions to perform numerous tasks that would otherwise have entailed much heavier development and testing. Thanks to the visual interface of Dataiku, the development team was able to build, test, and audit models much faster than normal.

To consolidate the risk signals from various sources, FSRA leveraged Dataiku’s capabilities for data integration, cleansing, and transformation. The team developed a unified view of risk signals derived from structured and unstructured data, including data from the internet, internal databases, and applicant-submitted media. By consolidating and synthesizing these signals, the “Decision Support” capability provided a clear and prominent risk indicator to support decision-making.

The collaborative nature of Dataiku allowed teams to seamlessly integrate these capabilities, ensuring a cohesive and comprehensive solution. Throughout the project, Dataiku facilitated collaboration, enabling users across different roles to interact with and contribute to the development process.

Finally, Dataiku’s emphasis on AI explainability, ease of oversight, and a standalone AI Governance offering suited FSRA’s needs to operationalize its newly established AI Governance Framework, which describes FSRA’s policy and guidelines concerning data science, machine learning (ML), and AI.

Drilling Into the Results 

Implementing Dataiku has significantly transformed day-to-day operations at FSRA. It has had a profound impact on business processes, leading to increased efficiency, enhanced decision making, and improved regulatory oversight. Key tactical achievements include:

• 80% reduction in the time to manually review applications
• Automated searches for over 150 risk signals that previously required manual intervention
• Deployed the solution from pilot-to-production in only 12 weeks

FSRA’s first AI-powered application (known as the Decision Support Portal or “DSP”), made possible by Dataiku, has led to significant improvements in background checks, data integration, and document analysis. By automating internet-based background checks, manual processes have been streamlined, allowing for faster and more comprehensive background checks. Integrating diverse data sources has optimized operations by providing a centralized view, improving data accessibility, and boosting productivity. 

Further, the DSP contributed to increased team efficiency, enhanced tech stack efficiency, and improved risk management and governance through transparency and explainability, as well as opportunities for upskilling and networking. By providing visibility into the inner workings of models and algorithms, Dataiku ensured transparent deployment, which instilled confidence and trust amongst FSRA stakeholders, as they could understand and interpret model results. 

By automating manual processes and offering advanced AI functionalities, Dataiku significantly reduced the time and effort required for complex regulatory tasks, resulting in increased speed and agility. This allowed FSRA to respond promptly to regulatory changes and industry dynamics, maximizing operational efficiency. Overall, Dataiku has greatly enhanced efficiency and compliance at FSRA.

A Crawl-Trust-Walk Approach to GenAI

Maurice Chang, AI/ML architect at FSRA — alongside business, transformation, and technology stakeholders — developed a crawl-trust-walk approach for deploying the technology. Simply announcing a plan to use ChatGPT to regulate financial services would likely have created a negative reaction, so instead the team began with a simple GenAI use case by augmenting the DSP that is already used in production, comfortably used by the business, and includes the 150 risk signals mentioned above.

The use case adds just one new risk signal: whether a particular case summary searched from the Canadian Legal Information Institute mentions a phrase as respondent, defendant, or appellant. Given the unstructured nature of case summaries, this simple “Y/N” signal could not be reliably produced using traditional NLP methods. Enter: GPTs.

After workers used the matched data, provided feedback, were comfortable with its quality and trusted it, then the team began socializing that it was enabled by GenAI. The first use case opened minds and doors. Workers were receptive to the prospect that this new technology could improve the quality and speed of their day-to-day tasks. The number of users and use cases multiplied in a matter of months and the team is continuing to experiment with additional GenAI use cases.

AI-Driven Cultural Change

With the introduction of Dataiku, the tools and resources for data science and AI have become more widely accessible to a larger number of individuals within the organization. Now, business analysts can directly engage in data science activities, leading to improved efficiency and more effective collaboration with business and IT professionals. This new level of involvement has allowed FSRA to explore the possibilities of data science and AI and has demonstrated to the business the potential impact and benefits of these technologies.

Furthermore, the Data Science Lab at FSRA has greatly benefited from the incorporation of enterprise MLOps capabilities. The ability to operationalize ML models and processes within Dataiku has allowed the Lab to better manage various aspects of their data science workflows. 

This has not only enhanced the productivity of the Lab, but also facilitated the deployment and management of AI systems across the organization. 

With these capabilities, FSRA can more easily implement and monitor the AI Governance framework they’ve developed, ensuring that the application of AI technologies aligns with the organization’s goals and regulations.