Bridging the gap between promises and practice

As AI continues to integrate into everyone’s lives, the importance of responsible AI is becoming increasingly crucial. However, there is a significant difference between the commitments made by companies like Google or Meta and what developers can implement in their AI/ML projects. In this blog post, we want to bridge the gap between corporate principals and things developers can implement immediately. We’ll highlight concrete tools, processes, and explain how to operationalize them.

Why responsible AI matters

With the increasing use of AI in our lives, trusting AI systems is crucial. AI systems can cause real-world harm, ranging from discrimination in hiring processes and loan denials for minorities to medical misdiagnoses and concerns about surveillance. Because of the increasing number of failing AI systems (see examples below), AI systems are scrutinized by users (for good reason). Those violations erode confidence in AI systems.

At the same time, regulatory pressure has increased with the emergence of the EU AI Act, U.S. state-level laws, and industry-specific requirements (e.g., ISO 42001).

Examples of AI failures

• Amazon’s AI recruiting tool: Amazon developed an AI hiring tool that systematically discriminated against women by penalizing resumes containing the word “women’s” or names of all-women colleges, trained on 10 years of predominantly male resumes. Sources: ACLU, MIT Technology Review

• Mortgage lending disparities: A 2022 UC Berkeley study found that African American and Latinx borrowers are charged nearly 5 basis points in higher interest rates than credit-equivalent white borrowers. Source: RFK Human Rights Center

• Medical misdiagnoses: CNN-based skin lesion classification systems trained primarily on white patients have approximately half the diagnostic accuracy for Black patients, whose lesions may have different characteristics. This is particularly serious as Black patients have the highest mortality rate for melanoma, with a 5-year survival rate of only 70% versus 94% for white patients. Source: National Library of Medicine

• Facial recognition accuracy: A 2018 study by the MIT Media Lab found error rates of 0.8% for light-skinned men, compared to 34.7% for darker-skinned women. Federal government research found that facial recognition systems work best on middle-aged white men and have the highest error rates for Black women. Source: ACLU

• AI and privacy: An independent researcher found that over 140,000 chats with different GenAI tools are publicly available and searchable through archive.org. These data from these chats include sensitive information and API tokens that were made finable when users selected the “share” option from the chatbot. Julian B.

The general corporate messaging

With the hype around AI and machine learning, corporations have highlighted themes around fairness, transparency, accountability, privacy, and safety. However, the messaging often remains stuck at the principles stage, focusing on public relations and talent attraction. The primary issue with most initiatives is that the concepts remain abstract without being operationalized. It is often impossible for developers to bridge the gap between aspirational principles and the actions they can implement.

The developer reality gap

While the corporate attention is great, developers receive only vague guidance. Often, corporate principles don’t translate easily into action. There has been a shortage of tools for detecting bias in models, explaining models, and testing safety. Much worse: In various companies, responsible AI is often treated as a “nice to have” when deadlines loom, rather than halting the entire release.

While we are in the midst of the AI hype cycle, many companies are prioritizing speed over features, fairness audits, red-teaming, or learning from model evaluations. Unfortunately, there is a significant misalignment in the metrics surrounding machine learning.

This misalignment is often unintentional and frequently rooted in a knowledge gap among developers or executives. Most software developers have not received training in AI ethics or safety. This pattern is often “enforced” by organizational silos, where teams focused on ethics are disconnected from the engineering workflows.

To help bridge this gap, Dataiku has developed the RAFT framework, which outlines four core principles for the responsible development and use of AI, as illustrated below. Based on these principles, we propose criteria and indicators to develop more trustworthy AI systems.

While we have proposed some example criteria and indicators in our RAFT framework, we encourage the developer community to add to or refine this list.

What developers can do

There are several ways developers can incorporate responsible machine learning into their workflows, aligning with the core principles of the RAFT framework.

Reliable and secure

With models and AI applications, evaluate the models or the APIs. Build time into your development sprint upfront to test models and understand when the prompt is failing. You can then extend the evaluations to include adversarial testing. This could include creating model inputs that generate a very different model response than what the system is designed for.

Probably, the most often overlooked option is proper documentation of the decisions made while developing a machine learning model or incorporating a model API. Keep records of tradeoffs and limitations. For example, if you are incorporating a model API from a model provider, keep records of which temperatures you have tested your prompt with (as you would for other parameters).

Fair and human-centric

Incorporating user feedback loops is an integral part of responsible AI. Beyond offering users a vital channel for real-time input on your AI application, this practice serves a dual purpose: it continuously expands your dataset, enabling ongoing fine-tuning of underlying models or optimization of prompts.

Beyond basic validation, truly responsible AI requires rigorous testing and examination. If you want to further test your models for bias and unfair outcomes, you can utilize specialized tools and open-source fairness libraries. These tools are designed to help further stress test the models under various demographic or sensitive attribute conditions.

Libraries like IBM’s AI Fairness 360 (AIF360) and Microsoft’s Fairlearn offer comprehensive frameworks for detecting and mitigating unfairness in machine learning models throughout the AI lifecycle.

• AIF360 offers a comprehensive suite of metrics for assessing fairness, algorithms for mitigating bias, and tutorials to guide practitioners in applying these techniques.

• Fairlearn focuses on a different aspect, primarily assessing model performance disparity across different groups and offering methods to improve fairness while maintaining accuracy.

Transparent and explainable

While documenting your design decision and model evaluations, you will learn tremendously how the model and your overall application will behave. Share the lessons learned and failures — your learnings will help others to build their intuition in understanding AI and how systems work. Share your stories when you were taken completely off guard in the model’s prediction or response generation. Share this learning with the community. This could be via a small lunch and learn talk with your colleagues, a short blog post or a talk at a local AI Meetup.

Model and data cards are excellent tools for sharing knowledge about your AI applications. They provide users with valuable insights into the limitations of the models and data, making them an effective way to communicate your learnings. By providing model and data cards, you automatically motivate other developers to share their learnings as well.

Utilizing these advanced tools is a crucial step in a robust responsible AI framework, moving beyond simple accuracy metrics to ensure the ethical and equitable deployment of models.

Dataiku’s commitment to responsible AI

Dataiku’s Open Source Lab is dedicated to fostering open and transparent responsible AI. We believe deeply in this principle, especially when it comes to understanding the decision-making of AI agents. To this end, we are committing engineering resources to two key areas: developing tools to understand AI agent decisions and providing privacy-preserving tools for users working with closed-source models (e.g., OpenAI).

If you are interested in learning more about what we do and how you can use the tools for your business applications or development projects, visit us at …dataiku.com or join our responsible AI office hours.

Conclusion

Bringing more responsible AI practices into the developer workflows often seems insurmountable. Transitioning from abstract corporate speak to tangible implementations is a challenge, but we hope this blog post outlines a way to start small. Getting started doesn't require overhauling an entire system; simple, high-impact concepts, such as maintaining thorough decision documentation and proactively sharing lessons learned across teams, offer an excellent starting point for building a more responsible AI development.

Call to action

If you are interested in learning more about what we do and how you can use the tools for your business applications or development projects, visit us at dataiku.com/open-source or connect with us via [email protected].