AI Governance

Designing Truly Human-Centered AI

Kyle Ledbetter
VP of Design
December 8, 2022
No items found.

Guidelines and best practices to ensure that AI systems are designed with human values and experiences at their core.

As we enter the era where AI has the potential to impact almost every aspect of our lives, there is a growing need to ensure that AI systems are designed with human values and experiences at their core. This is a high-level introduction to Human-Centered AI (HCAI), a Responsible AI methodology for anyone building or using AI systems (such as researchers, data scientists, developers, and product managers).

An Evolution of Human-Centered Design

As Head of Design at Credo AI, the first step in designing the Credo AI Responsible AI Platform was to ensure that we address and consider real people’s needs and concerns at every stage of our software design and development lifecycle. Humans are never an afterthought. The value and real world outcomes of our product cannot be unintended and accidental. Also, these needs change over time and must be constantly evaluated and revisited. It is our belief that these same approaches, methodologies, and principles can apply to the development and usage of Artificial Intelligence.

Context is Key

For truly Human-Centered Design, societal, cultural, and any contextual information is crucial to consideration of design. The same applies to Responsible AI. When designing, developing, and assessing an AI Use Case the unique blend of the industry, model type, training datasets, downstream populations impacted, and stakeholders involved must all be considered at every stage. This application of Human-Centered Design to AI is logical evolution as “Human-Centered AI”.

Human-Centered AI

  1. Humans must consider humans when developing AI
  2. Humans must be represented in AI training and validation data
  3. Humans must be involved in the governance and monitoring of AI
  4. Human stakeholders must be able to understand how the AI works
  5. Humans must be valued over performance and profitability of AI

Humans must consider humans when developing AI. 

As our Head of Product, Susannah Shattuck says, "if you’re not validating your system for alignment with your ethics and values until it’s already been built, right before deployment, then you’re setting yourself up for failure."

Evaluating and managing human impact of an AI system is something that needs to happen throughout the design, development, and deployment lifecycle, at every step along the way. You must start with the use case, the context, and how humans will be impacted. When we develop new technologies, the human aspect must be considered at the very beginning, not just evaluated before deployment.

Hypergiant offers a useful framework for evaluating a use case for ethical decision making, called “TOME” (Top Of Mind Ethics):

  1. Establishing Goodwill
    Does this choice reflect positive intent alignment with our company values?
  2. Categorical Imperative
    View all of society as our stakeholders. If every company in our industry, and every industry in the world used AI in the way proposed, what would the world look like? Is that a world we want to live in?
  3. Law of Humanity
    What impact will we have on society if we deploy AI in this way? Are we using people simply as a means to an end?

At Credo AI we onboard every new customer with an AI Use Case alignment session, we bring together diverse stakeholders to come together and align on the intent and impact of their AI Use Case. Also, in the Credo AI application, a Use Case prerequisite is to provide contextual information that will lead to recommendations for regulation and compliance, as well as technical assessment guidance.

(All) Humans must be represented in AI training and validation data. 

A large percentage of bias is introduced via the training data during model development. This is partially due to the challenges in the collection and access to data for training and validation.

By appropriately representing the people who will be impacted in AI training and validation data, you can gain assurance that AI systems will reflect human values and do not perpetuate biases or discrimination.

A dataset fairness assessment is a value method of evaluating the objectivity, completeness and robustness of the data in a given dataset. These assessments can be used to determine whether a set of data is biased or if it is representative of all groups within the population.

Humans must be involved in the governance and monitoring of AI. 

Not only must humans be involved, those humans must be diverse in background and profession. This is why Credo AI was designed as a multi-stakeholder product from the start. 

Humans must be involved and in-the-loop of governance and monitoring of AI. Although AI has proven to be generally reliable, it can still produce errors with serious consequences.

To avoid these possible issues, it’s important that organizations establish a safety net that includes human checks and balances for when something does not work as expected.

Human stakeholders must be able to understand how the AI works. 

A core tenant of Responsible AI is transparency, which includes explainability. Setting context for an AI Use Case is step 0 for transparency, and is the starting point of the journey working with our team.

A human simply cannot govern AI if they can't understand how it functions and what it's supposed to do. It sounds simple but it's extremely hard to do. Also, the downstream humans impacted by AI must be made aware AI is being used and how it's affecting them.

This transparency isn't just necessary for use cases like home loan approval and credit review, it’s also critical for retaining trust with employees and earning trust with potential candidates in the hiring process. Perhaps the most crucial need for transparency arises in the use of AI in the government, police, and military. 

Humans must be valued over performance and profitability of AI. 

The most common dilemma in adopting Responsible AI (RAI) risks is valuing profits and performance over people. As an AI development community, we must be aware of limitations and be ready to trade-off performance in favor of fairness. 

Once adopted, RAI governance and best practices make this decision easier. If you are aware and considering these factors earlier in the ML development cycle, it's far easier to acknowledge them and adjust accordingly before the business is sold on certain performance metrics. 

It is easy to put the bottom line first, but we must ensure that the human beings impacted by AI are our primary concern.

In closing, which scenario sounds better?

A: AI enhancing and augmenting humans to reach new heights (hopefully the obvious choice)


B: AI operating in the shadows and impacting our lives in unimaginable ways (yikes)

In the end, is the goal for humans to become data points and resources for a system run by AI? Or for humans to become augmented by AI like enhanced superheroes?

And if it’s a bit too science fiction to imagine being a superhero team, perhaps a super team of scientists, developers, designers, and product visionaries is more accurate…

Did you spot the use of AI throughout this publication?

All of these images were generated with DALL·E 2, and sections of text were edited with These are basic and humble beginnings of how AI can be used as an enhancement for a task, yet they’re already introducing incredibly thought provoking questions of ethics, licensing, rights, and responsibility.

You may also like

NYC AI Bias Audit: 7 Things You Need to Know About the Updated NYC Algorithmic Hiring Law

The clock is ticking, and we at Credo AI are committed to helping organizations understand what the recent updates and key aspects of the new regulation mean for them. In today’s blog post, we will outline seven things employers and employment agencies need to know about the requirements for NYC AI Bias Audits, also known as NYC Local Law No. 144.

Credo AI Lens™: the ultimate open-source framework for Responsible AI assessments

While ML capabilities have developed at a staggering pace, guidelines and processes to mitigate ML risks have lagged behind. That’s where AI Governance comes in, which defines the policies and processes needed to safeguard AI systems. While there are many components to successfully operationalize the responsible development of AI systems, a chief need is assessing AI systems to evaluate whether they behave adequately for their intended purpose. This assessment challenge is central to Credo AI’s efforts to develop AI Governance tools. In this blog, we introduce Credo AI Lens, our open-source assessment framework built to support the assessment needs of your AI Governance process.

CEO Message: A look at 2022 and a glimpse into 2023

It's been an impact-driven year, and while it is not possible to share all that has happened, I want to take a moment to highlight some industry-shaping product, policy, and ecosystem moments with you. In this year-end review, I am elated to spotlight some of Credo AI’s most significant achievements and reflect on our progress in addressing the challenges and opportunities of the AI industry.

Join the movement to make
Responsible Al a reality