Discrimination and Bias

This section is predominantly for AI engineers and key decision makers in the development and use of AI products and services. It also provides foundation knowledge for DPOs and risk managers.

Ways to meet our expectations:

• Consider implementing measures to avoid discrimination against people because of protected characteristics at all stages of the AI system life cycle.
• Implement algorithmic fairness techniques such as: 
• pre-processing (adjusting training datasets);
• in-processing (applying factors or constraints in model training); and 
• post-processing (interventions and outputs). 
• Conduct analysis of algorithmic fairness limitations, such as: 
• unequal distribution of protected characteristics; 
• intersectional discrimination if a person fits multiple protected characteristics; and 
• reliance on false positives and true negatives.

• Maintain evidence that could assist with your fairness compliance obligations as the controller throughout the AI system supply chain.
• Ensure fairness is considered at different stages of the AI system;
• project design (eg by mapping of objectives, examining decision scope, reviewing bias measurability, consulting stakeholders, and adopting participatory design);
• before processing personal information;
• during information collection and procurement;
• during data analysis, labelling and pre-processing;
• in AI model evaluation and monitoring; and
• when retiring or decommissioning.

Options to consider:

• Think about whether your system may influence other groups indirectly, or the possibility that not all people in the group will be impacted in the same way because of their different contexts.
• Effectively frame the real-world problem the AI seeks to solve and
• clearly articulate the objectives.
• Explain why the AI system is applied to specific groups of people and not others. 
• Consider what information is needed to ensure a representative, reliable and relevant output, and what technical approaches you can implement to mitigate bias. For example: 
• reweighting; 
• removing the influence of protected characteristics or proxies;
• researching population; and 
• flagging risks.

Ways to meet our expectations:

• Include protected characteristics in the AI model, where appropriate or necessary, to ensure the system does not discriminate against these characteristics.
• Test these characteristics thoroughly to ensure they produce the right outputs consistently.
• Assess and document in a DPIA the risks associated with including protected characteristics and how the risks have been mitigated.
• Assess whether it would be a disproportionate effort to ask for additional information from people in order to proactively include protected characteristics (bearing in mind data minimisation principles).

Options to consider:

• Document the decisions made before including protected characteristics in the system design documents.
• Consider any 'anti-classification', identifying and excluding proxies for certain protected characteristics. Ensure decisions are made independently of certain protected characteristics so that they do not directly influence the outcome of a decision.
• Implement reasonable safeguards to ensure fairness (eg deploying feature engineering or input representations that explicitly capture and encode protected characteristics in the AI system's input features). This includes creating feature embeddings, one-shot encoding, or categorical variables representing protected attributes to ensure their inclusion in the model's input space.
• Design software development kits (SDK's) with accessibility in mind to meet the needs of people with disabilities.

Ways to meet our expectations:

• Document and implement a testing process for discriminatory outputs and bias and monitor it before implementation.
• Include in your DPIA an assessment of the potential discriminatory risks associated with processing personal information, particularly concerning people who need extra support to protect themselves.
• Use existing policy, technical, user research and design expertise to help determine sector-wide risks.
• Consider discrimination and bias right from the start of the design phase, including AI potentially processing special category and biometric data.
• Conduct appropriate assessments before deploying AI as a service to ensure there isn't potential for discriminatory outputs or decisions to be made.
• Integrate human-in-the-loop validation processes to review and validate the AI system's outputs for fairness and equity before go-live. This includes involving appropriate experts in the validation process to provide feedback and identify potential biases or discriminatory patterns.
• Test using new data sets to confirm the same outcome is reached.
• Adapt, change or retrain the AI following the review if necessary before go live.

Options to consider:

• Mitigate the following different forms of bias, if necessary:
• Societal and structural bias (from human involvement in AI design and development).
• Sampling bias (characteristics disproportionately represented in sample training data).
• Data labelling (data incorrectly labelled or labels are inadequate to capture nuances).
• Aggregation bias (characteristics unidentifiable or erased and invisible due to groupings).
• Evaluation and deployment bias (accuracy or over-simplification of AI processing).
• Automation bias (humans involved assume AI is right by default and do not effectively review decisions or outputs).

• Check that there is no imbalance in the training data used to train the system (ie over representation of one characteristic or group) and that it is representative of the population. 
• Establish fairness metrics and evaluation criteria to quantitatively assess the fairness and equity of the AI system's outputs across different demographic groups and sensitive attributes.