Regulators and organizations are paying more attention to the ethical risks of AI.
New questions on ethical implications, data privacy, or public safety are studied seemingly daily.
In the financial sector, in particular, concerns have been raised that AI – implemented as a tool to increase fairness in credit scoring, can in fact have the opposite result.
The most discussed ethical issue in the past year has been about “biased AI,” which can impact credit scoring or access to financial services, particularly in underbanked communities.
New AI credit scoring models rely on so-called “deep learning” – trained on vast amounts of data to “learn” patterns and generate an output – such as a credit score. These approaches are heavily dependent on the quality of the training data – with inherent biases meaning that scores will reflect historical prejudices against, for example, people of colour.
Many organizations have started to offer guidance.
The European Commission published new guidelines for the development of “trustworthy AI.”
The Guidelines put forward a set of 7 key requirements that AI systems should meet in order to be deemed trustworthy. A specific assessment list aims to help verify the application of each of the key requirements:
- Human agency and oversight: AI systems should empower human beings, allowing them to make informed decisions and fostering their fundamental rights. At the same time, proper oversight mechanisms need to be ensured, which can be achieved through human-in-the-loop, human-on-the-loop, and human-in-command approaches
- Technical Robustness and safety: AI systems need to be resilient and secure. They need to be safe, ensuring a fall back plan in case something goes wrong, as well as being accurate, reliable and reproducible. That is the only way to ensure that also unintentional harm can be minimized and prevented.
- Privacy and data governance: besides ensuring full respect for privacy and data protection, adequate data governance mechanisms must also be ensured, taking into account the quality and integrity of the data, and ensuring legitimised access to data.
- Transparency: the data, system and AI business models should be transparent. Traceability mechanisms can help achieving this. Moreover, AI systems and their decisions should be explained in a manner adapted to the stakeholder concerned. Humans need to be aware that they are interacting with an AI system, and must be informed of the system’s capabilities and limitations.
- Diversity, non-discrimination and fairness: Unfair bias must be avoided, as it could could have multiple negative implications, from the marginalization of vulnerable groups, to the exacerbation of prejudice and discrimination. Fostering diversity, AI systems should be accessible to all, regardless of any disability, and involve relevant stakeholders throughout their entire life circle.
- Societal and environmental well-being: AI systems should benefit all human beings, including future generations. It must hence be ensured that they are sustainable and environmentally friendly. Moreover, they should take into account the environment, including other living beings, and their social and societal impact should be carefully considered.
- Accountability: Mechanisms should be put in place to ensure responsibility and accountability for AI systems and their outcomes. Auditability, which enables the assessment of algorithms, data and design processes plays a key role therein, especially in critical applications. Moreover, adequate an accessible redress should be ensured.
Other organisations such as the OECD, Dutch Central Bank and bank of England have similarly started to set barriers to the use of AI.
Ultimately, any AI/ML model is as biased as the data we feed it.
This means that the data engineer will become an increasingly important stakeholder ensuring that bias is both identified and mitigated in any teaching data set