Responsible Data and AI is all about careful consideration of the consequences of building systems powered by Data and AI, on society, business, and ecosystem- with the intention of being ethical and benevolent to all parties.
In this post we discuss the key considerations towards this goal for a Software Architect going through the Software Development lifecycle(SDLC).
This becomes all the more critical in today’s age of ChatGPT and Generative AI, for several reasons. For starters, most Large Language Models (LLMs) consume most of the internet content for training. And internet has a lot of biased content created by humans. Further these LLMs may generate blogs and articles on the web using this skewed training, and make this data available for further model trainings. So this can create a very vicious loop. Some vendors may look at using books to train the models, however many books and movies are also now generated by AI.
So one big challenge for Architect is checking what data was used to build 3rd party models and which data should they use for building new models.
With Generative AI, Architects must also anticipate challenges like offensive content creation, opaque black boxes, and data breaches. The antidote lies in embedding checks at each stage of the SDLC to counteract these challenges, ensuring robust and responsible systems.
Some of the quality attributes that are very relevant for Data and AI systems are listed below, and this list may increase as new uses of AI proliferate.
Fairness:
AI should not discriminate against people in a way that leads to adverse decisions. The credibility of the data directly impacts the AI models. Historically, data in the world already may have unintended bias and prejudices as we live in an unfair world. One way to mitigate this is by having diverse stakeholders who ensure that data is balanced and not skewed against any gender or ethnic groups.
Human + AI:
Although AI may be making decisions automatically, it is still safer for a human expert to validate the final outcomes. The level of oversight could be for every decision or at least for the risky decisions. For example, a Doctor may oversee the cancer diagnosis inferred by an AI system, as lives are at stake.
At the other end of the risk spectrum could be a news article summarizer, and this could be entirely AI driven.
Privacy and Security:
Personal data of individuals need to be protected and secure. Apart from data regulation compliance, there could be also risk of harm to citizens if their sensistive data is leaked. For example, the EU PAPAYA project uses privacy-preserving data analytics to extract insights.
Safety:
AI systems should be safe and operate with reliability. For example, a self driving autonomous vehicle may need to be exposed to all variations of real world traffic patterns to minimize risk of accidents by means of wrong AI inferences.
Transparency and explainability:
Users of the system need to have transparency on the Data used for AI systems, how it is used, and at what points in the application they are interacting with AI. Even a blog written by AI should honestly declare so, versus being furtive about it.
Explainability is about understanding how AI has arrived at a solution for a range of people affected by the system.
Accountability:
Enterprises adopting AI should have a clear governance structure that makes it clear who is responsible for actions of an AI system, implementation of AI software, on-going monitoring and quality assurance.
Environment:
AI systems must be developed with careful consideration of carbon emissions and the subsequent environmental impact. For example, training the GPT-3 text generating model is akin to driving a car to the moon and back, according to this post.
Perhaps one trend we are sure to see is smaller language models built on a domain. So this is the Goldilocks challenge- General purpose GPT models are too broad and enormous while the traditional Deep learning models were only built to consume a project data and solve for the project use case. One too big and another too narrow. Goldilocks dilemma! Retail e-commerce, banking, manufacturing, and many industries have lot of repeatable processes and metrics- with some thin layers of differentiated specifics on top. So, it is imminent that many domain specific reusable domain models may emerge to fill this space. Huggingface for example provides tools for developers to start building smaller language models using various fine tuning and distillation techniques.
In software requirements stage the architects must explicitly include NFRs such as privacy, fairness, explainability, and environmental impacts. These requirements should be mapped further into design, development, and testing stages.
Traditional Requirements documents have focused more on overall system behaviour. Going forward there is need for data-centric requirements engineering phase. Data sourcing, lineage, ingestion, pre-processing, metadata augmentation, data augmentation, storage, usage, and delivery are areas that must be probed as part of the requirements elicitation, analysis activities.
In Design phase, decisions need to be made on the need of AI system versus a classical system. And, if we need AI then do we need traditional ML/DL or Generative AI- and in what parts of the application use cases. The actual design may perhaps be a hybrid of the the above choices for many applications. And then the question of how adaptive will AI be - AI in suggestion mode or decision mode and can that be dynamically changed at runtime. How will human integrate with AI. Will human be involved in checking every AI decision, or just have oversight and step into the critical risky cases, or be completely hand-off and trust the AI? How will tradeoffs be addressed such as between privacy preserving architectures and the environmental impact of added compute cycles?
Deployment stage would have considerations such as frequency of calibrating models with new data, version control of AI lifecycle artefacts including Data, code, models over time.
Testing is possibly the most important function in this game, and need to be involved end to end. Testing for data quality before ingestion into Data pipeline, detect problematic prompts, prevent prompt injections, thwart impersonation by means of Deepfakes, detect privacy violations, and model drift due to constantly changing data patterns. Tools such as the AI incident database can also be leveraged by testers to anticipate the types of incidents and contribute back to the repository based on what they uncover in testing.
So, in conclusion, Software Architects wield the power to mold technology and ethics into a cohesive whole. By embracing the delicate art of balancing ethical trade-offs, architects pave the way for responsible innovation that propels society forward. The responsible data and AI pipelines of tomorrow are the products of your choices today.
References:
EU Artificial Intelligence Act.
India Government Data Protection Act
Small language models .