Engineers at Google proposed the concept of the model card in 2018 to increase ML transparency and create a way to communicate usage, ethics-informed evaluation, and limitations. The model card has since seen various levels of adoption, like this example for a Llama 3 model in Hugging Face. And as of the SAS Viya 2024.07, a model card is now available in SAS Model Manager.
The model card in SAS Viya was designed like a nutrition label for AI models. What sets the SAS Model Card apart from previous model cards is the use of descriptive visuals, to make model cards accessible to all personas involved in the analytics process, including data scientists, data engineers, MLOPs engineers, managers, executives, risk managers, business analytics, end-users, and any other stakeholder with access to the SAS Viya environment. And SAS is so confident in our model card design, we’ve even presented it before the US Congress. Additionally, most of our model card populates automatically as the model is developed, managed, and deployed in SAS Viya. With AI, trust is everything. With SAS Viya, it takes 5 minutes to build a model card that communicates performance, fairness, and transparency with AI.
SAS is taking a thoughtful approach to how it helps customers embrace AI, focusing on the practical realities and challenges of deploying AI in real industry settings,” said Eric Gao, Research Director at analyst firm IDC. “Model cards will be valuable for monitoring AI projects and promoting transparency.”
The model card has five sections: Overview, Model Usage, Data Summary, Model Summary, and Model Audit. The Overview section is the first thing you see when you open a classification or prediction model in SAS Model Manager. This section provides an at-a-glance summary of model health and critical factors so a decision maker can quickly understand if their model is a good candidate for deployment, needs work, or needs to be replaced as their production model.
The following sections allow teams to dig deeper so that they can pinpoint issues surfaced in the overview and determine what areas need to be addressed. The Model Usage consists of manual fields that highlight the intended usage, expected benefit, out-of-scope use cases, and limitations of the model. This provides critical information for ensuring that the model is being used as designed.
The Data Summary pulls information from SAS Information Catalog to provide metrics about the training data. This includes information about data completion, information privacy classifications, size, and status. With the data summary, users can quickly find if their model leverages sensitive or private variables as well as dive deeper into the data used to train the model.
The next section is the Model Summary, which graphs various metrics of model performance on the training, testing, and validation splits. The Model Summary also visualizes model fairness and model interpretability metrics, if available.
The final section is the Model Audit tab, which summaries the model’s performance on the latest performance monitoring data. Users can set thresholds in SAS Model Manager using the Key Performance Indicators (KPIs), which are used to determine if the model is currently performing as expected. When a model decays, it becomes less effective as a predictor. This means that organizations must act quickly to put a new model in production to ensure they can make accurate decisions using their models.
Model cards will be freely available on SAS Viya platforms running the 2024.07 release (or later) that include SAS Model Manager. If you have the SAS Viya, SAS Viya Advanced, or SAS Viya Enterprise foundational offerings, the model card will become available in your next update. Organizations adopting monthly updates will be able to pick up the model card immediately, whereas teams on the LTS path can pick up the model card in their next Long Term Stable release.
Don’t have access to the latest SAS Viya environment? Not a problem! We have a few paths available for trying the new model card:
- Take advantage of the free SAS Viya trial. The free trial is on the LTS release cadence, so you may need to wait until Winter for Model Cards to appear, but this is a free two-week trial of the SAS Viya Enterprise offering. You can add multiple users, so you can create and share the model card with your team. You can start this trial at any time and the trial environment takes care of deployment and infrastructure of the SAS Viya platform.
- Need more time? SAS Viya on the Azure Marketplace includes everything in the SAS Viya foundational offering and lets you pay-as-you-go. And just like the free trial, you can start at any time without worrying about the platform deployment or infrastructure. This offering is updated monthly, so you can try out the model card immediately!
- The SAS Hackathon is just around the corner and provides an opportunity to leverage the latest tooling on SAS Viya to solve a problem of your choice. Participating in the month-long Hackathon is free, but you do need to register by August 30th.
Once you have access to the latest SAS Viya environment, you can read more about how each section of the model card is completed in this article. Once you have tried out the model card, we would love to hear your thoughts! Please let us know in the comments what worked well, what use cases you tried, and what you would like to see in future iterations of the model card as we build out a base of recommended best practices.
Sophia Rowland, Kristi Boyd, Vrushali Sawant, and Allie DeLonay contributed to this article.
