Branchoria AI Sense AI Sense

Clear, practical explanations of what AI does, where it helps, and where it fails.

Within Model Cards

Where should a model not be used?

Model cards help decision-makers see where a system was tested, what it was intended to do, and when using it would exceed its validated scope.

On this page

  • What intended use statements should make clear
  • Why assisted use differs from automated decisions
  • How performance reporting exposes weak spots
Preview for Where should a model not be used?

Introduction

A model that performs well in testing can still become unsafe when deployed in the wrong setting. Model cards were created in part to address this problem. They are structured documents that describe what a model was designed to do, how it was evaluated, where it performs well, and where its creators believe it should not be used. Rather than treating deployment as a simple question of accuracy, model cards help organisations identify the boundaries of a model’s validated use. [arXiv]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

Model limits illustration 1 This matters because many AI failures arise from context changes rather than software defects. A model trained and tested for one environment may be deployed in another with different users, populations, risks, or decision stakes. By documenting intended uses, limitations, evaluation results, and known weaknesses, model cards make those deployment boundaries visible before decisions affect real people. [arXiv+2Alan Turing Institute]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

Where should a model not be used?

The most valuable model cards do not simply advertise capabilities. They identify situations where a model has not been validated or where its use could create unacceptable risk.

The original model card framework proposed by researchers at Google emphasised that documentation should clarify intended use cases and help minimise deployment in contexts for which a model is not well suited. The goal is not merely transparency but prevention of misuse. [arXiv]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

In practice, deployment boundaries often emerge from three questions:

  • Was the model tested in this environment?
  • Was it tested on people or conditions similar to those it will encounter?
  • Are the consequences of errors consistent with the original design assumptions?

If the answer to any of these questions is unclear, a model card should signal caution. A language model tested for drafting marketing text, for example, has not automatically been validated for medical diagnosis. A facial analysis system evaluated on adult populations may not be appropriate for use with children. The model itself may not have changed, but the deployment context has. [arXiv+2NIST]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

For governance teams, these statements create a documented basis for rejecting inappropriate deployments before they occur.

Model Cards for Model Reporting

Channel: ZazenCodes

Open on YouTube

What intended-use statements should make clear

An intended-use section is often the first place where unsafe deployment boundaries become visible.

A useful statement does more than describe a model’s purpose. It identifies the specific task, expected users, operational environment, and assumptions underlying evaluation. Model card guidance commonly recommends documenting both intended uses and out-of-scope uses so that readers can distinguish approved applications from unsupported ones. [VerifyWise+2Website]verifywise.aiModel Cards for Model Reporting | KI-Governance-BibliothekModel cards provide standardized documentation covering intended uses…

Strong intended-use documentation typically answers questions such as:

  • Who is expected to use the model?
  • What decisions is it meant to support?
  • What populations or environments were represented during testing?
  • Which uses are explicitly discouraged?
  • What level of oversight is required?

The difference between “supports content moderation review” and “automatically removes user content” is not a minor wording change. It signals a fundamentally different deployment boundary. Likewise, a model card that specifies use in research settings but not operational decision-making is communicating a governance constraint, not merely technical information. [arXiv+2Alan Turing Institute]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

Without these boundaries, organisations may assume that successful performance on one task justifies broader deployment than the evidence supports.

Why assisted use differs from automated decisions

One of the most important deployment distinctions concerns whether humans remain involved in decision-making.

Many AI systems perform adequately when providing recommendations to trained users but become substantially riskier when granted authority to make final decisions automatically. Model cards can expose this difference by documenting the conditions under which evaluations were conducted and the role humans were expected to play. [arXiv]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

Consider a model that identifies potentially harmful online content. If evaluators assumed human moderators would review flagged material, measured performance reflects a human–AI partnership. Deploying the same system as a fully automated enforcement mechanism changes the risk profile. False positives and false negatives now directly determine outcomes.

The same principle applies in healthcare, employment, education, and public services. A recommendation system may function as a decision-support tool while becoming unsafe when converted into an autonomous decision-maker. The model card helps reveal that the original validation did not necessarily cover the automated scenario. [arXiv+2NIST]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

For governance purposes, this distinction is crucial because oversight mechanisms often serve as safety controls. Removing those controls may push deployment beyond the model’s documented scope.

Model limits illustration 2

How performance reporting exposes weak spots

Aggregate accuracy figures can hide the conditions under which a model struggles. One of the most influential features of model cards is their emphasis on reporting performance across different groups and operating conditions rather than relying on a single headline metric. [arXiv]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

The original model card proposal specifically encouraged reporting results across demographic, cultural, geographic, and intersectional groups. This approach helps identify populations for which performance may be weaker than average. [arXiv]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

For deployment decisions, these details matter more than overall averages.

Imagine a model with: [arxiv.org]arxiv.orgOutside of machine…

  • High overall accuracy.
  • Strong performance for most users.
  • Significantly lower performance for a specific demographic group.

A summary metric may make the system appear ready for deployment. A detailed model card may reveal that certain groups face substantially higher error rates. The deployment boundary then becomes visible: the model may not be suitable for applications where those disparities create meaningful harm. [arXiv+2VerifyWise]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

Performance reporting can also expose weaknesses linked to language, geography, environmental conditions, image quality, data scarcity, or unusual user behaviour. These findings help organisations determine whether additional testing or safeguards are needed before deployment expands.

Securing the AI Stack: Model Deployment Security, AI Inference Vulnerabilities Explained

Channel: SystemDR - Scalable System Design

Open on YouTube

When a deployment exceeds the validated scope

A model card does not guarantee safety. Organisations can still ignore documented limitations. However, model cards create an evidence trail showing whether deployment decisions remain consistent with available validation evidence.

The broader AI governance community increasingly emphasises understanding context, intended use, and operational conditions before deploying AI systems. Frameworks such as the NIST AI Risk Management Framework stress that AI risks depend heavily on context of use and that organisations should define system boundaries before making deployment decisions. [CSER+3NIST+3NIST Publications]nist.govAI Risk Management Framework | NISTNIST has developed a framework to better manage risks to individuals, organizations, and society a…

A deployment may exceed validated scope when:

  • The model is used for a different task than originally evaluated.
  • The target population differs substantially from the tested population.
  • The operational environment changes.
  • Human oversight assumptions are removed.
  • The consequences of errors become more severe than those considered during evaluation.

In each case, the model card acts as a reference point. Decision-makers can compare the proposed deployment against documented evidence rather than relying on assumptions about general AI capability.

Model limits illustration 3

Model limits as a governance tool

Model cards are often described as transparency documents, but their governance value is more specific. They help organisations identify where confidence should stop.

A well-designed model card does not merely say that a model works. It explains where the evidence ends. Intended-use statements define acceptable applications, assisted-use guidance clarifies the role of human judgement, and detailed performance reporting reveals populations or conditions where reliability may deteriorate. Together, these elements expose unsafe deployment boundaries before a system is placed into a new context. [Website+3arXiv+3Alan Turing Institute]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

For anyone evaluating AI before deployment, that may be the most important function of a model card: not showing where a model can be used, but revealing where it should not. [arXiv]arxiv.orgarXiv Model Cards for Model ReportingModel Cards for Model ReportingOctober 5, 2018…Published: October 5, 2018

Model Cards: A Transparency Tool For AI Driven Marketing

Channel: AI Loves Data

Open on YouTube

Amazon book picks

Further Reading

Books and field guides related to Where should a model not be used?. Use these as the next step if you want deeper reading beyond the article.

Browse more on Amazon: AI Snake Oil Human Compatible Weapons of Math Destruction

As an Amazon Associate I earn from qualifying purchases.

eBay marketplace picks

Marketplace Samples

Example marketplace items related to this page. Use the search link to explore similar finds on eBay.

Using USA
Browse more on eBay.co.uk

Example items shown for inspiration; availability and pricing can change. Branchoria may earn a commission if you purchase through outbound eBay links.

Endnotes

  1. Source: arxiv.org
    Title: arXiv Model Cards for Model Reporting
    Link: https://arxiv.org/abs/1810.03993
    Source snippet

    Model Cards for Model ReportingOctober 5, 2018...

    Published: October 5, 2018

  2. Source: verifywise.ai
    Link: https://verifywise.ai/de/ai-governance-library/transparency-and-documentation/model-cards-paper
    Source snippet

    Model Cards for Model Reporting | KI-Governance-BibliothekModel cards provide standardized documentation covering intended uses...

  3. Source: arxiv.org
    Link: https://arxiv.org/pdf/1810.03993
    Source snippet

    Outside of machine...

  4. Source: nist.gov
    Link: https://www.nist.gov/itl/ai-risk-management-framework
    Source snippet

    AI Risk Management Framework | NISTNIST has developed a framework to better manage risks to individuals, organizations, and society a...

  5. Source: cser.ac.uk
    Title: response nist
    Link: https://www.cser.ac.uk/work/response-nist/
    Source snippet

    Response to NIST AI Risk Management Framework...24 Jan 2022 — 1. On the 'Map' function, the Paper notes that “Context refers to the doma...

  6. Source: nvlpubs.nist.gov
    Title: ai.100 1
    Link: https://nvlpubs.nist.gov/nistpubs/ai/nist.ai.100-1.pdf
    Source snippet

    NIST PublicationsArtificial Intelligence Risk Management Framework (AI RMF 1.0)by N AI · 2023 · Cited by 181 — The Framework is intended...

  7. Source: nist.gov
    Title: launch nist ai risk management framework
    Link: https://www.nist.gov/speech-testimony/launch-nist-ai-risk-management-framework
    Source snippet

    It provides a flexible, but structured and measurable approach to understand, measure and manage AI...Read more...

  8. Source: nvlpubs.nist.gov
    Title: AI.600 1
    Link: https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
    Source snippet

    Intelligence Risk Management Frameworkby N AI · 2024 · Cited by 91 — As GAI covers risks of models or applications that can be used acros...

  9. Source: nist.gov
    Title: risk management framework aims improve trustworthiness artificial
    Link: https://www.nist.gov/news-events/news/2023/01/nist-risk-management-framework-aims-improve-trustworthiness-artificial
    Source snippet

    NIST Risk Management Framework Aims to Improve...Jan 26, 2023 — The AI RMF provides a flexible, structured and measurable process that w...

  10. Source: airc.nist.gov
    Link: https://airc.nist.gov/airmf-resources/playbook/
    Source snippet

    NIST AI Resource CenterThe Playbook provides suggested actions for achieving the outcomes laid out in the AI Risk Management Frame...

  11. Source: airc.nist.gov
    Title: 5 sec core
    Link: https://airc.nist.gov/airmf-resources/airmf/5-sec-core/
    Source snippet

    RMF Core - AIRC - NIST AI Resource CenterThe AI RMF Core provides outcomes and actions that enable dialogue, understanding, and activitie...

  12. Source: airc.nist.gov
    Link: https://airc.nist.gov/airmf-resources/airmf/0-ai-rmf-1-0/
    Source snippet

    Summary - AIRC - NIST AI Resource CenterThe goal of the AI RMF is to offer a resource to the organizations designing, developing, deployi...

  13. Source: verifywise.ai
    Link: https://verifywise.ai/solutions/nist-ai-rmf
    Source snippet

    NIST AI RMF implementation guideAI system inventory and context mapping. Register every AI system with structured metadata covering inten...

  14. Source: arxiv.org
    Link: https://arxiv.org/pdf/2403.15646
    Source snippet

    Description. Primarily for law enforcement to identify suspects by matching faces with a vast internet-sourced database. Potential impact...

  15. Source: ai.google.dev
    Title: model card
    Link: https://ai.google.dev/gemma/docs/core/model_card
    Source snippet

    Google AI for Developers25 Feb 2025 — Human evaluation on prompts covering safety policies including child sexual abuse and exploitatio...

  16. Source: alan-turing-institute.github.io
    Title: Alan Turing Institute Model Cards
    Link: https://alan-turing-institute.github.io/tea-techniques/techniques/model-cards/
    Source snippet

    Model Cards - TEA TechniquesModel cards are standardised documentation frameworks that systematically document [machine learning]({{ 'machine-learning/' | relative_url }}) models th...

  17. Source: panaseer.com
    Link: https://panaseer.com/resources/blog/delivering-responsible-ai-with-model-cards
    Source snippet

    Delivering Responsible AI with Model CardsA model card should ideally contain sections covering: Basic information and solution overview...

  18. Source: edwinwenink.github.io
    Title: model card
    Link: https://edwinwenink.github.io/ai-ethics-tool-landscape/tools/model-card/
    Source snippet

    s for Model Reporting13 Jul 2021 — Model cards are short documents accompanying trained machine learning models that provide benchmarked...

  19. Source: storage.googleapis.com
    Title: Gemini 2.5 Computer Use
    Link: https://storage.googleapis.com/deepmind-media/Model-Cards/Gemini-2-5-Computer-Use-Model-Card.pdf
    Source snippet

    Model Card - Googleapis.com7 Oct 2025 — Model Cards are intended to provide essential information on Gemini models, including known limit...

Additional References

  1. Source: medium.com
    Link: https://medium.com/%40tahirbalarabe2/model-cards-explained-b14cd7c9439e
    Source snippet

    Model Cards Explained. Shoutout to Google | by TahirBy clearly stating intended [use cases]({{ 'use-cases/' | relative_url }}) and out-of-scope scenarios, Model Cards help no...

  2. Source: researchgate.net
    Link: https://www.researchgate.net/publication/328189552_Model_Cards_for_Model_Reporting
    Source snippet

    Model Cards for Model ReportingIn this paper, we propose a framework that we call model cards, to encourage such transparent model report...

  3. Source: aihealthcarecompliance.com
    Link: https://aihealthcarecompliance.com/resources/applicable-laws/nist-ai-rmf/

  4. Source: smartsuite.com
    Link: https://www.smartsuite.com/framework/nist-ai-risk-management-framework-ai-rmf-v1-0
    Source snippet

    NIST AI Risk Management Framework (AI RMF) v1.0The NIST AI RMF helps organizations identify, assess, and manage the risks associated with...

  5. Source: hyperproof.io
    Link: https://hyperproof.io/navigating-the-nist-ai-risk-management-framework/
    Source snippet

    Navigating the NIST AI Risk Management FrameworkUnderstand the NIST AI Risk Management Framework and learn how to govern AI risk, align s...

  6. Source: cdp.cooley.com
    Link: https://cdp.cooley.com/us-expands-artificial-intelligence-guidance-with-nist-ai-risk-management-framework/
    Source snippet

    cyber/data/privacy insightsUS Expands Artificial Intelligence Guidance with NIST AI Risk...Feb 8, 2023 — According to NIST, “Creating tr...

  7. Source: 2b-advice.com
    Title: model cards thats why model cards are so important for ki documentation
    Link: https://2b-advice.com/en/2025/09/16/model-cards-thats-why-model-cards-are-so-important-for-ki-documentation/
    Source snippet

    Why model cards are so important for AI documentation16 Sept 2025 — Model maps make AI comprehensible. They document the purpose, data or...

  8. Source: mbrenndoerfer.com
    Title: model cards documentation intended use limitations best practices
    Link: https://mbrenndoerfer.com/writing/model-cards-documentation-intended-use-limitations-best-practices
    Source snippet

    Model Cards: Documentation, Intended Use, and Limitations13 Mar 2026 — Learn how to write model cards that communicate intended use, trai...

  9. Source: practical-ai-act.eu
    Link: https://practical-ai-act.eu/latest/engineering-practice/model-cards/
    Source snippet

    Model cardsModel cards are a somewhat standardized form of documentation that provide a comprehensive overview of an AI model, including...

  10. Source: Tech Policy Press
    Title: why ai model cards are an urgent necessity for child safety
    Link: https://techpolicy.press/why-ai-model-cards-are-an-urgent-necessity-for-child-safety
    Source snippet

    Why AI 'Model Cards' Are an Urgent Necessity for Child...2 Apr 2026 — Based on our experience, organizations routinely give up on child...

Topic Tree

Follow this branch

Parent topic

Model Cards What AI labels should tell US before launch

Related pages 2