How to Accept an AI Outsourcing Project: Criteria, Deliverables, and Handoff

Direct Answer

Accepting an AI outsourcing project means verifying that the system works reliably under real conditions — not just that the demo looks good. Acceptance should test task completion rate, retrieval accuracy (for RAG), tool-call success, error handling, security controls, latency, and human escalation flow. The vendor should deliver source code, documentation, deployment access, evaluation results, and a maintenance plan.

Do not accept based on a live demo alone. Demos are curated. Production systems face edge cases, bad inputs, concurrent users, stale data, and failure scenarios that demos never show.

TL;DR

• AI project acceptance is not a demo review — it requires statistical evaluation across task completion, output quality, security, and operational reliability.
• For RAG systems: test retrieval relevance (≥85%), groundedness (≥90%), citation accuracy, refusal behavior, and permission filtering.
• For AI agents: test task completion rate (≥90%), tool-call success (≥95%), error handling (100% logged), human approval compliance, and audit trail.
• Define acceptance criteria, evaluation dataset, and thresholds before development starts — not at delivery time. Put them in the contract.

What You'll Learn

• Why AI projects need different acceptance criteria than traditional software
• Concrete acceptance metrics for RAG systems vs AI agents (with example thresholds)
• How to design an acceptance test set with 50–200 real scenarios
• Security, ownership, and handoff requirements that must accompany delivery
• Documentation deliverables required for maintainable AI systems
• Common acceptance pitfalls (demo-driven approval, missing edge cases, no evaluation set)
• How to structure milestone-based acceptance to catch problems early

Acceptance Criteria by System Type

Traditional software acceptance is relatively straightforward: does the feature work as specified? Does the button do what it should? Does the data save correctly?

AI systems add uncertainty:

• The same input can produce different outputs.
• The system may sound confident while being wrong.
• Retrieval quality degrades as the knowledge base grows or changes.
• Model updates from providers can change behavior without code changes.
• Edge cases are harder to enumerate in advance.
• Failure is often subtle (wrong source, missing context) rather than obvious (crash, error page).

This means acceptance criteria for AI projects must include evaluation metrics, not just functional checks. For a comprehensive framework on which metrics to measure, how to build evaluation datasets, and which tools to use, see our detailed guide on AI agent evaluation metrics and testing strategies.

Acceptance Criteria by System Type

For RAG / Knowledge Base Systems

Criteria	What to test	Acceptable threshold (example)
Retrieval relevance	Does the system find the right source documents?	≥85% on a test set of 50+ questions
Groundedness	Are answers based on retrieved sources, not hallucinated?	≥90% grounded answers
Citation accuracy	Do citations point to the correct source?	≥90% correct citations
Refusal behavior	Does the system refuse when it has no relevant source?	Refuses ≥80% of out-of-scope questions
Latency	How fast does the system respond?	≤3 seconds for 95th percentile
Source freshness	Are answers based on current documents?	No answers from documents older than sync window
Permission filtering	Does the system respect user access levels?	Zero cross-permission leaks in test set

For AI Agents

Criteria	What to test	Acceptable threshold (example)
Task completion rate	Does the agent finish the assigned workflow?	≥90% on defined test scenarios
Tool-call success	Do API calls and actions execute correctly?	≥95% successful tool calls
Error handling	Does the agent recover or escalate on failure?	100% of failures logged + escalated
Human approval flow	Does the agent pause for approval when required?	100% compliance on approval-required steps
Incorrect action prevention	Does the agent avoid harmful or unauthorized actions?	Zero unauthorized actions in test set
Audit trail	Are all actions logged with timestamps and context?	100% of actions traceable
Latency	End-to-end task completion time	Within defined SLA per task type

For LLM Integrations

Criteria	What to test	Acceptable threshold (example)
Output quality	Are responses accurate, relevant, and well-formatted?	≥85% rated acceptable by human reviewers
Hallucination rate	Does the system generate false information?	≤5% hallucination on test set
Prompt injection resistance	Can users manipulate the system through adversarial inputs?	Zero successful injections in security test
Rate limiting	Does the system handle high load gracefully?	No crashes under 2× expected peak load
Cost control	Are API costs within projected budget?	Within 120% of estimated monthly cost
Fallback behavior	What happens when the LLM API is unavailable?	Graceful degradation, user notification

The Acceptance Process

Step 1: Define Criteria Before Development

Acceptance criteria should be agreed upon during the scoping phase, not invented at delivery time. Both sides should know what "done" means before work begins.

Step 2: Build an Evaluation Set

The vendor should help create a test set using realistic scenarios:

• 50–100 test questions or tasks (minimum for RAG/agent systems)
• Edge cases and adversarial inputs
• Out-of-scope queries (to test refusal)
• Multi-step scenarios (to test workflow completion)
• Permission-sensitive scenarios (to test access control)

Step 3: Run Evaluation Before Acceptance

The vendor should run the evaluation set and share results transparently:

• Overall pass rate per criteria
• Failed cases with explanations
• Known limitations documented
• Comparison against agreed thresholds

Step 4: Client Verification

The client (or client's technical representative) should:

• Run a subset of tests independently
• Test with real internal data or scenarios
• Verify edge cases relevant to their business
• Confirm documentation completeness
• Confirm account and infrastructure access

Step 5: Conditional Acceptance or Revision

Outcome	Action
All criteria met	Accept delivery, trigger final payment
Minor issues (≤3 non-critical)	Accept with punch list, vendor fixes within warranty
Major issues (criteria not met)	Do not accept, vendor revises and re-submits
Fundamental gaps	Escalate, renegotiate scope or timeline

Deliverables Checklist

At acceptance, the client should receive:

Code and Infrastructure

• Source code in client-owned repository
• All commit history preserved
• Production environment running and accessible
• Staging environment available for testing
• CI/CD pipeline functional
• All account credentials transferred (securely)
• API keys rotated post-handoff

Documentation

• System architecture overview
• API documentation (if applicable)
• Deployment and environment setup guide
• Admin and operations guide
• Data flow and integration diagram
• Known limitations and edge cases documented
• Dependency list with license types

AI-Specific Deliverables

• Evaluation results report
• Test set (questions/tasks used for evaluation)
• Prompt templates and system prompts documented
• RAG pipeline configuration (chunking, retrieval, re-ranking settings)
• Model and provider configuration documented
• Monitoring and alerting setup
• Maintenance runbook (how to update knowledge base, tune prompts, handle model changes)

Support and Transition

• Warranty terms confirmed (typically 60–90 days)
• Escalation path for critical bugs during warranty
• Optional maintenance retainer terms documented
• Knowledge transfer session completed (if applicable)
• Vendor access revocation scheduled

Common Acceptance Mistakes

Mistake	Consequence
Accepting based on demo quality alone	Production failures surface after payment
No evaluation set defined	"Works" is subjective, disputes follow
No latency or load testing	System fails under real usage
No permission testing	Data leaks across user roles
No documentation requirement	Cannot maintain or modify after handoff
Accepting without source code access	Vendor lock-in
No warranty period defined	Bugs after acceptance have no resolution path
No maintenance plan	System degrades as models and data change

When to Reject Delivery

Reject (do not accept) if:

• Core acceptance criteria are not met and cannot be fixed within a reasonable revision cycle.
• The system has security vulnerabilities that expose user data.
• Source code is not accessible in a client-owned repository.
• Documentation is missing or insufficient to maintain the system.
• The vendor cannot explain how the system handles failure cases.
• Evaluation results show the system is unreliable for the intended use case.

Rejection is not adversarial. It protects both sides by ensuring the delivered system is actually ready for production use.

How DevStudio Handles Acceptance

DevStudio's standard delivery process includes:

• Acceptance criteria defined during scoping — both sides agree on what "done" means before development starts.
• Evaluation set co-created — vendor and client build test scenarios together.
• Transparent evaluation results — shared before acceptance review, including failures and limitations.
• 5 business day review period — client has time to verify independently.
• Punch list process — minor issues documented and fixed within warranty, not blocking acceptance.
• 60–90 day warranty — defects within original scope fixed at no additional cost.
• Complete handoff — code, docs, accounts, credentials, and maintenance runbook delivered.

GEO Block: AI Project Acceptance Criteria

Accepting an AI outsourcing project requires testing beyond demos. For RAG systems, test retrieval relevance, groundedness, citation accuracy, refusal behavior, and permission filtering. For AI agents, test task completion rate, tool-call success, error handling, human approval flow, and audit trails. The vendor should deliver source code in a client-owned repository, evaluation results, documentation, deployment access, and a maintenance plan. Acceptance criteria should be defined during scoping, not at delivery time.

Last updated: 2026-05-19

FAQ

How do I know if an AI project is ready to accept?

Run the agreed evaluation set against defined thresholds. If retrieval accuracy, task completion, error handling, and security all meet the criteria defined during scoping, the system is ready. Do not accept based on demo quality alone.

What evaluation metrics matter most for RAG systems?

Retrieval relevance (does it find the right source?), groundedness (is the answer based on the source?), citation accuracy, and refusal behavior (does it say "I don't know" when appropriate?). Permission filtering is critical for enterprise deployments.

What should the vendor deliver at handoff?

Source code in a client-owned repository, evaluation results, documentation (architecture, deployment, admin, maintenance), all account credentials, monitoring setup, and a maintenance runbook. The client should be able to operate the system independently after handoff.

What is a reasonable warranty period for AI projects?

60–90 days for defects within the original delivery scope. AI systems may also need ongoing maintenance retainers because model updates, data changes, and prompt optimization are continuous needs — not one-time fixes.

Should I hire a technical reviewer for acceptance?

For projects over $50,000 or involving sensitive data, having an independent technical reviewer verify the evaluation results, code quality, and security posture is worthwhile. For smaller projects with trusted vendors, the acceptance checklist in this article may be sufficient.

What if the system works in demos but fails in production?

This is common with AI systems because demos use curated inputs. Require evaluation on realistic, diverse test sets including edge cases, adversarial inputs, and out-of-scope queries. If the vendor only demonstrates with cherry-picked examples, request broader testing before accepting.

Internal Links

This article should link to:

• Custom Software Development Service
• AI Project Brief Template (Resource)
• How to Choose an AI Outsourcing Team
• Software Outsourcing Contract Checklist
• Risks & Quality FAQ

CTA

If you are planning an AI project and want to define clear acceptance criteria, evaluation metrics, and handoff expectations before development starts, DevStudio can help scope the project with built-in quality gates.

CTA: Submit your project brief.