Certified AI Operator (CAIO)

Section 1

Assessment Philosophy

The AI Trade School CAIO certification program employs a competency-based assessment model. Credentials are awarded on the basis of demonstrated mastery — not seat hours, content consumption, or time in program. Every assessment instrument answers a single question: can this individual perform this competency to a professional standard under authentic conditions?

Assessment protects three things simultaneously: the value of the credential, employer confidence in certified operators, and the institutional integrity of AI Trade School.

Assessment Principles

Applied

Assessment requires application to authentic tasks. Rote memorization of definitions, terminology, or procedures is insufficient to demonstrate competency.

Observable

Competency is demonstrated through work products, performances, and demonstrations. Self-report is not accepted as evidence of competency.

Measurable

All assessments use published criteria, rubrics, and scoring models with defined proficiency levels. No unconstrained subjectivity in evaluation.

Repeatable

Assessment instruments are standardized so that different evaluators, evaluating the same work, reach the same conclusion. Inter-rater reliability is a design priority.

Aligned

Every assessment is aligned to specific competency domains and credential requirements. No assessment exists without a direct connection to certification standards.

Transparent

All rubrics, criteria, and standards are published and available to candidates in advance. No hidden requirements or undisclosed evaluation criteria.

Ethical

Assessment is free from bias, conflict of interest, and arbitrary decision-making. Candidates have the right to fair evaluation and appeals processes.

What Assessment Is Not

The following do not constitute evidence of competency and are not used in certification decisions:

• Attendance or login records
• Content consumption metrics (videos watched, pages viewed)
• Peer ratings or endorsements
• Self-assessment or self-reported skill levels
• Forum participation (unless rubric-evaluated)
• Time spent on platform or in coursework

Certification decisions are made exclusively on the basis of evaluated performance against published standards.

Section 2

Three-Level Assessment Architecture

The CAIO assessment system operates across three sequential levels. Candidates cannot advance to a higher level without completing the prior level in full.

Assessment Flow

1. 1Complete all Tier 1 courses; pass all knowledge checks and activities.
2. 2Complete all Tier 2 courses; achieve Proficient or above on all rubric-evaluated artifacts.
3. 3Portfolio Readiness Review and Domain-Level Validation across all 8 competency domains.
4. 4Enroll as CAIO candidate upon successful domain validation.
5. 5Submit capstone project for evaluation.
6. 6Sit written examination under proctored conditions.
7. 7Complete oral defense before evaluation panel.
8. 8Institution reviews all assessment results and makes credential determination.

Section 3

Tier 1 Assessment Standards

Tier 1 courses serve a gatekeeping function. They verify foundational knowledge and basic competency required to enter applied coursework. Tier 1 performance does not contribute to CAIO certification scoring.

Grading Model

All Tier 1 courses are graded Pass / Not Yet Passed. There is no partial credit. Knowledge assessments permit up to 3 retakes with a 48-hour waiting period between attempts. Activity-based assessments permit 1 revision. Upon exhaustion of all attempts, candidates must wait 60 days and re-enroll in the course.

Section 4

Tier 2 Assessment Standards

Tier 2 courses assess applied competence through portfolio artifacts evaluated on a standardized 4-level rubric. These artifacts form the foundation of the candidate's professional portfolio and contribute directly to domain-level validation.

Unified Rubric Scale

Candidates are permitted 2 revisions per course. Revisions must be substantive — cosmetic changes without addressing evaluator feedback do not qualify. Failure to achieve Proficient after 2 revisions requires a 30-day waiting period followed by retaking the full course.

Section 5

Course-Level Rubrics

The following rubrics define the evaluation criteria for each Tier 2 course. Each artifact is evaluated across 5 criteria on the 4-level proficiency scale. All criteria must achieve Level 3 (Proficient) or above for course completion.

Section 6

Domain Validation

Domain validation bridges course-level and certification-level assessment. It evaluates whether a candidate has achieved competency across each of the 8 CAIO competency domains using evidence drawn from multiple courses and artifacts.

Domain Scoring Model

Portfolio Readiness Review

1. 1Verify all required artifacts are present and scored Proficient or above on all rubric criteria.
2. 2Evaluate aggregate evidence for each of the 8 competency domains across all submitted artifacts.
3. 3Assign M, C, R, or I to each domain based on the totality of evidence.
4. 4Document rationale for each domain score with specific evidence citations.
5. 5Communicate results to the candidate in writing with actionable feedback.

Remediation

Candidates scoring R (Needs Remediation) on any domain receive specific feedback identifying competency gaps and targeted remediation activities. Upon completion, the domain is reevaluated. Candidates scoring I (Insufficient) must retake the relevant course(s) in full.

Section 7

Capstone Assessment

The capstone project is the centerpiece of the CAIO certification assessment. It requires candidates to integrate all 8 competency domains into a single comprehensive project demonstrating professional-level AI operations competency. The capstone carries a weight of 40% in the final certification determination.

Required Deliverables

1. Organizational Context Analysis
2. AI System Design & Architecture
3. Implementation Documentation
4. Standard Operating Procedures (minimum 2)
5. Ethical & Risk Assessment
6. Impact Evaluation
7. Stakeholder Communication Package

A missing deliverable results in automatic return of the capstone without evaluation. All 7 deliverables must be present before the capstone enters the evaluation queue.

Capstone projects are evaluated by a minimum of 2 independent evaluators using the rubric defined in Appendix A. All 9 rubric dimensions must achieve Level 3 (Proficient) or above. One revision is permitted. Failure to achieve Proficient after revision requires a 90-day waiting period followed by submission of a new capstone project.

Section 8

Written Examination

Specifications: 100 questions total, 3-hour time limit, 80% passing threshold, minimum 70% per domain, remotely proctored, multiple exam versions drawn from a secured item bank. The written examination carries a weight of 30% in the final certification determination.

Domain Weight Distribution

One retake is permitted with a 30-day waiting period. The retake uses a different exam version. A second failure requires a 6-month waiting period followed by re-enrollment in the certification program.

Section 9

Oral Defense

The oral defense panel consists of a minimum of 2 qualified reviewers. Both panelists must independently rate the candidate as passing. In the event of a split decision, a third panelist is brought in to make the determination. The defense is conducted via live video, recorded with candidate consent. The oral defense carries a weight of 30% in the final certification determination.

Fail Conditions

• Candidate cannot explain their own system design or implementation decisions
• Candidate misrepresents AI capabilities or limitations
• Candidate demonstrates ethical blind spots on direct questioning
• Candidate cannot engage constructively with critical challenge or feedback
• Any dimension on the defense rubric scores below Level 3

One reattempt is permitted with a 30-day waiting period. The candidate must submit a written response addressing evaluator feedback before the reattempt. A second failure requires a 6-month waiting period and re-enrollment in the certification program.

Section 10

Unified Proficiency Scale

The following 4-point proficiency scale is applied identically across all rubric-evaluated assessments: Tier 2 course rubrics, the Capstone Rubric (Appendix A), and the Oral Defense Rubric (Appendix B). All evaluators are calibrated to this scale.

Section 11

Pass / Fail Logic

Certification Awarded When ALL Conditions Are Met

1. All 5 Tier 1 courses passed
2. All 5 Tier 2 courses passed with all rubric criteria at Level 3 or above
3. All 8 competency domains scored at Competent (C) or Mastery (M)
4. Capstone project scored Level 3 or above on all 9 dimensions
5. Written examination score of 80% or above overall, with no domain below 70%
6. Oral defense scored Level 3 or above on all 6 dimensions by both panelists

Certification Withheld If Any of the Following Apply

• Any Tier 1 or Tier 2 course not passed
• Any Tier 2 rubric criterion below Level 3
• Any competency domain scored R or I (unremediated)
• Any capstone dimension below Level 3
• Written exam score below 80% overall or any domain below 70%
• Any oral defense dimension below Level 3

Weight Summary

Section 12

Remediation & Reassessment

All revisions must be substantive. Cosmetic changes without addressing evaluator feedback do not qualify as a substantive revision. Candidates who have exhausted standard remediation opportunities may petition for additional attempts; petitions are reviewed on a case-by-case basis and are not guaranteed.

Section 13

Evaluator Standards

Qualifications

• Minimum 3 years of relevant professional experience
• Demonstrated domain competence in assigned evaluation areas
• Completed institutional Evaluator Training program
• Familiar with all rubrics, scoring models, and assessment frameworks
• No conflicts of interest with candidates under evaluation

Responsibilities

Evaluators are responsible for applying rubrics consistently and without bias, documenting all scoring decisions with evidence-based rationale, providing actionable feedback that guides candidate improvement, participating in regular calibration sessions, reporting any concerns regarding candidate integrity or assessment instrument validity, and maintaining strict confidentiality of all candidate materials and results.

Calibration

All evaluators undergo an onboarding training program that includes scoring benchmark artifacts at each proficiency level. Quarterly calibration sessions ensure ongoing alignment. Annual inter-rater reliability analysis targets a Cohen's kappa of 0.80 or above. Evaluators whose scores consistently deviate from calibrated standards receive additional training or are removed from the evaluation panel.

Section 14

Assessment Integrity

Academic Integrity

All submitted work must be the candidate's own. AI tools may be used as instructed within the curriculum, but submissions must demonstrate the candidate's personal understanding, judgment, and professional competence. The following constitute violations:

• Submitting AI-generated work without meaningful personal contribution
• Plagiarism of any form, including uncredited use of others' work
• Fabrication or falsification of data, results, or evidence
• Unauthorized collaboration on individually assessed work
• Sharing exam content, questions, or answers with other candidates

Exam Security

Written examinations are remotely proctored with identity verification, screen monitoring, and session recording. Multiple exam versions are drawn from a secured item bank to prevent content leakage. Any violation of exam security protocols results in immediate termination of the exam session and initiation of disciplinary proceedings.

Capstone Originality

Capstone projects are reviewed for evidence of genuine intellectual engagement. Submissions that appear AI-generated without meaningful candidate contribution are flagged for additional review and may be rejected. Consequences for integrity violations range from assignment failure to program dismissal and credential revocation, depending on severity.

Section 15

Records & Audit Trail

The following records are maintained for every candidate who enters the CAIO certification program: portfolio artifacts, capstone projects, completed rubric evaluation sheets, domain scoring records, examination results, oral defense recordings, remediation and appeals records, and final certification determinations.

Retention Schedule

All records are encrypted and access-controlled. Candidates may access their own records at any time. No third-party access to candidate records is permitted without explicit written consent from the candidate.

Section 16

Governance & Revision

• Annual review of all assessment instruments, rubrics, and scoring models
• Comprehensive review every 2 years incorporating external stakeholder input and industry alignment analysis
• Triggered review in response to significant developments in AI technology, regulation, or professional practice
• Version control: candidates are assessed under the version of the assessment system in effect at the time of their enrollment; all prior versions are archived for a minimum of 7 years

Continuous Improvement

The assessment system is continuously improved through analysis of: evaluator feedback and calibration data, candidate performance patterns, pass/fail rate analysis by assessment component, item-level analysis for written examinations, rubric reliability metrics, and stakeholder feedback from employers and industry partners.

Appendix A

Capstone Project Rubric

All 9 dimensions are evaluated on the 4-point proficiency scale. The candidate must achieve Level 3 (Proficient) or above on ALL 9 dimensions. No compensatory scoring.

Appendix B

Oral Defense Rubric

All 6 dimensions are evaluated on the 4-point proficiency scale. Both panelists must independently rate the candidate at Level 3 (Proficient) or above on all dimensions.

Appendix C

Domain Validation Checklist

Evaluators use the following checklist during the Portfolio Readiness Review. Assign M (Mastery), C (Competent), R (Needs Remediation), or I (Insufficient) for each domain based on aggregate evidence.

Appendix D

Certification Decision Flowchart

The following decision flowchart summarizes the sequential gates a candidate must pass to receive the CAIO credential.