SharadhaAI / ← All Courses
WhatsApp
Track 03 • Agentic Evaluation Course • 6 Sundays • Starts 15 Nov 🎓 Certificate Provided 💼 Reimbursable via Corporate L&D Limited Offer: ₹9,999

Agentic Evaluations for QA Engineers & SDETs
Become an AI Test Architect

Move from testing software to testing AI. In 6 Sundays, learn to evaluate LLM apps and AI agents: eval datasets, LLM-as-a-Judge, agent and RAG evals, red-teaming, and quality gates in CI/CD, built on the test skills you already have.

Course Overview

Every team shipping an LLM feature or AI agent now needs someone who can answer one question: is it good enough to release? That person needs test design thinking, automation discipline and a regression mindset, which QA engineers and SDETs already have. This course adds the other half: how LLMs and agents fail, how to measure non-deterministic output, and how to turn that into release gates.

🧪 Eval Design

Turn vague quality goals into rubrics, golden datasets and edge-case suites you can run on every change.

⚖️ LLM-as-a-Judge

Build judge pipelines, calibrate them against human labels, and know when a code check is the better grader.

🤖 Agent & RAG Evals

Evaluate tool-call correctness, agent trajectories, groundedness and hallucination using real traces.

🛡️ Red-Teaming & CI Gates

Test for prompt injection and unsafe actions, then run evals as regression gates in CI/CD.

Your Future Role

You finish this track ready for roles that test and govern AI systems, not just software.

AI Test Architect

Owns the quality strategy for AI products: what to measure, which graders to trust, and when a release is blocked.

AI Quality Engineer

Builds and maintains eval suites for LLM features and agents, and runs them in CI on every prompt or model change.

Agentic Evals Engineer

Evaluates multi-step agents: tool use, trajectories, recovery from errors, safety and cost.

🎯 You'll be able to demonstrate
  • ✓ Design an eval dataset and rubric for a real LLM or agent use case
  • ✓ Build and calibrate an LLM-as-a-Judge, and explain where it can be wrong
  • ✓ Evaluate agent trajectories and RAG groundedness from traces
  • ✓ Red-team an AI system for prompt injection and unsafe tool actions
  • ✓ Explain how eval results support AI governance and compliance reviews
  • ✓ Ship an eval suite as a CI quality gate, as a portfolio capstone

Syllabus Breakdown

6 Sessions • Sundays • 2.5 Hrs/Session
Session 1 • Sun 15 Nov

How LLM Apps & Agents Fail

Non-determinism, hallucination, tool misuse and context limits, seen through a tester's eyes. Moving from assert-equals to properties, pass rates and thresholds.

Tools: Claude / OpenAI API • You build: a failure-mode catalogue for a real AI app

Session 2 • Sun 22 Nov

Your First Eval Harness

Golden datasets, rubrics and code-based graders, run as tests with pass-rate thresholds.

Tools: pytest, DeepEval • You build: an eval suite for an LLM feature

Session 3 • Sun 29 Nov

LLM-as-a-Judge & RAG Evaluation

Writing judge rubrics, calibrating judges against human labels, judge bias, and measuring groundedness, relevance and hallucination in RAG systems.

Tools: DeepEval, Ragas • You build: a calibrated judge and a RAG eval

Session 4 • Sun 6 Dec

Agent & Trajectory Evaluation

Tool selection and argument correctness, loops and recovery, multi-turn testing, cost and latency, and debugging agents with traces.

Tools: Arize Phoenix or Langfuse • You build: trajectory evals for a real agent

Session 5 • Sun 13 Dec

Red-Teaming, Guardrails & AI Governance Intro

Prompt injection, jailbreaks and unsafe tool actions (OWASP Top 10 for LLMs), guardrail testing, and an introduction to AI governance: NIST AI RMF, the EU AI Act and ISO/IEC 42001, and the test evidence they expect from QA teams.

Tools: promptfoo red-team, NeMo Guardrails • You build: a red-team report

Session 6 • Sun 20 Dec

Evals in CI & Production + Capstone

Evals as regression gates on every prompt or model change, monitoring live traces in production, and presenting your capstone.

Tools: GitHub Actions, promptfoo • You build: an eval suite running as a CI quality gate

Real Tools, Real Evaluation Workflows

You work hands-on with the open-source tools AI teams use to evaluate LLM apps and agents, and learn what to measure, how to measure it reliably and when to block a release, so your skills carry over to any evaluation platform.

Eval Harness

Python, pytest, DeepEval

RAG Metrics

Ragas

Tracing

Arize Phoenix or Langfuse

Red-Teaming & Guardrails

promptfoo, NeMo Guardrails

CI Quality Gates

GitHub Actions

AI Governance Intro

NIST AI RMF, EU AI Act, ISO/IEC 42001

Hosted By

RR

Rama Rajeswari

AI Practitioner and Entrepreneur. With 20+ years of industry experience and an IIT Guwahati AI certification, Rama specializes in enterprise Agentic systems, multi-agent workflows, and LLM evaluation pipelines. Spanning startups to Fortune-scale MNCs, her focus is on practical, battle-tested tool application over textbook theory.

Previous Enterprise Experience:
Chargebee Cisco Oracle Circles

Who Is This Course For?

QA & Automation Engineers

Manual or automation testers ready to move into AI testing roles.

SDETs

Framework builders who want to own evaluation and quality gates for AI features and agents.

QA Leads & Managers

Leads who need to define a test strategy for AI products their teams are shipping.

Frequently Asked Questions

What is the schedule? ↓
15 hours of live training: 6 sessions of 2.5 hours, one every Sunday, from 15 November to 20 December. Each week has a short hands-on lab, plus optional mid-week office hours for doubts and project reviews.
Do I need AI or machine learning experience? ↓
No. Session 1 covers what testers need to know about LLMs and agents. You should be comfortable with basic Python or another automation language and know how a test framework works.
How is this different from the AI Engineer Track? ↓
The AI Engineer Track teaches you to build agents, RAG and multi-agent systems. This track teaches you to test and evaluate them. It is shorter and designed around the skills QA engineers and SDETs already have.
How is this different from the Agentic QA track? ↓
The Agentic QA track uses AI to test software. This track teaches you to test AI itself: evaluating LLM apps and agents. If your team is starting to ship AI features, this is the track for you.
Which tools will I use? ↓
It is both. Every session is hands-on with named tools (pytest, DeepEval, Ragas, promptfoo, Phoenix or Langfuse, GitHub Actions), and you learn the evaluation methods behind them so they carry over to any platform your employer uses.
What will I have at the end? ↓
A capstone eval suite for a real AI agent in your own GitHub repository, running as a CI quality gate, plus a verified certificate. Both are built to show in interviews.