QA Engineer

Job Description:

Responsibilities

• Build test plans for AI products across large combinatorial matrices: input variants, user roles, content states, deployment surfaces.
• Maintain golden-question test suites that catch regressions when models, prompts, retrieval, or content change.
• Design coverage for the things AI products specifically get wrong: hallucinations, citation drift, prompt injection, context bleed, guardrail bypass, role escalation, stale or unapproved content surfacing where it shouldn't.
• Translate ambiguous quality requirements into concrete, repeatable test cases—especially where audit trails, access controls, and content provenance matter.
• Executive & Design
• Run structured test passes across web UIs and APIs; investigate failures deeply enough to give engineers a useful starting point, not just a screenshot.
• Validate evidence citations point to the right sources and that "pending review" or restricted content stays excluded from AI responses where the rules require it.
• Verify role-based access at the UI and API layers—users see what they should and nothing more.
• Verify data integrity using read-only SQL queries; cross-check what the UI shows against what the database holds.
• Exercise APIs directly via Postman, Insomnia, or curl; read OpenAPI specs and form requests without help.
• File defects that are clear, reproducible, and ranked—structured enough that an engineer can act without a meeting.
• Sustain Coverage
• Maintain regression coverage as products evolve: prompts change, retrieval indexes update, models get swapped, content gets added.
• Partner with engineers on evaluation rubrics, golden datasets, and acceptance criteria for AI behaviour—then verify the criteria actually hold in production.
• Stay current on AI/LLM evaluation tooling and apply what's useful: golden datasets, regression rubrics, hallucination scoring, structured eval frameworks.
• Maintain test documentation that survives team turnover: clear, structured, and findable.

.Qualification & Experience    

• 5+ years manual and automation QA experience in product environments, production-grade.
• Hands-on experience testing AI / LLM-based products: output validation, hallucination detection, edge-case coverage, prompt injection awareness.
• Strong test-planning skills for large combinatorial test matrices.
• Hands-on with test management tooling: TestRail, Zephyr Scale, Xray, or equivalent.
• API testing fluency: Postman or Insomnia; comfortable reading OpenAPI/Swagger specs and exercising endpoints directly.
• Browser-based debugging skills: Chrome DevTools (Network, Console, Application), cross-browser verification.
• Bug tracking and workflow tooling: Jira, Linear, or equivalent.
• SQL basics: read-only queries to verify data state in PostgreSQL or equivalent.
• Documentation discipline: clear, structured test plans and defect reports in Confluence, Notion, or equivalent.
• Comfort with enterprise content workflows, audit-trail verification, and role-based access checks.
• A clear-eyed view of where current AI tooling helps and where it falls short.
• Curiosity, persistence, and a willingness to dig past the surface symptom into the underlying cause.
• Bonus Skills / Experience
• Understanding of MLOps, model serving, scaling and monitoring workflows (e.g., BentoML, MLflow, Vertex AI, AWS Sagemaker)
• Exposure to evaluation tools (Ragas, Promptfoo, DeepEval, LangSmith)—read-only or interpretive use is fine.
• Automation experience with Playwright, Cypress, or equivalent—even if the role is primarily manual.
• Browser Stack or similar cross-browser/device platforms.
• GDPR-style data handling verification experience.
• Pharma, healthcare, or other regulated-industry exposure.
• Familiarity with EU AI Act or similar AI governance and compliance frameworks