[Remote] Senior Manager, AI Performance & Operations

Note: The job is a remote job and is open to candidates in USA. Devoted Health is on a mission to improve the health and well-being of older Americans by providing integrated healthcare solutions. The Senior Manager, AI Performance & Operations will oversee the performance and continuous improvement of AI workflows, ensuring they meet operational outcomes and quality standards.

Responsibilities

• Post-Launch Accountability: Own the health of major voice and text AI workflows, ensuring business, quality, safety, and operational outcomes are met
• Portfolio Oversight: Maintain a current view of all major workflows, including status, maturity, failure modes, and next actions
• Performance & Dashboards: Monitor weekly indicators (containment, escalation, complaints) and oversee production dashboards to track behavior independently
• Diagnosis & Incident Response: Lead response for AI regressions; distinguish between model, prompt, or orchestration issues and drive post-mortems to closure
• Continuous Improvement & Optimization: Run structured cycles (hypothesis, testing, rollout/rollback) for "mini-launches" like model swaps, prompt changes, and tool updates. Recommend evidence-based accelerations, such as model migrations or deeper tool-orchestration changes
• Scalable Evaluation: Design eval systems using golden datasets, LLM-as-judge workflows, and human review queues for production and pre-production
• Expert Review: Guide new workflows during incubation to ensure they launch with proper instrumentation, eval coverage, and safety models
• Executive Communication: Translate technical changes into plain language for leadership, explaining what broke, why it matters, and who is fixing it

Skills

• Strong production experience with LLMs, RAG, voice AI, agentic orchestration, or equivalent ML systems
• Demonstrated experience designing or operating rubrics, scoring frameworks, regression tests, and production-sampling workflows
• A clear mental model for distinguishing failure modes, such as model reasoning, tool failure, retrieval issues, and workflow logic
• Comfort with active experimentation—including prompt/context tuning, retrieval redesign, and safe model migrations—beyond simple issue triage
• Excellent cross-functional judgment and the ability to communicate clearly in ambiguous, high-accountability environments
• Deep personal accountability with a drive to own results rather than just producing recommendations
• Experience in healthcare, regulated workflows, contact center operations, or high-stakes service quality/reliability functions

Benefits

• Employer sponsored health, dental and vision plan with low or no premium
• Generous paid time off
• $100 monthly mobile or internet stipend
• Stock options for all employees
• Bonus eligibility for all roles excluding Director and above; Commission eligibility for Sales roles
• Parental leave program
• 401K program
• Our total rewards package is for full time employees only. Intern and Contract positions are not eligible.

Company Overview

Company H1B Sponsorship