Senior AI/ML Engineer

Kozak Agency is seeking a Senior AI/ML Engineer to improve production AI systems through evaluation, experimentation, and system design. The role focuses on diagnosing failures in agent workflows, designing evaluation metrics, and optimizing system performance for real users.

• Design evaluation strategies for LLM and agent workflows
• Create metrics and KPIs for AI system performance
• Build and maintain evaluation datasets
• Debug production AI failures systematically
• Improve agent orchestration and workflows
• Diagnose failures across agent pipelines
• Refine system prompts and agent interactions
• Contribute to recommendation systems, itinerary optimization, and LLM-based reasoning systems
• Collaborate with backend engineers on engineering integration

• 5+ years of experience in AI/ML (minimum 3 years if exceptionally strong)
• Strong AI/ML fundamentals and understanding of evaluation metrics (precision, recall, F1, etc.)
• Experience with prompt design and agent workflows
• Evaluation-driven mindset with ability to think in metrics and design experiments
• Experience with production LLM integrations
• Proficiency in Python, Go, or similar production languages
• Ability to read and modify production code
• Strong English proficiency (Upper-intermediate or Advanced)
• Nice to have: recommendation systems experience, optimization/constraint-based systems, computer vision, evaluation frameworks, Golang experience, startup experience

• Position type: Full-time
• Location: Remote (any location)
• Start date: ASAP