Senior AI Infrastructure Engineer

Overview

Job Description

Responsibilities

• - Design, build, and maintain the inference infrastructure that powers Sword Health's AI products, ensuring models are served with high throughput, low latency, and cost efficiency.
• - Own the end-to-end deployment pipeline for AI models - from real-time computer vision powering movement analysis to large language models driving conversational AI experiences.
• - Architect and scale Kubernetes clusters for GPU-accelerated workloads, including autoscaling strategies, resource scheduling, and multi-model serving.
• - Build and operate the infrastructure behind Sword Health's real-time AI agents, including WebRTC cluster provisioning and deploying speech-to-text and text-to-speech capabilities at low latency.
• - Drive inference scaling strategies - evaluate and implement techniques such as speculative decoding, continuous batching, and model parallelism to meet growing demand without proportionally increasing costs.
• - Develop and maintain Infrastructure as Code (Terraform) and GitOps workflows tailored to GPU-enabled, AI-specific environments.
• - Instrument and monitor AI inference systems, building observability around GPU utilization, model latency, throughput, and error rates to ensure reliability and performance.
• - Collaborate closely with ML Engineers, Data Scientists, and Product teams to translate model requirements into robust, production-ready infrastructure.
• - Evaluate emerging AI infrastructure tools, frameworks, and hardware to keep Sword Health at the cutting edge of inference performance and efficiency.
• - Mentor team members on AI infrastructure best practices, fostering knowledge sharing around GPU workloads, model serving patterns, and production ML systems.

Required Skills

• - 5+ years of experience in infrastructure engineering, with at least 2 years focused on AI/ML workloads in production environments.
• - Strong experience with Kubernetes for orchestrating GPU-accelerated workloads, including scheduling, resource management, and autoscaling for inference services.
• - Hands-on experience with model serving and inference optimization frameworks for both real-time computer vision and large language model workloads.
• - Solid understanding of LLM inference optimization techniques, including speculative decoding, batching strategies, quantization, and inference scaling patterns.
• - Experience provisioning and managing infrastructure for real-time AI systems, including WebRTC clusters and AI agent architectures.
• - Familiarity with real-time video/computer vision inference pipelines and the infrastructure challenges of processing continuous visual data streams at low latency.
• - Familiarity with speech-to-text and text-to-speech serving infrastructure and the challenges of running voice AI at low latency.
• - Experience with Infrastructure as Code (Terraform or similar) and GitOps methodologies for managing complex, GPU-enabled environments.
• - Working knowledge of GPU infrastructure - NVIDIA CUDA ecosystem, multi-GPU setups, and GPU monitoring/profiling.
• - Strong Linux systems fundamentals and networking knowledge, particularly for latency-sensitive, real-time workloads.
• - Fluent in English (written and oral).
• - A proactive, ownership-driven mindset - you see a bottleneck in an inference pipeline and you fix it before it becomes a problem.

Benefits

• - A stimulating, fast-paced environment with lots of room for creativity;
• - A bright future at a promising high-tech startup company;
• - Career development and growth, with a competitive salary;
• - The opportunity to work with a talented team and to add real value to an innovative solution with the potential to change the future of healthcare;
• - A flexible environment where you can control your hours (remotely) with unlimited vacation;
• - Access to our health and well-being program (digital therapist sessions);
• - Remote or Hybrid work policy.