Brief outages caused by Ray head node restarts were no longer acceptable. In this post, we dive into our migration from the RayCluster CRD to the RayService CRD, which enabled rolling updates, external GCS storage, and more. We share how we tackled challenges such as unpredictable deployments, slow Ray worker nodes start-up, and ensuring high availability with Dragonfly. If you want to understand how to make Ray workloads more resilient, predictable, and production-ready on Kubernetes, this post walks through our practical solutions and lessons learned.