503 during blue-green deployment

[jan-machacek-kosik]

We get 503 errors during deployment as all members are temporarily in unhealthy state.
We tried setting ignore_new_hosts_until_first_hc to false but it didn't help.

We know pods are succesfully deployed because we have rigorous startup/liveness/readiness probes and Argo Rollout analysis before we switch blue-green versions.

When we disable active healthcheck everything works fine.

Any idea what could be the cause?

We are trying to achieve zero-downtime blue-green deployment with active healthchecks for our microservices using Envoy as a Gateway.

Cluseter configuration:

{
   "version_info":"b0c07ee734f23d9f889ff74e00fd0ac27133e95240cc3c103c69158a217c561a",
   "cluster":{
      "@type":"type.googleapis.com/envoy.config.cluster.v3.Cluster",
      "name":"httproute/jm2-eoc-k3w/int-https/rule/3",
      "type":"EDS",
      "eds_cluster_config":{
         "eds_config":{
            "ads":{
               
            },
            "resource_api_version":"V3"
         },
         "service_name":"httproute/jm2-eoc-k3w/int-https/rule/3"
      },
      "connect_timeout":"60s",
      "per_connection_buffer_limit_bytes":32768,
      "lb_policy":"LEAST_REQUEST",
      "health_checks":[
         {
            "timeout":"1s",
            "interval":"30s",
            "unhealthy_threshold":3,
            "healthy_threshold":1,
            "tcp_health_check":{
               
            },
            "event_logger":[
               {
                  "name":"stdout-log",
                  "typed_config":{
                     "@type":"type.googleapis.com/envoy.extensions.health_check.event_sinks.file.v3.HealthCheckEventFileSink",
                     "event_log_path":"/proc/self/fd/1"
                  }
               }
            ]
         }
      ],
      "circuit_breakers":{
         "thresholds":[
            {
               "max_retries":1024
            }
         ]
      },
      "dns_lookup_family":"V4_ONLY",
      "outlier_detection":{
         
      },
      "common_lb_config":{
         "healthy_panic_threshold":{
            
         },
         "locality_weighted_lb_config":{
            
         }
      },
      "ignore_health_on_host_removal":true
   },
   "last_updated":"2024-08-21T04:25:40.519Z"
}

[arkodg]

hey @jan-machacek-kosik this sounds like #4564