actuator/health/grpc endpoint continue reporting grpc services as SERVING even if grpc server has been stopped

[chetan5375]

Steps to Reproduce

• Create a spring boot app with lognet
• Create any simple grpcService
• Create a restController having a reference to GrpcServerRunner
• Expose a rest end point in this restController to stop grpc server
• Deploy and run app in your local kubernetes cluster (You can run as spring boot app as well, doesn't have to be deployed to cluster)
• Tomcat and grpc server will be started. RestController and grpc services will get registered
• grpc service will be discovered and its status will be set to SERVING by default
• Now once everything is up and looking good hit the rest endpoint to shutdown grpcServer
• Now tomcat server is up and grpcServer is down
• Send a request to /actuator/health/grpc end point and it will continue to report the service as SERVING

Analysis

• Lognet's DefaultHealthStatusService seems to be having a bug
• It extends ManagedHealthStatusService whose onShutdown() method's documentation clearly indicates that "Implementation is advised to set status of all services as ServingStatus.NOT_SERVING"
• But DefaultHealthStatusService's onShutdown() implementation simply delegates the call to healthStatusManager.enterTerminalState(); without changing the statuses of services to NOT_SERVING in its locally maintained map

References

• https://github.com/LogNet/grpc-spring-boot-starter/blob/master/grpc-spring-boot-starter/src/main/java/org/lognet/springboot/grpc/autoconfigure/actuate/GRpcActuateAutoConfiguration.java
• https://github.com/LogNet/grpc-spring-boot-starter/blob/master/grpc-spring-boot-starter/src/main/java/org/lognet/springboot/grpc/health/ManagedHealthStatusService.java
• https://github.com/LogNet/grpc-spring-boot-starter/blob/master/grpc-spring-boot-starter/src/main/java/org/lognet/springboot/grpc/health/DefaultHealthStatusService.java

Fix

Any local state maintained by DefaultHealthStatusService should be cleared out and marked as NOT_SERVING upon onShutdown() method call.

[chetan5375]

@jvmlet did you got a chance to look at this issue?