Reconciler with enabled "generationAwareEventProcessing" reconciles unchanged custom resource upon startup

[ivanchuchulski]

Bug Report

What did you do?

Hello,

We've encountered the behavior where we're using the default Reconciler settings, which has generationAwareEventProcessing turned on as well as a default maxReconciliationInterval set to 10 hours. Our CR implements the ObservedGenerationAwareStatus interface, so it has the observedGeneration property present in the status and it's handled by the framework.

We're seeing that upon Operator startup a reconciliation is triggered, when there is an already created custom resource in the cluster which has been reconciled already and has metadata.generation equal to status.observedGenertion.

The problem in this scenario is that the Operator is being updated and has updated logic, triggered by a pipeline automation script update. An update to the CR is going to follow in the next pipeline step. However, the triggered reconciliation captures the current CR in the cluster before the updated one can be applied. That in some cases leads to breaking dependent resources spec and subsequent failure (e.g. old incompatible images read from the CR set to an updated Deployment dependent resources spec encoded in the Operator code).

What did you expect to see?

No reconciliation triggered on the CR, which has it's generation and observed generation matching properties.

Environment

Kubernetes cluster type:

Gardener K8s 1.33.5

java-operator-sdk version is 4.4.4

Java version 17

Possible Solution

Not sure if the behavior could be caused by the maxReconciliationInterval being not set after the update or just informer caches being empty.

[csviri]

Hi @ivanchuchulski , this is intentional. It is the default behavior / best practice to reconcile resources on controller startup.

Note that you can implement this on your own check at the start of the reconiliation.

[ivanchuchulski]

Hello @csviri

Thanks for the quick response. Could you elaborate, what check do you mean we could implement?
Could you point out what is the reason for the proactive reconciliation technically - is it the empty caches or you've just added explicitly this logic?

Best Regards,
Ivan Chuchulski

[csviri]

Could you point out what is the reason for the proactive reconciliation technically - is it the empty caches or you've just added explicitly this logic?

In general all the controller frameworks do this, the reason comes from the theory regarding those controllers, usually during the reconciliation controllers tend to react on events, and check all the resources what they manage. So if a controller was down for some time on start it checks if everything still in place.

Could you elaborate, what check do you mean we could implement?

you could do (oversimplified) :

 @Override
  public UpdateControl<MyCustomResource> reconcile(
      MyCustomResource resource,
      Context<MyCustomResource> context) {
    if (resource.getMetadata().getGeneration() == resource.getStatus().getObservedGeneration()) {
         return;
    }    
    // omitted code
    return UpdateControl.noUpdate();
  }

if that fits your use case.

[ivanchuchulski]

Hello,
Thanks for the explanation. The check for the generations is exactly what I thought. Probably you meant to write the UpdateControl.noUpdate() when the if condition is true.

However, we think we can't simply rely on this check in the following cases:

• there is a periodic reconciliation triggered as you know is in place, with default time is 10 hours - if we just compare the generations, they'd match and we won't check reconcile any other dependent resource
• if some dependent resource gets changed/deleted, this would trigger a reconciliation, but this won't affect the generation of the CR - with the plain check, we'd again omit reconciling the dependents
• one more thing we have implemented on our end - as we have a Deployments as dependent resource, we have a check after we call reconcile on the dependent resource, to check if the Deployment is ready (reflecting it's Pods become ready) and if not, we put a specific state in the state and we reschedule a reconciliation with return UpdateControl.patchStatus(myCR).rescheduleAfter(CHECK_INTERVAL, TimeUnit.SECONDS);. That in turn updates the observedGeneration via the framework logic and in the next reconciliation, the check for equality would simply not be enough.

I was thinking including some additional metadata in the status, like nextReconcileTime or lastReconcileTime, as I've seen the latter for sure in some other projects, but I'm not sure how to implement that.

Best Regards,
Ivan Chuchulski

[csviri]

Yes, those also makes sense, of course that is the simplified view on generations.

Was thinking about this:

An update to the CR is going to follow in the next pipeline step. However, the triggered reconciliation captures the current CR in the cluster before the updated one can be applied.

This might be quite specific on your usecase, but you could add some annotation based conversion logic inside the controller implemnetation, or simply update the custom resources before running the operator.