ArgoCD Notification Tips

Introduction

ArgoCD is a great tool for syncing resources to Kubernetes from Git. With these deployments, we want to receive notifications on how this process goes. We learnt a few lessons about using ArgoCD Notifications to accomplish this and wanted to share them with the community as it could help others.

What is ArgoCD Notifications and how does it work?

ArgoCD Notifications is a tool written by the ArgoCD folks to make notifying various channels about the status of syncs. There are other tools that can do the same thing but I liked the tailored approach of something specific for ArgoCD.

The ArgoCD Notification app is essentialy a controller that looks out for ArgoCD Application resources(using informers) and uses the object to craft a notification and deliver it to configured receivers like Slack or custom webhooks. This object includes the status field so it can see what the status is of the resource.

How Status Works

There are two important statuses, the status of the sync and the health of the application. The status of the sync is whether the resources in Kubernetes match what’s in Git. The health of the application is defined by the health of the resources as reported by Kubernetes. For example, does the deployment have the specified number of pods ready?

Accurate Kubernetes Health Status

This one caught me out. By default, Kubernetes is configured to class any process that has been ready for 0 seconds to be Ready. If you have probes, the Ready timer starts when the probes are successful. Without probes, this timer starts when the process starts.

If you have no probes, the deployment status can proceed to the successful state quickly and ArgoCD then sets the application resource sync and health status to be green and so notification is swift.

This is great if all your deployments are successful. If you have an app that starts and exits with an exit 1 , Kubernetes will restart the pod. Again, it will start the process, set the pod to Ready and the process begins all over again into a state of CrashLoopBackOff, causing the deployment status to flap between Ready and Unready . This causes ArgoCD to continually update the Application resource status object. ArgoCD Notifications picks up on each of these status changes and can cause a flood of messages to your Slack channel or receiver. This occurred with some of our apps that didn’t have probes. To combat this, we did the following:

1. Add probes where there were no probes before
2. Set the spec.minReady option to 30 seconds

This stopped the flapping notifications although it added some acceptable delay.

Accurate Failure Notifications

By default, Kubernetes doesn’t class many things as a hard failure. Having a self-recovering system can muddy the waters with what is failed or in recovery. This means that out of the box, very few things are classed as hard failures. InvalidImageName is one of them. We wanted to a notification if we couldn’t complete a deployment. To do this, we set spec.progressDeadlineSeconds . This setting configures the deadline for how long we should wait for a deployment to be classed as successful before classing it as a failure. We now no longer incorrectly class a deployment as successful following the addition of spec.minReady so now we can set an appropriate deadline. By default, progressDeadlineSeconds is set to 600 seconds, we set it to 300 to be a little stricter.

Notification Trigger Conditions

ArgoCD Notifications comes with a number of default triggers that are a nice starting point. We typically want to rely on an app being deployed successfully or failed. The default triggers for successful actions rely on just the sync status. To get accurate notifications on deployments, we added the health status in as an additional data source. The condition is now as follows (I know about the name ;)):

- name: on-health-healthy
  condition: app.status.sync.status == 'Synced' and app.status.health.status == 'Healthy'
  template: app-health-healthy

Enriching Slack Messages

The default templates for ArgoCD are a nice start but pretty basic. We use ArgoCD to sync developer applications to Kubernetes and the developers will want to know more information about the app they’re deploying. For example, what’s the url of the app or what was commit message or commit id?

ArgoCD allows creating new templates so that we can customise these notifications. The best way I’ve found to do it is to add these custom bits of detail to the ArgoCD Application resource as an annotation. We can then pull these out in the template. An ArgoCD Application resource snippet is as follows:

{
  "apiVersion": "argoproj.io/v1alpha1",
  "kind": "Application",
  "metadata": {
    "name": "example",
    "namespace": "argocd",
    "finalizers": [
      "resources-finalizer.argocd.argoproj.io"
    ],
    "annotations": {
      "recipients.argocd-notifications.argoproj.io": "slack:deployments",
      "author": "<SLACK_ID>",
      "environment": "production",
      "app_url": "https://test.example.com",
      "commit_msg": "This is a test commit",
      "commit": "aabbcc123"
    }
  },
  ...
}

We can then consume details such as this like so. This example is done in a Helm template:

...
{
  "title": "App URL",
  "value": "{{ `{{.app.metadata.annotations.app_url}}` }}",
  "short": true
}, {
  "title": "Commit URL",
  "value": "<{{ `{{.app.metadata.annotations.commit_url}}` }}|{{ `{{.app.metadata.annotations.commit}}` }}>",
  "short": false
}, {
...

As Helm templates render Golang templates, we’ve had to include the extra curly brackets and comment out the ArgoCD templating so that it get’s passed through and rendered by ArgoCD Notificationds and not Helm.

As a bonus, here’s how to handle Slack user mentions via these templates:

<@{{ `{{.app.metadata.annotations.author}}` }}>

Just make sure your author has the Slack ID(e.g. UF12345), not their display name (e.g. Harry).

Conclusion

I hope these tips and tricks helped you out. There was one more issue I remember where ArgoCD Notifications was sending notifications for a previous resource but I honestly can’t remember how I fixed it! If you have any questions or suggestions or fixes, please let me know and I’ll be sure to add them or address. Thanks for reading! ❤