Right now make use of AWS Elasticbeanstalk for most of the production services, and the provided AWS CloudWatch monitoring covers the basics of instance health.
...but...
We aren't exclusively AWS and we want to start collecting some custom metrics from some of the services we have deployed. Instead of trying to hack in more custom metrics into AWS cloudwatch, we've decided to see if we can deploy something we had more control over.
After spending a little time test driving possible solutions, I've come to the conclusion that I want to implement a sensu monitoring service in our infrastructure. And as part of that, I want to build some reusable Fedora srpms for sensu to make the process easier.
Why Sensu?
The sensu client-server model is highly extensible and seems to fit well across any infrastructure type from dedicated on-premises metal, to public containerized applications.I really like the sensu client self-registration approach so that when I spin up instances the associated monitoring checks just work, like how I've come to expect CloudWatch to work in that regard.
And I love the Nagios plugin compatibility, there are bits of static infrastructure here and there that we still monitor with Nagios, it would be really nice to consolidate that monitoring into a common monitoring system and still be able to make use of the existing Nagios plugin packages in Fedora and EPEL.
I'll also most likely be coupling the sensu service with graphite/grafana for metric visualization. And instead of running a separate statsd service to feed custom metrics to graphite, I'm going to look into using the sensu statsd extension to service statsd udp packets so we can make use of statsd metrics as well as sensu check based metrics.
Why Fedora Packages?
At the end of the day I'm deploying production on CentOS, but I do much of my development using Fedora. I'd really like to spin up a set of sensu packages I can cleanly rebuild for Fedora and CentOS and eventually get them officially built as part of the repositories ecosystems for those distributions.Looking forward a little bit further, what I really want is to be able to have reasonably optimized packaging for sensu so that I can put together sensu enabled k8s pods using simple buildah scripting so we can deploy production services in a more flexible manner. But Rome wasn't built in a day, for right now just getting a monitoring solution up and running to support our existing AWS production is the primary goal.
I'm just getting started on this work. I'm going to be using a github repository to document any roadblocks and to hold the fedora packaging bits as they evolve towards being submittable for inclusion in the Fedora repositories. If you are interested in following along in the effort, you can start tracking https://github.com/jspaleta/sensu-fedora-bringup
.
Watch this space for more sensu tips and tricks.
