Part 66: Add AI generated summaries to Zabbix or Grafana dashboards

AI wisdom

Now that my (admittedly repetitive) Generative AI blog has been up and running for a while, it's time to give you ideas how to utilize this AI weirdness inside your Zabbix or Grafana, too.

How about an automatically updated alert summary for the next on-call person around the time the shift is about to change? There would be multiple ways of doing that, in my case I did it like this.

Create a cronjob

My cronjob runs the following every hour: 

python3 ./zabbix_journalist.py --question "Create an on-call handover summary about the following Zabbix alerts" >/tmp/gpt4all_observations.txt

Create Zabbix item

Next I created a Zabbix item which just grabs the content of the text file.

Zabbix item

What's that preprocessing there? To make it not store the same story multiple times, I added Discard unchanged.

And it's done! Now, Zabbix picks up the latest and greatest story whenever it gets published.

Add a Plain text widget

With Plain text widget, you can then show the story. If you add some HTML formatting to your AI generated blurb, then the widget can parse that, too. My example is just plain text.

Plaintext widget

... but wait! Isn't all this just the same I already did in Part 62 of this blog? Yes, kind of. But this way you'll get your stories stored in Zabbix database and can utilize them elsewhere, too.

Add it to Grafana

So, if like many of us, you are using Zabbix for monitoring, collecting the data and building complex alert rules, and Grafana for visualizations, or you just have Grafana as your single pane of glass for all the data sources you have around, then this is for you.

Adding the AI summaries to Grafana is easy. Just use the Zabbix data source, choose Text, and find the item you created for your summaries.

Zabbix data source

For showing the text, I guess there are many ways; one of them is to use Grafana's built-in Table, but how boring is that? With Dynamic Text plugin you can have a text panel with dynamic contents.

Dynamic Text plugin

Alternatively, if you don't have the Grafana Zabbix plugin in use but can access the Zabbix API, you might use something like JSON API plugin or the space shuttle cockpit that is Infinity data source as it supports all the things you can imagine of.

Infinity plugin

I hope these hints do help you to kickstart your journey with AI in observability.

Add new comment

Restricted HTML

  • Allowed HTML tags: <a href hreflang> <em> <strong> <cite> <blockquote cite> <code> <ul type> <ol start type> <li> <dl> <dt> <dd> <h2 id> <h3 id> <h4 id> <h5 id> <h6 id>
  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.
Buy me a coffee

Like these posts? Support the project and Buy me a coffee