Smule Grapher (SMG) - Concepts Overview

Concepts overview

RRD objects

Intervals, runs and scheduler

SMG has the concept of regular (every interval seconds) runs where it will execute the commands specified in the objects configs. All objects having the same interval config value will be processed during the respective interval run (also see the next section on pre_fetch and run trees). All per-interval runs execute in their own separate thread pool, so that e.g. slow hourly commands do not interfere too much with fast every-minute commands. Aggregate RRD object updates happen after all regular rrd objects have been updated, at the end of the interval run. Thread pools are configurable via the $interval_def global.

By default SMG will use its internal scheduler to trigger the regular runs. Without getting into too much detail here it will trigger (separate) runs for every discovered unique rrd object (or plugin) interval value, whenever the applicable time comes. If desired one can also use an external scheduler like cron. For this to work one needs to set smg.useInternalScheduler in application.conf to false and then schedule the smgscripts/run-job.sh script as cron jobs (per interval, passed as param to run-job.sh) as needed.

pre_fetch and run command trees

One important thing when trying to get and update RRDs for tens of thousands of values from remote servers every minute is to reduce the number of remote connections one has to make to the target (monitored) servers. For example in mrtg, nagios (and if not misatken - in cacti too) every check/graph object configured results in a call to a separate external command getting remote data and/or a SNMP get.

Instead (and ideally) we would use a single command to get all the values we want from a given host and then possibly produce many rrd objects and graphs from these. Note that this is how Prometheus works with its /metrcis calls but that only works over http.

This is where SMG’s pre_fetch simplifies things - it is a special config object defining an id and a command to execute. Then RRD objects can have a pre_fetch attribute specified with a value - the mentioned pre_fetch id.

This allows the pre_fetch command to store its ouput data cached somehere locally on the SMG host and then the actual RRD object commands do not need to go to the target server but can use the cached local data. For example we use a pre_fetch command to get all SNMP values we care about from a given host in a single shot (about sysload, cpu usage, i/o, mem etc). Then we update many RRD objects from the pre-fetched data. Since recently pre_fetch also supports passing the command stdout directly to the child command stdin without the need to use temp files.

In addition pre_fetch itself can specify another pre_fetch command to be its parent. That way one can define a hierarchical command tree structure (“run tree”) where many child commands depend on a single parent command and a set of parents can define a common parent of their own etc. Example use case for this is to define a top-level ping -c 1 ip.addr pre_fetch command for given host and then all other pre_fetches (or actual rrd objects) for that host can have the ping pre_fetch defined as parent and will not run at all if e.g. the host is down (not ping-able). Having such setup will also make SMG only send a single “failed” alert if a host is down (vs alerts for each individual command).

An important note is that SMG will execute child pre-fetch commands in sequence (but not rrd object commands which are always parallelized up to the max concurrency level configured for the interval). The intent is to be able to limit the number of concurrent probes hitting some target host to 1 per interval. This behavior can be overriden by setting the child_conc property on the pre-fetch command to more than the default 1 (setting the number of max concurrent threads that can execute child pre-fetches of this pre-fetch).

Period since and period length

Filters

SMG maintains the list of all objects available for graphing in memory.

Whenever it has to display some set of (different objects) graphs, the selection happens via a filter over the global list. This is how one selects which graphs to see on the page. This is also why object ids are very important, so that you can easily select classes of objects to display together.

Remote filter

Text Regex filter

• when set, only objects where any of object id, title, command, object labels, var labels or measurement units is matched by the specified regular expressions will be matched by the filter.

Prefix filter

• when set, only object ids having the specified prefix string (starting with it) will be matched by the filter. Note that this is not a regexp but a direct string comparison (and e.g dots are not special match-all symbols).

Suffix filter

• when set, only object ids having the specified suffix string (ending with it) will be matched by the filter. Note that similar to prefix filter this is not a regexp but a direct string comparison.

Regex filter

• when set, only object ids matching the specified regular expressions will be matched by the filter.

Regex Exclude filter

• when set, only object ids NOT matching the specified regular expressions will be matched by the filter.

Parent Regex filter

• when set, only objects whose parent ids match the specified regular expressions will be matched by the filter.

Labels filter

Index filter

Indexes

Custom dashboards

Graph options

These generally allow one to set some non-default graph options, when desired. By default SMG will plot the data for the requested period but also plot the previous period of the same length using dotted lines. It will also plot a dashed line at the 95 perecntile value for each plotted variable. Both of these can be disabled using the respective check-box in the filter UI.

Step

By default rrdtool will pick the “best” possible step (what period a data point in the graph will correspond to) based on the available in RRAs data and the requested period. Usually that will be the highest available resolution data which would also fit on the graph. E.g. for graphs updated every minute, SMG will display 1 data point per minute (1-min average) on a 24-hours graph. If we want to see the data at 5-minute average values we could set step to 300 (seconds) and get that.

MaxY

Sometimes one would get big “spikes” in the data where for a short period the data points values get orders of magnitude higher than normal. When displaying such a graph one usually sees a huge spike and the values around it look like a flat line, close to 0. So in order to see details around such spikes one can set the MaxY (maximum Y) value of the graphs. Such a graph will have a gap where the spike was occurring but the surrounding time series data can be seen in a good detail.

MinY

Similar to MaxY, MinY determines the minimum Y value which will be displayed on the graphs.

Rows and columns

Whenever SMG displays the graphs resulting from the given filter it uses the Rows and Columns form parameter values to limit the displayed graphs (it is unlikely that we want 10000 graphs displayed on one page). So one would see at most “Columns” columns of graphs in a “row” (may be less, depending on screen size) with at most “Rows” rows per page.

Sorting and grouping

SMG supports sorting a displayed page of graphs by the average value of a given object variable. The object variable is specified via the 1-based integer position of the variable within the list of object ones - the x-sort dashboard filter form field.

For example if you have page of graphs showing network switch ports (objects with two variables - one for incoming and one for outgoing traffic) and if you want to sort them by their outgoing traffic (that would be the second var in the object’s vars list) you would set x-sort = 2. The default x-sort value (0) implies no sorting.

One important note is that sorting is an expensive operation - it involves a rrdtool fetch command in addition to the graph command (rrdtool graph) and the former is almost as expensive as the later. Because of this currently SMG will only sort within the currently displayed page of graphs. This is mainly to avoid cases where (possibly by mistake) one can try to sort all graphs (possibly hundreds of thousands) by setting x-sort to 1 on a match-all filter.

The current workaround to that limitation is to set a high-enough rows parameter so you get all the graphs you want to sort on one page which you can sort after.

A special case of sorting is group by (with value of x-sort=-1, or the GroupBy button). In this case graphs will be grouped for display based on the Group By drop down value.

Aggregate functions

SMG supports “aggregating” graphs for objects which have compatible set of time series and grouped based on the Group By drop down value mentioned above. This works using one of the currently available functions (these are available as buttons on the graphs display page):

• GROUP - just putting the same type graphs together in a single graph
• STACK - stacking the graph lines on top of each other
• SUM - summing the same-type lines form the multiple objects together
• SUMN - same as SUM except that NaN values are treated as 0. This is mostly useful when you want to see some historical data for a sum of graphs but where some of these were created later. With SUM one would see gaps where one of the component has NaN value for the time point where SUMN will happily conver the NaNs to 0 and still display a line.
• AVG - similar to SUM but displaying graphs for the average value of the same-typed lines.
• MAX - similar to SUM but displaying graphs for the max value across the same-typed lines.
• MIN - similar to SUM but displaying graphs for the min value across the same-typed lines.

By default (and when the filter results in objects from multiple remotes) SMG will do the aggregation “by remote”, i.e will produce separate aggregate images for given set of identical var-definition objects - one per remote instance. One can request a Cross-remote aggregation by selecting the respective check-box, before clicking on the aggregate function button. This works by downloading the remote rrd files locally and then producing an image from them so it is expected to be somewhat slower.

View objects and calculated graphs

In addition to the mentioned RRD objects, SMG also supports View objects. These are defined on top of RRD objects - every View object must reference a RRD Object. Technically SMG cares only about RRD (update) objects when doing the interval runs for updates but only cares about View objects when filtering for display.

By default every RRD Object is also a View object so one does not need to explicitly define View objects except in some cases described below.

Remotes

SMG was designed with multiple data centers (a.k.a. “remotes”) in mind from the start (this was actually one of the drivers for us to create SMG).

We run multiple Data Centers (DCs) at Smule. Normally one does not want to run cross-dc monitoring probes (often simply because its too slow). Also one can not expect a single monitoring instance (whether SMG, mrtg or cacti) to be able to keep up with a huge number of every-minute updates, whether across multiple data centers or within the same DC. So having one humongous monitoring instance does not work and normally people using mrtg or cacti would split the graphs across multiple instances to reduce the number of graphs each individual instance needs to keep up with. The draw-back is that one has to go to different host/url to get the relevant graphs and there is no single “namespace” where one can view graphs from multiple instance in the same UI.

This is where SMG remotes come into play. Similar to mrtg and cacti we would split the stuff we want graphed into multiple instances. Each of these instances will maintain and update its own subset of the rrd objects.

However in SMG one can define remote instances (host:port) where it is able to get the configured (RRD+View) objects in the remote instance and “merge” these into the local objects list. All remote ids are prefixed with @<remoteid>. when merged into the local object ids namespace to avoid name conflicts (the ‘@’ symbol is otherwise forbidden in object ids).

Authentication

SMG has always been a mostly read-only system - everything it does is configured by an admin or a configuration management system via the yaml config. The only “write” operations exposed via the UI are “silence” and “acknowledge”. In addition it is not hard to put SMG behind an authenticated reverse procy and delegate the auth to an external corporate authentication system. Because of these implementing auth was never considered a very high priority, yet as of v1.4 SMG supports some built-in authentication and authorization.

SMG has two types of APIs - the “Local UI” method handlers and the “API” method handlers (the later uised almost exclusively for SMG remotes support). Because of that there are also two types of users internally - “regular” and “system”. The “system” user is used to authenticate API calls and “regular” users are used for accessing the UI. Normally the system user also carries info about the regular user on behalf of which the action is executed.

In addition SMG supports 3 possible roles:

• VIEWER - all read operations are allowed, can not silence/acknowledge alerts
• ADMIN - all read operations are allowed, can also silence/acknowledge alerts
• ROOT - everything is allowed, including ability to reload conf and access “privileged” plugins

Intenally SMG has the concept of “User provider” which is an interface allowing the system to authenticate individual requests. There are a few such built-in providers:

• Anonymous user provider - allowing SMG to work with the built-in authentication effectively disabled.
• System user provider - allowing SMG to use headless accounts for certain tasks. This can work based on whitelisted IPs/ranges or secret tokens
• Password user provider - allowing admins to define users and passwords as part of the SMG config
• Plugin user providers - the Plugin API now has an extension point to implement custom authentication providers. The built-in “auth” plugin exposes one such - “Trusted header” user provider. This assumes that SMG is behind and authenticated reverse proxy (no direct access allowed) and can use headers set by that proxy to identify user handle, name and role.

Plugins

SMG supports plugins (configured in application.conf) which are just Scala classes implementing a specific interface (trait in Scala terms).

At a high level plugins can do a bunch of things:

• each plugin has an interval associated with it. Its run() method will be executed by the scheduler as needed, every interval seconds.

• plugins can provide custom RRD objects implementations. For example some data may be available for graphing later - e.g. logs rotated hourly where every hour we want to graph the previous hour (now rotated) data, e.g. per minute. This does not fit in the usual SMG model of run-this-command-every-X-minutes-to-get-data-points and we have plugin(s) which can do this kind of stuff. Also, there may be some cases where you don’t even know all the objects you want to graph upfront (so that you could configure them). A custom SMG plugin can dynamically discover objects as they appear.

• provide custom indexes - similar to custom objects, plugin can provide relevant indexes too (normally - grouping the custom plugin objects in some way which makes sense)

• plugin can also provide arbitrary display of some relevant data (e.g. calc plugin) and also provide “object actions” to provide display of data specific to an object (e.g. the jsgraph plugin providing “Zoom” and “Historam” actions)

• Plugin can implement monitor checks (via valueChecks method) to extend the available by default alert-[warn|crit]-[gt[e]|lt[e]|eq] options. E.g. one can check for value anomalies etc. See the mon plugin for example.

• Plugins can implement custom dashboard items - TODO

• Plugins can implement custom user providers - TODO

• Plugins can implement “internal commands” for various reasons including e.g. keeping persistent JMX connections between runs or more efficient parsing of things like OpenMetrics (Prometheus) and CSV stats,

Currently available plugins

Monitoring

SMG main function is to fetch many values from various services and keep RRD databases updated using that information. Also in traditional setups one might run mrtg or cacti (or SMG) for graphing and nagios for alerting. This implies that a lot of values are fetched twice - once for nagios and once for graphing. So it makes sense to be able to define alerts based on the wealth of values SMG fetches and avoid the double-hit on the target servers.

The original idea for SMG monitoring was to be implemented as plugins fetching the data from the rrd files and determining whether values are within thresholds. This also allows one to check data at arbitrary (as available in the RRD files) resolution for desired periods. Unfortunately there is one significant issue with that approach - it is too heavy to apply on all objects (based on experience it can almost half the number of objects SMG is capable to update every minute on given hardware).

This is not acceptable so now (as of v0.2) SMG has built-in monitoring based on a “live” (in memory) data feed consisting of all the values fetched for RRD updates. One can define alerts (acceptable values/ranges) in SMG as part of the object variables configurations or as part of indexes (including “hidden” indexes which sole current purpose is to define alerts).

Anomaly detection (via the “mon” check plugin)

The SMG Mon check plugin anomaly detection works based on a “threshold for difference” (default = 1.5) and two time periods - shorter (default 30 minutes) and longer (default - 30 hours). Will use the default values in the examples below.

First, for any value which has a “spike check” configuration (“alert-p-mon-anom”) enabled SMG will start keeping in-memory stats about the most recent values. The number of stats that will be kept depends on the object interval compared to the configured check periods - e.g. for a graph updated every minute, 30 minutes would map to 30 data points and 30h to - 1800. SMG keeps the short period values (30 in the example) at full granularity. For the long period data instead of keeping all individual 1800 values SMG will group them in overlapping chunks each with the size of the short period or each chunk will cover 30 minutes. How that works is that internally SMG keeps up to half-short period (30 / 2 = 15) values more at full granularity (in addition to the 30 short-term ones) named “previous short term” values. Whenever that list size increases to half the short-period, its values are combined with the first half of the short-period values (for a full short-period sized chunk) and SMG will add an aggregate value to the “long term” stats list before truncating the “previous short term” list to empty (and also throwing away a long term stat if above max size). The actual aggregation consist of calculating the following stats for the chunk:

Running and troubleshooting

Run in a container

• mkdir -p /opt/smg/data /etc/smg/conf.d

• docker run -d –name smg -p 9000:9000 -v /opt/smg/data -v /etc/smg/conf.d/ gcr.io/asen-smg/smulegrapher:latest

• Point your browser to http://$DOCKER_HOST:9000 (the local metrics stats should show up in a minute or two)

• Then add stuff under /etc/smg/conf.d (and/or /opt/smg/data/conf/autoconf.d) and to reload conig use one of:

  • docker exec smg /opt/smg/inst/smg/smgscripts/reload-conf.sh
  • curl -X POST http://$DOCKER_HOST:9000/reload

Run in k8s

Check the k8s/ dir for example deployment yamls, including in-cluster monitoring with auto-discovery (similar to Prometheus)

Classic mode

Pre-requisites

SMG needs the following software to run.

• SMG needs bash and gnu timeout to work out of the box. Bash is available on pretty much any unix installation and gnu timeout may have to be installed (on e.g. Mac). It should be possible to run under Windows using cygwin but never tried.

Note: The “bash” and “timeout” commands are actually configurable in application.conf and can be replaced with other commands with similar functionality. E.g. in theory it should be possible to replace bash with cmd.exe.

Installation from .tgz bundle

SMG comes as a pre-built tgz archive named like smg-$VERSION.tgz (e.g. smg-0.3.tgz). You can unpack that anywhere which will be the SMG installation dir. Here are example commands to unpack SMG in /opt/smg:

# cd /opt
# wget https://github.com/asen/smg/releases/download/v1.3/smg-1.3.tgz
# tar -xf smg-*.tgz
# mv smg-* smg # (or ln -s smg-* smg)

SMG comes with start and stop and scripts which in the above example would be available as:

• /opt/smg/start-smg.sh - use this to start SMG as a daemon process. It requires JAVA_HOME to be pointed to a supported Java installation.

• /opt/smg/stop-smg.sh - use this to gracefully stop the running as a daemon SMG process.

Installation from sources

• This is mostly meant for development purposes.

Reverse proxy

• see the k8s/ deployment with nginx in front of the Play app
• example apache Reverse proxy conf

        Listen 9080

        <VirtualHost *:9080>
            # optional config to intercept /proxy/ requests to the respective remotes
            # and not have the SMG JVM do the proxying
            ProxyPass        /proxy/some-dc/  http://smg1.some-dc.myorg.com:9080/
            ProxyPassReverse /proxy/some-dc/  http://smg1.some-dc.myorg.com:9080/
            ProxyPass        /proxy/other-dc/ http://smg2.other-dc.myorg.com:9080/
            ProxyPassReverse /proxy/other-dc/ http://smg2.other-dc.myorg.com:9080/

            # do not proxy static files/images but serve them directly
            # from the configured SMG $img_dir
            ProxyPass  /assets/smg !
            Alias /assets/smg /opt/smg/public/smg

            # the actual proxy to SMG
            ProxyPass        /  http://localhost:9000/
            ProxyPassReverse /  http://localhost:9000/
        </VirtualHost>

• application.conf

• logs

Developer documentation

Use the source …

Development setup

• Install JDK 11+<

• Install rrdtool and coreutils (for gnu timeout command). E.g. with homebrew on Mac:

  $ brew install rrdtool coreutils

• (Only applicable on Mac) SMG currently needs to use gnu timeout where the default timeout command on Mac is different. There are two ways to address this (after installing coreutiles and gtimeout)

  • change application.conf:

    # Use smg.timeoutCommand to override the timeout command executable (e.g. gtimeout on mac with homebrew)
    smg.timeoutCommand = "gtimeout"
    

  • Link /usr/local/bin/timeout to /usr/local/bin/gtimeout on your Mac (that assumes that /usr/local/bin is before /usr/bin in your $PATH

    ln -s /usr/local/bin/gtimeout /usr/local/bin/timeout
    

  <

• Run:

    $ cd smg
    $ JAVA_HOME=$(/usr/libexec/java_home -v 11) ./run-dev.sh