Metrics

Agones controller exposes metrics via OpenCensus. OpenCensus is a single distribution of libraries that collect metrics and distributed traces from your services, we only use it for metrics but it will allow us to support multiple exporters in the future.

We choose to start with Prometheus as this is the most popular with Kubernetes but it is also compatible with Stackdriver. If you need another exporter, check the list of supported exporters. It should be pretty straightforward to register a new one. (GitHub PRs are more than welcome.)

We plan to support multiple exporters in the future via environment variables and helm flags.

Backend integrations

Prometheus

If you are running a Prometheus instance you just need to ensure that metrics and kubernetes service discovery are enabled. (helm chart values agones.metrics.prometheusEnabled and agones.metrics.prometheusServiceDiscovery). This will automatically add annotations required by Prometheus to discover Agones metrics and start collecting them. (see example)

Prometheus Operator

If you have Prometheus operator installed in your cluster, make sure to add a ServiceMonitor to discover Agones metrics as shown below:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: agones
  labels:
    app: agones
spec:
  selector:
    matchLabels:
        agones.dev/role: controller
  endpoints:
  - port: web

Finally include that ServiceMonitor in your Prometheus instance CRD, this is usually done by adding a label to the ServiceMonitor above that is matched by the prometheus instance of your choice.

Stackdriver

We support the OpenCensus Stackdriver exporter. In order to use it you should enable Stackdriver Monitoring API in Google Cloud Console. Follow the Stackdriver Installation steps to see your metrics on Stackdriver Monitoring website.

Metrics available

Name Description Type
agones_gameservers_count The number of gameservers per fleet and status gauge
agones_gameserver_allocations_duration_seconds The distribution of gameserver allocation requests latencies histogram
agones_gameservers_total The total of gameservers per fleet and status counter
agones_fleets_replicas_count The number of replicas per fleet (total, desired, ready, allocated) gauge
agones_fleet_autoscalers_able_to_scale The fleet autoscaler can access the fleet to scale gauge
agones_fleet_autoscalers_buffer_limits The limits of buffer based fleet autoscalers (min, max) gauge
agones_fleet_autoscalers_buffer_size The buffer size of fleet autoscalers (count or percentage) gauge
agones_fleet_autoscalers_current_replicas_count The current replicas count as seen by autoscalers gauge
agones_fleet_autoscalers_desired_replicas_count The desired replicas count as seen by autoscalers gauge
agones_fleet_autoscalers_limited The fleet autoscaler is capped (1) gauge
agones_gameservers_node_count The distribution of gameservers per node histogram
agones_nodes_count The count of nodes empty and with gameservers gauge
agones_gameservers_state_duration The distribution of gameserver state duration in seconds. Note: this metric could have some missing samples by design. Do not use the _total counter as the real value for state changes. histogram
agones_k8s_client_http_request_total The total of HTTP requests to the Kubernetes API by status code counter
agones_k8s_client_http_request_duration_seconds The distribution of HTTP requests latencies to the Kubernetes API by status code histogram
agones_k8s_client_cache_list_total The total number of list operations for client-go caches counter
agones_k8s_client_cache_list_duration_seconds Duration of a Kubernetes list API call in seconds histogram
agones_k8s_client_cache_list_items Count of items in a list from the Kubernetes API histogram
agones_k8s_client_cache_watches_total The total number of watch operations for client-go caches counter
agones_k8s_client_cache_last_resource_version Last resource version from the Kubernetes API gauge
agones_k8s_client_workqueue_depth Current depth of the work queue gauge
agones_k8s_client_workqueue_latency_seconds How long an item stays in the work queue histogram
agones_k8s_client_workqueue_items_total Total number of items added to the work queue counter
agones_k8s_client_workqueue_work_duration_seconds How long processing an item from the work queue takes histogram
agones_k8s_client_workqueue_retries_total Total number of items retried to the work queue counter
agones_k8s_client_workqueue_longest_running_processor How long the longest running workqueue processor has been running in microseconds gauge
agones_k8s_client_workqueue_unfinished_work_seconds How long unfinished work has been sitting in the workqueue in seconds gauge

Dashboard

Grafana Dashboards

We provide a set of useful Grafana dashboards to monitor Agones workload, they are located under the grafana folder :

Dashboard screenshots :

grafana dashboard autoscalers

grafana dashboard controller

Installation

When operating a live multiplayer game you will need to observe performances, resource usage and availability to learn more about your system. This guide will explain how you can setup Prometheus and Grafana into your own Kubernetes cluster to monitor your Agones workload.

Before attemping this guide you should make sure you have kubectl and helm installed and configured to reach your kubernetes cluster.

Prometheus installation

Prometheus is an open source monitoring solution, we will use it to store Agones controller metrics and query back the data.

Let’s install Prometheus using the Prometheus Community Kubernetes Helm Charts repository.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm upgrade --install --wait prom prometheus-community/prometheus --version 11.16.2 --namespace metrics \
    --set server.global.scrape_interval=30s \
    --set server.persistentVolume.enabled=true \
    --set server.persistentVolume.size=64Gi \
    -f ./build/prometheus.yaml

For resiliency it is recommended to run Prometheus on a dedicated node which is separate from nodes where Game Servers are scheduled. If you use the above command, with our prometheus.yaml to set up Prometheus, it will schedule Prometheus pods on nodes tainted with agones.dev/agones-metrics=true:NoExecute and labeled with agones.dev/agones-metrics=true if available.

As an example, to set up a dedicated node pool for Prometheus on GKE, run the following command before installing Prometheus. Alternatively you can taint and label nodes manually.

gcloud container node-pools create agones-metrics --cluster=... --zone=... \
  --node-taints agones.dev/agones-metrics=true:NoExecute \
  --node-labels agones.dev/agones-metrics=true \
  --num-nodes=1

By default we will disable the push gateway (we don’t need it for Agones) and other exporters.

The helm chart supports nodeSelector, affinity and toleration, you can use them to schedule Prometheus deployments on an isolated node(s) to have an homogeneous game servers workload.

This will install a Prometheus Server in your current cluster with Persistent Volume Claim (Deactivated for Minikube and Kind) for storing and querying time series, it will automatically start collecting metrics from Agones Controller.

Finally, to access Prometheus metrics, rules and alerts explorer use

kubectl port-forward deployments/prom-prometheus-server 9090 -n metrics

Now you can access the prometheus dashboard http://localhost:9090.

On the landing page you can start exploring metrics by creating queries. You can also verify what targets Prometheus currently monitors (Header Status > Targets), you should see Agones controller pod in the kubernetes-pods section.

Now let’s install some Grafana dashboards.

Grafana installation

Grafana is a open source time series analytics platform which supports Prometheus data source. We can also easily import pre-built dashboards.

First we will install Agones dashboard as config maps in our cluster.

kubectl apply -f ./build/grafana/

Now we can install the Grafana Community Kubernetes Helm Charts from
their repository. (Replace <your-admin-password> with the admin password of your choice)

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

helm upgrade --install --wait grafana grafana/grafana --version=5.7.10 --namespace metrics \
  --set adminPassword=<your-admin-password> -f ./build/grafana.yaml

This will install Grafana with our prepopulated dashboards and prometheus datasource previously installed

Finally to access dashboards run

kubectl port-forward deployments/grafana 3000 -n metrics

Open a web browser to http://localhost:3000, you should see Agones dashboards after login as admin.

Stackdriver installation

In order to use Stackdriver monitoring you must enable the Monitoring API in the Google Cloud Console. You need to grant all the necessary permissions to the users (see Access Control Guide). Stackdriver exporter uses a strategy called Application Default Credentials (ADC) to find your application’s credentials. Details can be found in Setting Up Authentication for Server to Server Production Applications.

The default metrics exporter installed with Agones is Prometheus. If you are using the Helm installation, you can install or upgrade Agones to use Stackdriver, using the following chart parameters:

helm upgrade --install --wait --set agones.metrics.stackdriverEnabled=true --set agones.metrics.prometheusEnabled=false --set agones.metrics.prometheusServiceDiscovery=false my-release-name agones/agones --namespace=agones-system

With this configuration only Stackdriver exporter would be used instead of Prometheus exporter.

Using Stackdriver with Workload Identity

If you would like to enable stackdriver in conjunction with Workload Identity, there are a few extra steps you need to follow:

  1. When setting up the Google service account following the instructions for Authenticating to Google Cloud, create two IAM policy bindings, one for serviceAccount:PROJECT_ID.svc.id.goog[agones-system/agones-controller] and one for serviceAccount:PROJECT_ID.svc.id.goog[agones-system/agones-allocator].

  2. Pass parameters to helm when installing Agones to add annotations to the agones-controller and agones-allocator Kubernetes service accounts:

helm install my-release --namespace agones-system --create-namespace agones/agones --set agones.metrics.stackdriverEnabled=true --set agones.metrics.prometheusEnabled=false --set agones.metrics.prometheusServiceDiscovery=false --set agones.serviceaccount.allocator.annotations."iam\.gke\.io/gcp-service-account"="GSA_NAME@PROJECT_ID\.iam\.gserviceaccount\.com" --set agones.serviceaccount.controller.annotations."iam\.gke\.io/gcp-service-account"="GSA_NAME@PROJECT_ID\.iam\.gserviceaccount\.com"

To verify that metrics are being sent to Stackdriver, create a Fleet or a Gameserver and look for the metrics to show up in the Stackdriver dashboard. Navigate to the Metrics explorer and search for metrics with the prefix agones/. Select a metric and look for data to be plotted in the graph to the right.

An example of a custom dashboard is:

stackdriver monitoring dashboard

Currently there exists only manual way of configuring Stackdriver Dashboard. So it is up to you to set an Alignment Period (minimal is 1 minute), GroupBy, Filter parameters and other graph settings.

Troubleshooting

If you can’t see Agones metrics you should have a look at the controller logs for connection errors. Also ensure that your cluster has the necessary credentials to interact with Stackdriver Monitoring. You can configure stackdriverProjectID manually, if the automatic discovery is not working.

Permissions problem example from controller logs:

Failed to export to Stackdriver: rpc error: code = PermissionDenied desc = Permission monitoring.metricDescriptors.create denied (or the resource may not exist).

Last modified October 18, 2021: Add winterpixels rocket bot royale to list of agones links (#2316) (3fa5c57)