Datadog monitoring at Sourcegraph

Datadog is external tool used for monitoring.

RFC

Its purpose is not to replace existing monitoring and alerting, but to extend it with more options including:

  • metrics
  • logs
  • APM & traces
  • RUM (real user monitoring)
  • etc.

See a short demo of our Datadog integration

Content

Login

Datadog can be accessed through Okta at this link

Use cases

Dashboards

Datadog documentation

All available dashboards can be access here.

DevOps team is using Cloud Uptime as overall Cloud Production overview.

Metrics

Datadog documentation

Logs

Datadog documentation

Facets are are user-defined tags and attributes from indexed logs. They allow to query, group and aggregate logs by faceted fields.

APM and traces

Datadog documentation

Synthetic tests

Datadog documentation

Sourcegraph Cloud production Smoke tests example

Real User Monitoring

Datadog documentation

Monitors

Monitors are used to notify teams and manage alerts at a glance on the Alerting platform.

Datadog documentation

Sample Smoke tests monitor

Managing Datadog resources

The long term idea is to use terraform datadog provider to persist state of all resources. During the first part of trial period, terraform will be only importing changes made via UI, to allow people explore Datadog capabilities.

Terraform implementation

FAQ

  1. Cannot search and group logs by field from the log.
    • Ensure that field is a facet.

Running Datadog

To run the agent locally, you can run the agent in docker with the following command:

docker run -d --cgroupns host \
              -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v /proc/:/host/proc/:ro \
              -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
              -p 127.0.0.1:8126:8126/tcp \
              -e DD_API_KEY=<DD_API_KEY>     \
              -e DD_APM_ENABLED=true \
            gcr.io/datadoghq/agent:latest

You will also need to modify the site-config (typically dev-private) to specify the other tracer

   "observability.tracing": {
     "sampling": "selective",
-    "type": "opentracing"
+    "type": "datadog"
   },

Lastly, run Sourcegraph with the following command: DD_ENV=dev DD_PROFILE_ALL=true sg start These env vars are need to ensure Zoekt uses the Datadog agent.

You can access the traces the same way as specified here

Enabling Datadog on Cloud

To enable Datadog on Cloud, you need to:

  1. Configure the site-config to contain the following:
   "observability.tracing": {
     "type": "datadog"
   },
  1. Set the env var “DD_ENV” on the indexed-search service. This should be set to the environment its being deployed to, ie “dev”, “staging”, “prod”.