Datadog monitoring at Sourcegraph
Datadog is external tool used for monitoring.
Its purpose is not to replace existing monitoring and alerting, but to extend it with more options including:
- metrics
- logs
- APM & traces
- RUM (real user monitoring)
- etc.
See a short demo of our Datadog integration
Content
Login
Datadog can be accessed through Okta at this link
Use cases
Dashboards
All available dashboards can be access here.
DevOps team is using Cloud Uptime as overall Cloud Production overview.
Metrics
Logs
Facets are are user-defined tags and attributes from indexed logs. They allow to query, group and aggregate logs by faceted fields.
APM and traces
Synthetic tests
Real User Monitoring
Monitors
Monitors are used to notify teams and manage alerts at a glance on the Alerting platform.
Managing Datadog resources
The long term idea is to use terraform datadog provider to persist state of all resources. During the first part of trial period, terraform will be only importing changes made via UI, to allow people explore Datadog capabilities.
FAQ
- Cannot search and group logs by field from the log.
- Ensure that field is a facet.
Running Datadog
To run the agent locally, you can run the agent in docker with the following command:
docker run -d --cgroupns host \
-v /var/run/docker.sock:/var/run/docker.sock:ro \
-v /proc/:/host/proc/:ro \
-v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \
-p 127.0.0.1:8126:8126/tcp \
-e DD_API_KEY=<DD_API_KEY> \
-e DD_APM_ENABLED=true \
gcr.io/datadoghq/agent:latest
You will also need to modify the site-config (typically dev-private) to specify the other tracer
"observability.tracing": {
"sampling": "selective",
- "type": "opentracing"
+ "type": "datadog"
},
Lastly, run Sourcegraph with the following command:
DD_ENV=dev DD_PROFILE_ALL=true sg start
These env vars are need to ensure Zoekt uses the Datadog agent.
You can access the traces the same way as specified here
Enabling Datadog on Cloud
To enable Datadog on Cloud, you need to:
- Configure the site-config to contain the following:
"observability.tracing": {
"type": "datadog"
},
- Set the env var “DD_ENV” on the indexed-search service. This should be set to the environment its being deployed to, ie “dev”, “staging”, “prod”.