Enabling executors

Enabling executor on a managed instances is initiated by #wg-shipping-executors. There are a few logical steps to follow.

  1. Update the managed instances terraform to include executor modules
  2. Setup billing alert for the executor resources

Deploy executors

Update terraform config

Open $CUSTOMER/terraform.tfvars and uncomment the executors block

$COMPANY is already substituted with actual value during instance creation time, so you only need to uncomment the tfvars block without changing anything else

-# enable_executors                   = true
-# executor_instance_tag              = "$COMPANY-executors"
-# executor_metrics_environment_label = "$COMPANY"
-# executor_min_replicas              = 0
-# executor_max_replicas              = 4
-# executor_notification_email        = "REDACTED" #wg-shipping-executors
+enable_executors                   = true
+executor_instance_tag              = "$COMPANY-executors"
+executor_metrics_environment_label = "$COMPANY"
+executor_min_replicas              = 0
+executor_max_replicas              = 4
+executor_notification_email        = "REDACTED" #wg-shipping-executors

Apply the terraform module

cd $CUSTOMER
terraform apply

Add the executor token to the site configuration of the instance (note: this must be run in the $CUSTOMER directory)

mg executors set-token --token $(terraform output -raw executor_proxy_password)

Confirm executors is actually working

Then run the command below, which will

  • increase the minimal replica of executors auto scaling group to 1
  • retry until either the executors are up or a 5 minute timeout has been reached, after which you should check configuration errors if it fails
  • scale auto scaling group back to 0
mg executors check

Wrapping up

Commit your changes and open a PR. Let #wg-shipping-executors know it is ready!

Setup billing alert

As of , we give customers a fixed amount (e.g. $500) of computing credits for executors usage, so we need to setup billing alerts.

At a high level, we will create a new monitoring alerting channel (via email) in the GCP project, and configure a budget alert in our organization billing account. All billing alerts will be delivered to #wg-shipping-executors

Create slack email integration

  • Customers-facing managed instances alert email can be found here
  • Dev and demo instances alert email can be found here

Create budget

  1. Go to our billing dashboard - budgets & alerts
  2. Click CREATE BUDGET
  3. Create the budget with the following information

Name: sourcegraph-managed-$CUSTOMER-executors Time range: Custom range From: CURRENT DATE (check no end date) Projects: sourcegraph-managed-$CUSTOMER Services: All services Labels: executor_tag = $CUSTOMER-executors Credits: ✅ Promotions and others: ✅

create-budget-01

Budget type: Specified amount Amount: 500

create-budget-03

Only keep the 100% alert threshold rule

Uncheck Email alerts to billing admins and users

Link Monitoring email notification channels we created from the terraform module. Find the email channel by filtering the GCP project from the UI, make sure you select the right project sourcegraph-managed-$CUSTOMER.

create-budget-04

create-budget-02

Troubleshooting

Ensure instance groups are there

$ gcloud compute instance-groups list --zones=$zone --project=$PROJECT
NAME                             LOCATION       SCOPE  NETWORK                MANAGED  INSTANCES
batches--sourcegraph-executor    us-central1-a  zone   sourcegraph-executors  Yes      0
codeintel--sourcegraph-executor  us-central1-a  zone   sourcegraph-executors  Yes      0

Ensure minNumReplicas is greater than 0

$ gcloud compute instance-groups managed describe batches--sourcegraph-executor --zone=$ZONE --project=$PROJECT --format=json | jq '.autoscaler.autoscalingPolicy'
{
  "minNumReplicas": 1,
}

$ gcloud compute instance-groups managed describe codeintel--sourcegraph-executor --zone=$ZONE --project=$PROJECT --format=json | jq '.autoscaler.autoscalingPolicy'
{
  "minNumReplicas": 1,
}

Ensure there is an active instance belong to one of the instance group (notes the batches--sourcegraph-executor-rqfs instance). Sometimes it may take GCP longer to spawn a new instance, so be patient. If no new instance is created for an unreasonable amount of time, consult GCP documentation for next step.

$ gcloud compute instances list --project=$PROJECT
NAME                                          ZONE           MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS
batches--sourcegraph-executor-rqfs            us-central1-a  c2-standard-8  true         10.0.1.58    35.222.34.224   RUNNING
default-red-instance                          us-central1-a  n2-standard-8               10.2.0.3                     RUNNING
sourcegraph-executors-docker-registry-mirror  us-central1-a  n1-standard-2               10.0.1.2     35.239.105.148  RUNNING

If above all check out, visit the Compute Engine Console and check logs of the executor instance for more troubleshooting.