Spot Blocks with Autoscaling groups

Steps to create Spot Blocks with Autoscaling groups to handle Spot unavailability:-

1. Create a launch template with a defined AMI, Instance type required for spot block, Security group defined in the Network Interfaces and ‘Request Spot block’ selected with the Block duration defined.

1. Use this template for Scheduled spot Autoscaling group to create the Scheduled Actions, keeping On-Demand percentage to be 0%

1. Enable Group Metrics collection from Monitoring Tab of this Autoscaling group(You should see below metrics in this tab after enabling):-

1. Create the On-Demand Autoscaling group keeping on-demand % as 100% with Minimum size depending on how much minimum instance you want to keep in this group and define the instances required based on the priority.

1. Goto Cloudwatch metrics and select Autoscaling-Group metrics and select the two metrics, GroupDesiredCapacity and GroupInServiceInstances for the Scheduled-Spot-Block group and move to Graphed Metrics Tab.

1. Click on Math Expression and use Start with empty expression and in the new metric row use the id’s to define a difference. Here I am using m1-m2 to get the diff between desired and in-service.

1. Now, click on the new expression row’s Alarm action which will bring up Alarm console and configure the alarm to have a threshold of greater than or equal to 1

1. Goto On-Demand Autoscaling group and then in the Scaling policies add the Increase group policy like below:-

1. Now, for This on-demand group Scale Down policy we should not do any scale down policy through metrics for your use case, rather we should use Scheduled action to reduce the capacity to ‘0’ by changing desired count at a certain time when the original block duration was supposed to finish.
2. To reduce the Desired count on the Scheduled-Spot-Block group if On-demand instances are launched we can use a Cloudwatch event for this on-demand group and each time an event is received we should just decrease the desired count of the Scheduled-Spot-Block group by 1, so that once on-demand instances are launched then we don’t launch spot blocks to save cost as on-demand is already up.

Below will be the cloudwatch rule:-

The lambda will just be a boto3 code which first calls the ‘describe autoscaling group’ and gets current desired count and then sends another call ‘set_desired_capacity’ to reduce it by 1

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/autoscaling.html#AutoScaling.Client.describe_auto_scaling_groups

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/autoscaling.html#AutoScaling.Client.set_desired_capacity