Start Instances

The GorillaStack Start Instances action interacts with your EC2 Instances using the DescribeInstances and StartInstances API actions.

Starting Instances with Attached Encrypted EBS Volumes

It is possible for an EC2 Instance to remain in a Stopped state even if it is successfully targeted by the Start Instances Action. The StartInstances API, which is used by this action, can silently fail to start EC2 Instances that have attached encrypted EBS Volumes.

Why Does this Happen?

The AWS Key Management Service (KMS) is leveraged to perform encryption on EBS Volumes.

Each KMS Customer Master Key (CMK) has a Key Policy which defines who has permission to use the CMK and what API actions a privileged entity can perform using the CMK.

The default CMK Key Policy on a non AWS managed Key does not grant the EC2 service the privileges it requires to attach an encrypted EBS Volume to an EC2 Instance as part of a StartInstances operation.

How to Remedy the Issue?

  1. Determine which KMS CMKs are used to encrypt each EBS Volume attached to EC2 Instances which you are trying to start
  2. Update each CMK's Key Policy, ensuring principal of least privilege is observed by utilizing the kms:ViaService condition (see example below)
From the AWS KMS Best Practices Whitepaper
{
  "Sid": "Allow for EC2 Use",
  "Effect": "Allow",
  "Principal": {
    "AWS": "arn:aws:iam::111122223333:root"
  },
  "Action": [
    "kms:CreateGrant",
    "kms:ListGrants",
    "kms:RevokeGrant"
  ],
  "Resource": "*",
  "Condition": {
    "StringEquals": {
      "kms:ViaService": "ec2.<<region>>.amazonaws.com"
    }
  }
}

The statement in this policy provides the specified IAM principal the ability to create, list, and revoke grants for Amazon EC2. Grants are used to delegate a subset of permissions to AWS services, or other principals, so that they can use your keys on your behalf. In this case, the condition policy explicitly ensures that only Amazon EC2 can use the grants. Amazon EC2 will use them to re-attach an encrypted EBS volume back to an instance if the volume gets detached due to a planned or unplanned outage. These events will be recorded within AWS CloudTrail when, and if, they do occur for your auditing.