Monitoring your AWS Infrastructure changes

Monitoring application and infrastructure helps maintain acceptable uptime / SLA for any business. However with different Cloud providers (here AWS) and several remote teams, it becomes equally important to monitor infrastructure changes as well to identify out-of-compliance events, accelerate incident investigations, security breaches in a timely manner.

AWS CloudTrail is a service that enables governance, compliance, operational auditing, and risk auditing of your AWS account. With CloudTrail, you can log, continuously monitor, and retain account activity related to actions across your AWS infrastructure.

With several remote teams working and CloudTrail logging all of AWS account activity, including actions taken through the AWS Management Console, AWS SDKs, command line tools and other AWS services, it becomes even more important to be notified of critical events and changes. This will also accelerate incident investigations.

Here, I will discuss about Security Automation on how we can get notification if there is any changes on AWS Infrastructure. I will quote examples for notifying any changes in Security Groups and EC2 instance status like if anyone Started, Stopped, Terminated or created new instance.

Note: You must enable AWS CloudTrail for this to work.

Security Group Changes Notification

This CloudWatch rule will help you to get notified on slack, when anyone makes changes to the security group.

1. Goto CloudWatch → Events → Rules
2. Create a New Rule and Choose Event Pattern. Edit the Event Pattern Preview and Add the below block

{
  "source": [
    "aws.ec2"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "ec2.amazonaws.com"
    ],
    "eventName": [
      "AuthorizeSecurityGroupEgress",
      "AuthorizeSecurityGroupIngress",
      "RevokeSecurityGroupEgress",
      "RevokeSecurityGroupIngress"
    ]
  }
}

3. In the target, choose SNS Topic of your choice, preferably Slack or Email and in Configure Input, choose Input Transformer. Input Transformer filters the entire event based on the defined template.

Enter below text in 1st Text Block. This will define various available fields in CloudTrail event against a variable.

{"changetype":"$.detail.eventName","sgid":"$.detail.requestParameters.groupId","region":"$.detail.awsRegion","username":"$.detail.userIdentity.principalId"}

Enter below text in 2nd Text Block. This will form a filtered message using the variables to be sent to the SNS topic.

"The user <username> has initiated <changetype> for Security Group with id <sgid> in <region>."

The above event will be triggered as soon as CloudTrail detects any changes to the Security group and sends notification to the SNS Topic.

Sample Notification Event –

“The user ASFDS6F32B23IU3D32:iam.user has initiated AuthorizeSecurityGroupIngress for Security Group with id sg-0gd73rbjdhbcew in us-east-1.”

“The user ASFDS6F32B23IU3D32:iam.user has initiated RevokeSecurityGroupIngress for Security Group with id sg-0gd73rbjdhbcew in us-east-1.”

EC2 Instance Changes Notification

This CloudWatch rule will help you to get notified on slack, when anyone Start / Stop / Launch / Terminate EC2 instances.

1. Goto CloudWatch → Events → Rules
2. Create a New Rule and Choose Event Pattern. Edit the Event Pattern Preview and Add the below block

{
  "source": [
    "aws.ec2"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "ec2.amazonaws.com"
    ],
    "eventName": [
      "RunInstances",
      "StartInstances",
      "StopInstances",
      "TerminateInstances"
    ]
  }
}

3. In the target, choose SNS Topic of your choice, preferably Slack or Email and in Configure Input, choose Input Transformer. Input Transformer filters the entire event based on the defined template.

Enter below text in 1st Text Block. This will define various available fields in CloudTrail event against a variable.

{"instanceid":"$.detail.requestParameters.instancesSet","changetype":"$.detail.eventName","region":"$.detail.awsRegion","username":"$.detail.userIdentity.principalId"}

Enter below text in 2nd Text Block. This will form a filtered message using the variables to be sent to the SNS topic.

"The user <username> has initiated <changetype> for instance with instance id <instanceid> in <region>."

The above event will be triggered as soon as CloudTrail detects any changes to EC2 events like starting / stopping / launching / terminating of EC2 instances and sends notification to the SNS Topic.

Sample Notification Event –

“The user ASFDS6F32B23IU3D32:iam.user has initiated StopInstances for instance with instance id {items:[{instanceId:i-0a6asdf2345ghj}]} in us-east-1.”

“The user ASFDS6F32B23IU3D32:iam.user has initiated StartInstances for instance with instance id {items:[{instanceId:i-0a6asdf2345ghj}]} in us-east-1.”

Similarly, you can also write CloudWatch event rules to notify for critical AWS Infra changes like Route Table changes, VPC changes, S3 Policy changes, IAM changes etc.

This article has been written by Binny Oza, Principal Devops Engineer at Techpartner. For more, visit www.techpartner.in or contact us at info@techpartner.in.