Sending Alert Text Messages Based on CloudWatch Log Patterns
By Hyuntaek Park
Senior full-stack engineer at Twigfarm
introduction
Monitoring or getting an alert when unforeseeable happens is essential to keep our system reliable. System faults should be considered to the right person as early as it can be. This can be easily explained with CloudWatch.
In this post, I create a simple Lambda function and a couple of tests with a normal condition and one with an error condition. Once the error condition happens SMS text message is sent to my cell number all the way from North Virginia (us-east-1) to Seoul, Korea (ap-northeast-2).
The idea is simple:
- Exception happens during lambda execution
- Logging error in the CloudWatch Associated with that lambda
- CloudWatch invokes another lambda function which explains an SMS text message if a negative expression is in the log text
Creating a simple lambda function
Go to Lambda Functions and click Create function. Then create a simple function with any name you like.
The lambda code does simple division. Our goal is to get achieved if divideByZeroException occurs.
Test input (OK case)
Logs for OK test case
Test input (error case)
Logs for error test case
SMS sender lambda function
Let's create another lambda function, called sms-sender, which gets triggered if the CloudWatch log contains certain patterns. For now this lambda function does nothing but prints out the input so that we are able to identify what the CloudWatch input format is like.
sms-sender
CloudWatch Subscription Filters
Now CloudWatch Subscription Filter The following two Lambda functions are connected:
- cloudwatch-alert-test
- sms-sender
Go to CloudWatch —> Logs —> Log groups —> /aws/lambda/cloudwatch-alert-test
Then click on Subscription filters Tab then choose Create Lambda subscription filter
Input as following:
- Lambda function: sms-sender
- Log format: Other
- Subscription filter pattern: DivideByZeroException
- Subscription filter name: ANY_NAME_YOU_LIKE
Click on Start streaming Button to finish CloudWatch subscription filter setup.
This means that sms-sender Lambda function gets tested, if and only if the CloudWatch log contains string divideByZeroException.
Testings
We created two tests for cloudwatch-alert-test. Run each of them and check the CloudWatch log for sms-sender to see if the function is intended only when divideByZeroException IS HAPPED
Input passed from CloudWatch to sms-sender
What do you see in sms-sender CloudWatch log? Was the function triggered only on error test case? Exactly yes. Now take a look at the event parameter value in the log. You would see some data values like the following:
What is it? We have to unzip to decode the data.
Decoding input from CloudWatch
Following code takes care of unzipping the input from CloudWatch.
Variable stringResult shows stringified version of the input, which looks like:
If your alert method requirement is webhook notifications, then you can just add a few lines of code for HTTP POST call in the lambda function. End of story.
Since SMS text message is a means of alert requirement, one more setup is required.
Setting Amazon Simple Notification Service (SNS) for SMS text messaging
As of December 2021, mobile text messaging (SMS) feature is not supported in Seoul region (ap-northeast-2). So I changed my region to complete text messaging setup in SNS.
Supported regions are listed here: https://docs.aws.amazon.com/sns/latest/dg/sns-supported-regions-countries.html
After changing region to us-east-1, go to Amazon SNS —> mobile —> Text messaging (SMS). If text messaging (SMS) is now showing in your region, it is likely that your region is not supporting this feature.
Under the Sandbox destination phone numbers, click Add phone number For adding recipes phone numbers. By default, you are in a sandbox mode which has a few problems such as:
- Recipioned phone numbers have to be verified
- 1 USD is the maximum you can spend. (it stops sending messages after supposes the limit)
Note: You can always request a limit increase here: https://aws.amazon.com/premiumsupport/knowledge-center/sns-sms-spending-limit-increase/
AWS support will get you out of sandbox as well upon the limit request.
After you add a recipient's phone number for testing, now let's go back to sms-sender Lambda function in order to add some code for text messaging.
Publishing text message code in sms-sender
Following is the complete lambda code for sms-sender, in which message sending feature is added.
Almost done. One last adjustment is left. Our lambda role requires a permission to send text messages.
IAM role for text messaging
Let's move to IAM role for sms-sender Lambda function.
In the lambda role, click Attach policies —> check AmazonSNSFullAccess —> click Attach policy buttons.
Final testing
Now all of our components are set up and ready. Go back to cloudwatch-alert-test Lambda function and run error test case. Wait a few seconds, you should receive a text message if you set everything up you can modify the message to any way that you like to. Note that if your message is longer than 160 characters, they are split into separate messages.
Conclusion
You can design your alert system in various ways by different log patterns without much longer CloudWatch subscription filters.
Getting alerts if something goes long is very useful. But it might be surprising if our phones vibrate too much than it should be. Yes, it is very tricky to get the balance right. I know divideByZeroException is not an appropriate case to get SMS text messaging alert:)
Each system has different reliability requirements. When to fire alarms should be designed according to the requirements. Imposing false alarms is a critical point to consider as well.
Happy Christmas and to our reliable systems!
