Sending Alert Text Messages Based on CloudWatch Log Patterns
By Hyuntaek Park
Senior full-stack engineer at Twigfarm
Introduction
Monitoring or getting an alert when unexpected happens is essential to keep our system reliable. System faults should be notified to appropriate person as early as it can be. This can be achieved easily with CloudWatch.
In this post, I create a simple lambda function and a couple of tests with a normal condition and one with an error condition. Once the error condition happens SMS text message is sent to my cell number all the way from North Virginia (us-east-1) to Seoul, Korea (ap-northeast-2).
The idea is simple:
- Exception happens during lambda execution
- Logging error in the CloudWatch associated with that lambda
- CloudWatch invokes another lambda function which sends an SMS text message if a predefined expression is in the log text
Creating a simple lambda function
Go to Lambda Functions and click Create function. Then create a simple function with any name you like.
The lambda code does simple division. Our goal is to get notified if DivideByZeroException occurs.
Test input (OK case)
Logs for OK test case
Test input (Error case)
Logs for error test case
SMS sender lambda function
Let’s create another lambda function, called sms-sender, which gets triggered if the CloudWatch log contains certain pattern. For now this lambda function does nothing but prints out the input so that we are able to identify what the CloudWatch input format is like.
sms-sender
CloudWatch Subscription Filters
Now CloudWatch Subscription Filter connects following two lambda functions:
- cloudwatch-alert-test
- sms-sender
Go to CloudWatch –> Logs –> Log groups –> /aws/lambda/cloudwatch-alert-test
Then click on Subscription filters tab then choose Create Lambda subscription filter
Input as following:
- Lambda function: sms-sender
- Log format: Other
- Subscription filter pattern: DivideByZeroException
- Subscription filter name: ANY_NAME_YOU_LIKE
Click on Start streaming button to finish CloudWatch subscription filter setup.
This means that sms-sender lambda function gets invoked, if and only if the CloudWatch log contains string DivideByZeroException.
Testings
We created two tests for cloudwatch-alert-test. Run each of them and check the CloudWatch log for sms-sender to see if the function is invoked only when DivideByZeroException is happened.
Input passed from CloudWatch to sms-sender
What do you see in sms-sender CloudWatch log? Was the function triggered only on error test case? Probably yes. Now take a look at the event parameter value in the log. You would see some data value like the following:
What is it? We have to unzip to decode the data.
Decoding input from CloudWatch
Following code takes care of unzipping the input from CloudWatch.
Variable stringResult shows stringified version of the input, which looks like:
If your alert method requirement is webhook notifications, then you can just add a few lines of code for HTTP POST call in the lambda function. End of story.
Since SMS text message is a means of alert requirement, one more setup is required.
Setting Amazon Simple Notification Service (SNS) for SMS text messaging
As of December 2021, mobile text messaging (SMS) feature is not supported in Seoul region (ap-northeast-2). So I changed my region to complete text messaging setup in SNS.
Supported Regions are listed here: https://docs.aws.amazon.com/sns/latest/dg/sns-supported-regions-countries.html
After changing region to us-east-1, go to Amazon SNS –> Mobile –> Text messaging (SMS). If Text messaging (SMS) is now showing in your region, it is likely that your region is not supporting this feature.
Under the Sandbox destination phone numbers, click Add phone number for adding recipients’ phone numbers. By default, you are in a sandbox mode which has a few restrictions such as:
- Recipients’ phone numbers have to be verified
- 1 USD is the maximum you can spend. (it suddenly stops sending messages after reaching the limit)
Note: you can always request limit increase here: https://aws.amazon.com/premiumsupport/knowledge-center/sns-sms-spending-limit-increase/
AWS support will get you out of sandbox as well upon the limit request.
After you add a recipient’s phone number for testing, now let’s go back to sms-sender lambda function in order to add some code for text messaging.
Publishing text message code in sms-sender
Following is the complete lambda code for sms-sender, in which message sending feature is added.
Almost done. One last adjustment is left. Our lambda role needs a permission to send text messages.
IAM role for text messaging
Let’s move to IAM role for sms-sender lambda function.
In the lambda role, click Attach policies –> check AmazonSNSFullAccess –> click Attach policy button.
Final testing
Now all of our components are set up and ready. Go back to cloudwatch-alert-test lambda function and run error test case. Wait a few seconds, you should receive a text message if you set up everything correctly. You can modify the message to any way that you like to. Note that if your message is longer than 160 characters, they are split into separate messages.
Conclusion
You can design your alert system in various ways by applying different log patterns without much hassle utilizing CloudWatch subscription filters.
Getting alerts if something goes wrong is very useful. But it might be annoying if our phones vibrate too frequently than it should be. Yes, it is very tricky to get the balance right. I know DivideByZeroException is not an appropriate case to get SMS text messaging alert :)
Each system has different reliability requirements. When to fire alarm should be designed carefully according to the requirements. Avoiding false alarms is crucial point to consider as well.
Happy Christmas and to our reliable systems!
