Slides and descriptions from AWS Digital Training : CloudWatch Insights by Manbeen Kohli
Fully managed, highly scalable, log analytics capabilities
Alarms Operational Visabiligy in seconds ...
Features : Works with any log sent to CloudWatch
AWS or on-premises applications
Any being sent to CloudWatch
Simple Powerful Querying
Writing Queries - Simple Query Language
sample queries, common descriptions
Fields, FIlter, Java regex
one or more log fields
numeric, string, datatype manipulation , conditional and mathematical operations
Filter : 1 or more log events java style regex
Stats : aggregation , Sort, Limit
Parse : ephemeral field creation can be used later on in the query.... in this case "@severity " field
Progamatic access : to Logs Insights : automated solutions
Example Insights Demostration :
- Open CloudWatch Console and then click "insights"
- Select log you wish to query
- Specify timeframe
- enter a query / or use a sample query
- "Busiest" . IP address that has transferred the most/maximum amount of data over the last hour :
- fields bytes, srcAddr, dstAddr, @timestamp
- automatically discovers "fields" from AWS services such as ( Route53, Lambda, CloudTrail, VPC flowlogs , any json format ) list fields under Discovered fields .
- calculate over next 5 min and sort by desc ,
- then filter by that IP address
stats sum(bytes) as mbytes by srcAddr, dstAddr, bin(5m) | sort mbytes desc | filter srcAddr='18.214.60.60"
- Now max min and avg byte transfer for all IP over 5 min intervals
stats avg(bytes), min(bytes), max(bytes) by bin(5m)
Or use Visualization to see graph of results .
- History of queries
- Export to dashboard so you can see
- CloudTrail logs ( using a sample query ) notice that "Discovered fields" from cloud trail logs
Application logs : autodiscoverd the fields that were defined by system logs int his case sent to cloudwatch . looking for error, exception or failed
- And using parse command to create a field called "severity" and then use the count(*)
Cost : $0.005 per GB in US East (N. Virginia)
- Pricing is based on the amount of ingested log data scanned for each query; you pay $0.005 per GB in US East (N. Virginia), with similar prices in the other regions.
Custom Metrics to set alarms to accelerate troubleshooting
Reference Links :
