Build Operational Analytics Pipeline on AWS Modern Data Architecture

Build Operational Analytics Pipeline on AWS Modern Data Architecture

ยท

2 min read

โžก๏ธ This architecture enables customers to perform operational analytics in batch and real-time using log information from operational data sources. Operational insights are visualized using an OpenSearch Dashboard and alerts are distributed using Amazon Simple Notification Service (Amazon SNS).

2022-07-30 (20).png

1) Logs are streamed to Amazon CloudWatch from various data sources.

2) Amazon CloudWatch subscription filters stream log data to Amazon Kinesis Firehose for distribution.

3) Logs intended for Batch Processing are delivered to an Amazon Simple Storage Service (Amazon S3) bucket. It will serve as a back up for storing the raw logs.

4) Logs intended for Realtime Analysis are delivered to Amazon Kinesis Data Analytics.

5) Amazon Kinesis Data Analytics derives metrics, which are sent to a Lambda function for processing. The Lambda function transforms the metrics and ingests them into Amazon OpenSearch. If a condition is met that requires alerting, the Lambda function will publish a message to the Amazon Simple Notification Service (Amazon SNS) for distribution.

6) Amazon Simple Notification Service notifies consumers of the alerts by publishing the alert to an Email and SMS notification topics.

7) AWS Glue collects the logs from Amazon S3 and performs bulk processing, transformation and analysis. AWS Glue uses the Amazon OpenSearch connector to ingest the results into Amazon OpenSearch. 8) An OpenSearch Dashboard is used to visualize the metrics from Amazon OpenSearch.

You can download the Implementation Guide from this link ๐Ÿ‘€๐Ÿ‘‡ https://d1.awsstatic.com/architecture-diagrams/ArchitectureDiagrams/build-operational-analytics-pipeline-on-AWS-modern-data-architecture.pdf?did=wp_card&trk=wp_card

I hope you will find it useful for your organization! ๐Ÿค“

[IMPORTANT]

  • This material is not mine. This material was written exclusively by Amazon Web Services.

I am sharing this with a bigger IT audience to help folks in their quest to learn AWS. I don't intend to monetize anything, therefore everything I share will always be free. I'm not interested in profiting off the knowledge of others.

Sharing relevant and helpful content is something I've always thought is a great approach to support the tech community. I'll keep offering my viewers insightful content. Thank you for supporting.

Did you find this article valuable?

Support AditModi's Blog by becoming a sponsor. Any amount is appreciated!

ย