➡️ This architecture enables customers to build end-to-end modern data analytics platforms using AWS and Snowflake.
1) Data is collected from multiple data sources across the enterprise, software as a service (SaaS) applications, edge devices, logs, streaming data, and social media networks.
2) Based on the type of data source, AWS Database Migration Service, AWS DataSync, Amazon Kinesis, Amazon Managed Streaming for Apache Kafka, AWS IoT Core, AWS Glue and Amazon AppFlow are used to ingest the data into the data lake in AWS.
3) Amazon S3 is used for fully managed, highly available and scalable data lake storage.
4) AWS Glue is used to extract, transform and ingest data across multiple data stores. Amazon EMR provides the cloud big data platform for processing vast amounts of data using open source analytics framework. AWS Lambda and Amazon EC2 provide compute capability for data enrichment needs.
5) Amazon Managed Workflows for Apache Airflow (MWAA ) or AWS Step Functions is used for orchestrating end-to-end data pipelines.
6) AWS Lake Formation makes it easy to build, secure and manage your data lake, providing single place to enforce data classification and manage fine-grained access. AWS IAM and AWS STS provides ability to manage access permissions and temporary credentials.
7) Snowflake is used as virtual data warehouse with ability to query Amazon S3 using external tables, and automated and continuous data ingestion using SnowPipe.
8) Amazon SageMaker can be used to build, train, and deploy machine learning (ML) models, and add intelligence to your Applications. Amazon QuickSight provides ML-powered business intelligence (BI).
You can download the Implementation Guide from this link 👀👇 https://d1.awsstatic.com/architecture-diagrams/ArchitectureDiagrams/data-platform-on-aws-and-snowflake-ra.pdf?did=wp_card&trk=wp_card
I hope you will find it useful for your organization! 🤓
[IMPORTANT]
- This material is not mine. This material was written exclusively by Amazon Web Services.
I am sharing this with a bigger IT audience to help folks in their quest to learn AWS. I don't intend to monetize anything, therefore everything I share will always be free. I'm not interested in profiting off the knowledge of others.
Sharing relevant and helpful content is something I've always thought is a great approach to support the tech community. I'll keep offering my viewers insightful content. Thank you for supporting.