The Data Engineering Lifecycle and AWS Storage Systems

AWS Certified Machine Learning Engineer – Associate · Data Preparation for Machine Learning · By Super Admin

Machine learning sits within the broader data engineering lifecycle, and choosing the right AWS storage system depends on how and when data is generated and consumed.

Machine learning is part of a broader discipline known as data engineering, and ML is one of three ways data is served within that discipline. Understanding where the lifecycle begins and how storage choices are structured is essential for designing effective ML data pipelines on AWS.

The Data Engineering Lifecycle

The data engineering lifecycle begins with generation, which focuses on where data originates. Data can be generated by a variety of source systems, such as a database, an Internet of Things (IoT) device, or a streaming system, and a data engineer designs processes to consume data from these systems but might not be able to control them. Once generated, data is ingested in its raw format into a storage system located in AWS, where it is stored and subsequently processed before moving on to analytics, machine learning, and reverse ETL.

Choosing the Right AWS Storage System

For the exam, it is necessary to ensure an appropriate data store is chosen based on the specific use case. For example, you may want to store raw data as object storage in an S3 bucket because the data is produced in batches, and this type of storage is relatively cheap compared to other means; conversely, if data is produced and consumed in near real time, you may want to store it in a streaming storage medium such as Amazon Kinesis Data Streams. AWS storage systems can be viewed hierarchically: a top layer of storage abstractions, a middle layer of commonly used storage systems and types, and a bottom layer of underlying infrastructure components, with several AWS services generally available for each storage system.

Understanding how to extract stored data and prepare it for a selected ML algorithm, used to train models and derive inferences, is also required for the exam. Choosing the most appropriate data store to collect and host data for an ML solution depends on the core AWS services available to ingest data from different sources, along with the main use cases that apply to them.

Key takeaways

  • ML is one of three ways data is served within the data engineering discipline.
  • The lifecycle progresses from generation to ingestion, storage, processing, analytics/ML, and reverse ETL.
  • Data store choice depends on use case, such as batch data in S3 versus streaming data in Kinesis Data Streams.
  • AWS storage can be viewed as layered abstractions, systems, and underlying infrastructure components.
0.0 (0 ratings)

Discussion (0)

No discussion on this article yet — be the first to comment.

Please sign in to join the discussion.