Data Formats, Algorithm Compatibility, and Access Patterns for ML Ingestion

AWS Certified Machine Learning Engineer – Associate · Data Preparation for Machine Learning · Bởi Super Admin

Choosing the right data format for ML ingestion depends on algorithm compatibility and data access patterns, and AWS services can be matched to structured, semi-structured, or unstructured data.

Selecting the right data format and understanding how it aligns with algorithms and access patterns allows a machine learning engineer to optimize performance, improve scalability, and reduce processing time when ingesting and storing data on AWS.

Common Data Formats

AWS supports commonly used formats including CSV, JSON, Apache Parquet, Apache ORC, Apache Avro, and RecordIO. CSV is widely used for structured data in tabular format, while JSON is ideal for document-based, semi-structured data. Apache Parquet and Apache ORC are columnar formats designed to optimize storage read and write operations for large datasets: because column values are stored in contiguous memory locations, column-specific compression is efficient in storage space, column-specific encoding and compression techniques can be used, and queries that fetch specific column values do not need to read the entire row, improving performance. Apache Avro stores data in a row-based format and relies on schemas; when Avro data is read, the schema used when writing it is always present, allowing each data component to be written with no per-value overhead, making serialization fast and small, and facilitating use with dynamic scripting languages because data and its schema are fully self-describing. If the program reading Avro data expects a different schema, this can be resolved because both schemas are present. RecordIO is a data format used primarily by Apache MXNet, a deep learning framework, which divides data into individual chunks called records and prepends to every record its length in bytes, followed by the data.

Algorithm Compatibility and Data Access Patterns

A key driver in choosing a data format is the format supported by the ML algorithm intended for the problem, since the algorithm will be trained on the data, so compatibility between the format and the algorithm's requirements is crucial for performance and efficiency. Amazon SageMaker built-in algorithms each accept specific formats; for example, BlazingText accepts a text file with one sentence per line, DeepAR forecasting accepts JSON Lines or Parquet, Linear learner and K-means accept RecordIO-protobuf or CSV, and XGBoost accepts CSV, LibSVM, or Parquet. Another important driver is the data access pattern, which defines how producers and consumers interact with data to meet business needs and involves documenting how data is queried, stored, and retrieved. The main factors are data size, which helps determine effective data partitioning; data shape, where organizing data to match query requirements can enhance speed and scalability; and data velocity, where understanding peak query loads helps optimize partitioning for better I/O capacity.

Matching AWS Services to Data Structure

AWS services can be grouped based on whether data is structured, semi-structured, or unstructured. Structured data services include Amazon RDS, Amazon Aurora, Amazon Redshift, Amazon S3, and Amazon Athena. Semi-structured data services include Amazon DynamoDB, Amazon DocumentDB, Amazon Athena, and Amazon S3. Unstructured data services include Amazon S3, Amazon Rekognition, Amazon Transcribe, and Amazon Comprehend. Amazon S3 is the most flexible service because it can be used for all three data categories, and it can be supplemented with Amazon Athena to query data directly from S3 without formatting the data or managing infrastructure.

Key takeaways

  • Columnar formats like Parquet and ORC optimize read/write performance and compression for large datasets.
  • Avro is row-based and self-describing because it stores its schema alongside the data.
  • SageMaker built-in algorithms each require specific accepted data formats, such as RecordIO-protobuf or CSV.
  • Data access patterns are defined by data size, shape, and velocity, which guide partitioning and service choice.
  • Amazon S3 is the most flexible AWS service, supporting structured, semi-structured, and unstructured data.
0.0 (0 đánh giá)

Thảo luận / Bình luận (0)

Chưa có thảo luận nào cho bài viết này. Hãy gửi bình luận đầu tiên nhé!

Vui lòng Đăng nhập để tham gia thảo luận.