{"id":177334,"date":"2024-01-21T17:58:24","date_gmt":"2024-01-21T16:58:24","guid":{"rendered":"https:\/\/liora.io\/en\/?p=177334"},"modified":"2026-08-08T13:17:29","modified_gmt":"2026-08-08T12:17:29","slug":"aws-glue-what-is-it-whats-it-for","status":"publish","type":"post","link":"https:\/\/liora.io\/en\/aws-glue-what-is-it-whats-it-for","title":{"rendered":"AWS Glue: What is it? What&#8217;s it for?"},"content":{"rendered":"\n<p><strong>AWS Glue is a fully managed, scalable data processing service that enables users to run serverless ETL (Extract, Transform, Load) workflows, freeing them from the need to manage the underlying infrastructure.<\/strong><\/p>\n\n\n<h2 class=\"wp-block-heading\" id=\"a-reminder-about-etl-processes\">A reminder about ETL processes<\/h2>\n\n\n<p><strong>ETL<\/strong> is a process designed to guarantee data quality and availability. It is divided into 3 phases:<\/p>\n\n\n<ul class=\"wp-block-list\"><li><strong>Extraction<\/strong>: recovery of data from various sources<\/li><li><strong>Transformation:<\/strong> <a href=\"https:\/\/liora.io\/en\/data-cleaning-definition-methods-and-relevance-in-data-science\">cleansing, normalizing and modifying data to make it usable<\/a><\/li><li><strong>Loading:<\/strong> loading transformed data into a final environment, such as a database or <a href=\"https:\/\/liora.io\/en\/data-warehouse-2\">data warehouse.<\/a><\/li><\/ul>\n\n\n<figure class=\"wp-block-image size-full\" style=\"margin-top:32px;margin-bottom:32px\"><img alt=\"Source : Informatica.com\" decoding=\"async\" height=\"527\" loading=\"lazy\" src=\"https:\/\/liora.io\/app\/uploads\/2023\/04\/image3-5-e1681996722906.png\" style=\"width:100%;height:auto\" width=\"1080\"\/><figcaption class=\"wp-element-caption\">Source : Informatica.com<\/figcaption><\/figure>\n\n\n<h2 class=\"wp-block-heading\" id=\"how-is-aws-glue-structured\">How is AWS Glue structured?<\/h2>\n\n\n<p><strong>AWS Glu<\/strong>e jobs perform the necessary extraction, transformation and loading of data from a source to a destination. The following diagram shows the architecture of <strong>AWS Glue,<\/strong> and then we describe the various elements:<\/p>\n\n\n<figure class=\"wp-block-image size-full\" style=\"margin-top:32px;margin-bottom:32px\"><img alt=\"Illustration for How is AWS Glue structured?\" decoding=\"async\" height=\"576\" loading=\"lazy\" src=\"https:\/\/liora.io\/app\/uploads\/2023\/04\/image1-7.png\" style=\"width:100%;height:auto\" width=\"768\"\/><\/figure>\n\n\n<ul class=\"wp-block-list\"><li><strong>Data Catalog:<\/strong> this is the permanent metadata storage in<strong> AWS Glue<\/strong>. It contains table definitions, job definitions, etc.<\/li><li><strong>Database:<\/strong> a set of table definitions for associated data catalogs.<\/li><li><strong>Crawler:<\/strong> a program that connects to a data source to extract its data and determine its structure. It then uses this information to create table definitions in the data catalog.<\/li><li><strong>Connection:<\/strong> this<strong> AWS Glue<\/strong> connection is the data catalog that contains the information needed to connect to a certain data store.<\/li><li><strong>Classifier:<\/strong> determines the data schema.<strong> AWS Glue<\/strong> provides classifiers for the most common file types, such as CSV, Json, etc.<\/li><li><strong>Data store<\/strong>: repository for persistent data storage.<\/li><li><strong>Data source<\/strong>: this is the entry point used for the transformation process.<\/li><li><strong>Data target<\/strong>: the target to which the transformed data will be written.<\/li><li><strong>Job:<\/strong> the business logic required for ETL jobs, made up of the various elements required.<\/li><\/ul>\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex is-content-justification-center wp-container-core-buttons-is-layout-5ee10de4\" style=\"margin-top:32px;margin-bottom:32px\"><div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"\/en\/courses\/cloud-dev\/aws-solutions-architect\">Learn how to use AWS Glue<\/a><\/div><\/div>\n\n\n<h2 class=\"wp-block-heading\" id=\"aws-glue-features\">AWS Glue features <\/h2>\n\n\n<p><strong>AWS Glue<\/strong> allows you to fully manage your <strong>ETL processes<\/strong> through a variety of features, the most important of which are listed below:<\/p>\n\n\n<p>.tg  {border-collapse:collapse;border-spacing:0;}\n.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;\n  overflow:hidden;padding:10px 5px;word-break:normal;}\n.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;\n  font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}\n.tg .tg-yj5y{background-color:#efefef;border-color:inherit;text-align:center;vertical-align:top}\n.tg .tg-dvid{background-color:#efefef;border-color:inherit;font-weight:bold;text-align:left;vertical-align:top}\n.tg .tg-y698{background-color:#efefef;border-color:inherit;text-align:left;vertical-align:top}<\/p>\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table>\n<colgroup>\n<col\/>\n<col\/>\n<col\/>\n<\/colgroup>\n<thead>\n<tr>\n<th><img alt=\"Illustration for AWS Glue features\" decoding=\"async\" height=\"100\" loading=\"lazy\" src=\"https:\/\/liora.io\/app\/uploads\/2023\/04\/image7-2.png\" width=\"100\"\/><\/th>\n<th>Data Collection and Integration<\/th>\n<th>AWS Glue allows for the collection and integration of data from various sources, including databases, flat files, streaming data, etc.<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td><img alt=\"Illustration for AWS Glue features\" decoding=\"async\" height=\"93\" loading=\"lazy\" src=\"https:\/\/liora.io\/app\/uploads\/2023\/04\/image9-1.png\" width=\"100\"\/><\/td>\n<td>Data Transformation<\/td>\n<td>Provides a set of tools for transforming data, including data processing functions, filtering, sorting, joining, and more.<\/td>\n<\/tr>\n<tr>\n<td><img alt=\"Illustration for AWS Glue features\" decoding=\"async\" height=\"100\" loading=\"lazy\" src=\"https:\/\/liora.io\/app\/uploads\/2023\/04\/image5-4.png\" width=\"100\"\/><\/td>\n<td>Data Catalog<\/td>\n<td>Allows for the creation and management of a metadata catalog that facilitates data discovery, search, and analysis.<\/td>\n<\/tr>\n<tr>\n<td><img alt=\"Illustration for AWS Glue features\" decoding=\"async\" height=\"100\" loading=\"lazy\" src=\"https:\/\/liora.io\/app\/uploads\/2023\/04\/image11-1.png\" width=\"100\"\/><\/td>\n<td>ETL Task Execution and Scheduling<\/td>\n<td>AWS Glue enables the scheduling and execution of ETL tasks to process data at scale.<\/td>\n<\/tr>\n<tr>\n<td><img alt=\"Illustration for AWS Glue features\" decoding=\"async\" height=\"100\" loading=\"lazy\" src=\"https:\/\/liora.io\/app\/uploads\/2023\/04\/image4-7.png\" width=\"100\"\/><\/td>\n<td>Workflow Automation<\/td>\n<td>Offers workflow automation features to orchestrate complex tasks involving multiple steps.<\/td>\n<\/tr>\n<tr>\n<td><img alt=\"Illustration for AWS Glue features\" decoding=\"async\" height=\"100\" loading=\"lazy\" src=\"https:\/\/liora.io\/app\/uploads\/2023\/04\/image8-2.png\" width=\"100\"\/><\/td>\n<td>Custom Jobs<\/td>\n<td>Enables the creation of custom jobs to address specific use cases. Custom jobs can be created using common programming languages such as Python and Scala.<\/td>\n<\/tr>\n<tr>\n<td><img alt=\"Illustration for AWS Glue features\" decoding=\"async\" height=\"99\" loading=\"lazy\" src=\"https:\/\/liora.io\/app\/uploads\/2023\/04\/image2-9.png\" width=\"100\"\/><\/td>\n<td>Error Handling<\/td>\n<td>Allows for the management of errors encountered during data processing, such as syntax errors or connectivity issues.<\/td>\n<\/tr>\n<tr>\n<td><img alt=\"Illustration for AWS Glue features\" decoding=\"async\" height=\"100\" loading=\"lazy\" src=\"https:\/\/liora.io\/app\/uploads\/2023\/04\/image6-5.png\" width=\"100\"\/><\/td>\n<td>Monitoring<\/td>\n<td>AWS Glue provides monitoring features to track ETL job performance, detect errors and performance issues, and optimize resource utilization.<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n<h2 class=\"wp-block-heading\" id=\"advantages-and-disadvantages-of-aws-glue\">Advantages and disadvantages of AWS Glue<\/h2>\n\n\n<p>Before embarking on using and learning<strong> AWS Glue,<\/strong> it&#8217;s important to consider both its advantages and disadvantages:<\/p>\n\n\n<p>.tg  {border-collapse:collapse;border-spacing:0;}\n.tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;\n  overflow:hidden;padding:10px 5px;word-break:normal;}\n.tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px;\n  font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;}\n.tg .tg-d78e{background-color:#9aff99;text-align:center;vertical-align:top}\n.tg .tg-cmwg{background-color:#ffccc9;text-align:center;vertical-align:top}\n.tg .tg-s47q{background-color:#9aff99;font-size:20px;font-weight:bold;text-align:center;vertical-align:top}\n.tg .tg-rjvs{background-color:#ffccc9;font-size:20px;font-weight:bold;text-align:center;vertical-align:top}<\/p>\n\n\n<figure class=\"wp-block-table is-style-stripes\"><table>\n<colgroup>\n<col\/>\n<col\/>\n<\/colgroup>\n<thead>\n<tr>\n<th>Advantages<\/th>\n<th>Disadvantages<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>Large-scale data management<\/td>\n<td>High costs for small businesses or small-scale projects, despite being a fully managed service<\/td>\n<\/tr>\n<tr>\n<td>Fast data processing<\/td>\n<td>Steep learning curve<\/td>\n<\/tr>\n<tr>\n<td>Integration with other AWS services<\/td>\n<td>Limited workflow customizations<\/td>\n<\/tr>\n<tr>\n<td>Support for multiple programming languages<\/td>\n<td>Requires expertise in data engineering<\/td>\n<\/tr>\n<tr>\n<td>Fully managed platform<\/td>\n<td rowspan=\"2\"><\/td>\n<\/tr>\n<tr>\n<td>Built-in metadata catalog<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n<h2 class=\"wp-block-heading\" id=\"conclusion\">Conclusion<\/h2>\n\n\n<p>As you&#8217;ve probably gathered by now, AWS Glue is a fully managed <a href=\"https:\/\/liora.io\/en\/etl-or-extract-transform-load-definition-and-use\">Amazon AWS ETL workflow service.<\/a> Its great power and flexibility nevertheless require a steep learning curve and a very substantial investment in order to set it up to meet the required needs.<\/p>\n\n\n<div class=\"wp-block-group has-background has-global-padding is-layout-constrained wp-container-core-group-is-layout-7441ce17 wp-block-group-is-layout-constrained\" style=\"border-left-color:#ff5c43;border-left-width:4px;border-radius:12px;background-color:#fff5f2;margin-top:32px;margin-bottom:32px;padding-top:24px;padding-right:28px;padding-bottom:24px;padding-left:28px\">\n\n<p style=\"margin-top:0;margin-bottom:16px;font-size:clamp(14.642px, 0.915rem + ((1vw - 3.2px) * 0.575), 22px);font-style:normal;font-weight:600\">Related articles<\/p>\n\n\n<ul class=\"wp-block-list\" style=\"margin-top:0;margin-bottom:0;padding-left:20px\">\n<li><a href=\"https:\/\/liora.io\/en\/aws-elastic-load-balancer-the-solution-that-distributes-network-traffic\" rel=\"noopener\" target=\"_blank\">AWS Elastic Load Balancer: The solution that distributes network traffic<\/a><\/li>\n<li><a href=\"https:\/\/liora.io\/en\/jam-aws-the-playful-learning-platform-from-amazon\" rel=\"noopener\" target=\"_blank\">Jam AWS: The playful Amazon learning platform<\/a><\/li>\n<li><a href=\"https:\/\/liora.io\/en\/aws-lambda-introduction-to-the-serverless-function\" rel=\"noopener\" target=\"_blank\">AWS Lambda: Introduction to the Serverless Function<\/a><\/li>\n<li><a href=\"https:\/\/liora.io\/en\/aws-certification-what-is-it-and-how-do-i-get-it\" rel=\"noopener\" target=\"_blank\">AWS Certification: What is it and how do I get it? <\/a><\/li>\n<li><a href=\"https:\/\/liora.io\/en\/aws-sagemaker-a-guide-for-using-the-platform\" rel=\"noopener\" target=\"_blank\">AWS SageMaker: A guide for using the platform<\/a><\/li>\n<li><a href=\"https:\/\/liora.io\/en\/5-aws-launches-and-announcements-making-developers-life-easy-in-2022\" rel=\"noopener\" target=\"_blank\">5 AWS launches and announcements making developers\u2019 life easy in 2022<\/a><\/li>\n<\/ul>\n\n<\/div>\n\n\n<div class=\"wp-block-buttons is-layout-flex wp-block-buttons-is-layout-flex is-content-justification-center wp-container-core-buttons-is-layout-5ee10de4\" style=\"margin-top:32px;margin-bottom:32px\"><div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"\/en\/courses\/cloud-dev\/aws-solutions-architect\">Discover the AWS Glue service<\/a><\/div><\/div>\n\n","protected":false},"excerpt":{"rendered":"<p>AWS Glue is a fully managed, scalable data processing service that enables users to run serverless ETL (Extract, Transform, Load) workflows, freeing them from the need to manage the underlying infrastructure. A reminder about ETL processes ETL is a process designed to guarantee data quality and availability. It is divided into 3 phases: Extraction: recovery [&hellip;]<\/p>\n","protected":false},"author":76,"featured_media":177351,"comment_status":"open","ping_status":"open","sticky":false,"template":"elementor_theme","format":"standard","meta":{"_acf_changed":false,"editor_notices":[],"footnotes":""},"categories":[2433],"class_list":["post-177334","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-ai"],"acf":[],"_links":{"self":[{"href":"https:\/\/liora.io\/en\/wp-json\/wp\/v2\/posts\/177334","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/liora.io\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/liora.io\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/liora.io\/en\/wp-json\/wp\/v2\/users\/76"}],"replies":[{"embeddable":true,"href":"https:\/\/liora.io\/en\/wp-json\/wp\/v2\/comments?post=177334"}],"version-history":[{"count":2,"href":"https:\/\/liora.io\/en\/wp-json\/wp\/v2\/posts\/177334\/revisions"}],"predecessor-version":[{"id":210161,"href":"https:\/\/liora.io\/en\/wp-json\/wp\/v2\/posts\/177334\/revisions\/210161"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/liora.io\/en\/wp-json\/wp\/v2\/media\/177351"}],"wp:attachment":[{"href":"https:\/\/liora.io\/en\/wp-json\/wp\/v2\/media?parent=177334"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/liora.io\/en\/wp-json\/wp\/v2\/categories?post=177334"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}