Have you worked with AWS Glue, and if so, can you describe a use case for it in your previous projects?
Yes. I do have experience with AWS Glue. Glue is basically a fully managed ETL service that makes it easy for users to prepare and load their data for analytics.
Here is how we used Glue in one of my previous projects: Our system deals with a high volume of transactional data, real-time stock prices, historical market data, and other related financial datasets from various sources including relational databases, log files, and streaming data, all residing in different locations, such as on-premises servers and various AWS services.
Problem: We had to create a unified data catalog where analysts could see all available data, understand its structure, and run analytics without worrying about the underlying infrastructure. Also, we required an automated way to transform and move this data into our central data lake on Amazon S3.
Solution : So, we decided to use AWS Glue, primarily because of three reasons :
- Glue provides a data catalog feature which can automatically discover and catalog datasources using crawlers.
- Our team was a mix of developers and entry level data engineers. So, there was not enough expertise to build a datalake, infrastructure and setup an EMR cluster on our own. Besides, this is a very typical ETL job which does not require much customisations to the underlying infrastructure. EMR could be an overkill.
- Our jobs only run twice a week and hence a managed serverless infrastructure could help us same time and money.
We have extensively used Glue crawlers to automatically scan the data from different sources, extract metadata, and create table definitions in the Glue Data Catalog.
Without crawlers, we would have spent a lot of time in writing the metadata extraction logic on our own from all these sources. It saved us time and avoided any potential human errors.
Also any changes in our source data will automatically be captured by the crawlers in next scan.
Analysts or Developers need not worry about the changes in source data unless there are any breaking changes. Any breaking changes should still be manually coordinated and implemented.
In addition, our analysts started using the glue catalog as a central repository for their analytical needs.
Since lot of developers are new to the data engineering , Glue's Visual ETL helped us to define parsing, transformations and cleaning routines from the console. All our jobs were written in PySpark.
Scheduling and Automation: We scheduled Glue ETL jobs to run at specific intervals, ensuring that our data lake was consistently updated with fresh data.
Querying data with Athena: Once the data was cataloged and available in S3, we utilized Amazon Athena to query the data directly from S3, and analysts could use Amazon Quicksight for visualization.
Data in Glue Catalog can be easily queried from Athena using SQL queries. We also use Athena JDBC drivers in java code to query that data for some analytical pruposes.