Data Engineer Things
Insights and ideas on data and engineering.
Trending Now
Latest stories
Granular Look at Left, Semi, and Anti Joins in PySpark
Granular Look at Left, Semi, and Anti Joins in PySpark
In data operations, understanding the inner-working of the various types of joins can optimize query performance and accuracy. Spark…
Understanding Snowflake Table Locks
Understanding Snowflake Table Locks
A hands-on look at table locks.
Automate Dbt Date Logic with Python — Part 2
Automate Dbt Date Logic with Python — Part 2
Simplifying Our Models and Tests From Part 1 Using Meta Config
The Inheritance Schema Design Pattern for MongoDB Data Modelling
The Inheritance Schema Design Pattern for MongoDB Data Modelling
In the world of NoSQL databases, particularly MongoDB, designing an efficient data model is crucial for optimal application performance…
How I build an ETL pipeline with AWS Glue, Lambda, and Terraform
How I build an ETL pipeline with AWS Glue, Lambda, and Terraform
A Step-by-Step Guide
Enhance your data quality tests with the dataform-assertions package
Enhance your data quality tests with the dataform-assertions package
dbt is no longer the only choice for testing data pipelines
My Data Pipeline Orchestrators Journey
My Data Pipeline Orchestrators Journey
Originally Posted at: www.junaideffendi.com
I spent 5 hours understanding more about the Delta Lake table format
I spent 5 hours understanding more about the Delta Lake table format
All insights from the paper: Delta Lake: High-Performance ACID Table Storage over Cloud Object Stores
What is something we have but don’t own and is never working when you need it.
What is something we have but don’t own and is never working when you need it.
Testing is difficult but pains could be eased with unified tooling. Here we explore the pros and cons of testing with new tools to help us
Installing (and Switching between) Different Versions of Python
Installing (and Switching between) Different Versions of Python
How to install and switch between different python versions.
How We Integrate 1000++ Hive Tables into Data Warehouse Without ETL Seamlessly
How We Integrate 1000++ Hive Tables into Data Warehouse Without ETL Seamlessly
Migrating our data warehouse to Greenplum enables us to access data from Hive in real-time, eliminate storage issue, and much more!
Why Stream Processing is a Terrible Market (Yet We Are Still Investing in It)
Why Stream Processing is a Terrible Market (Yet We Are Still Investing in It)
TL;DR: It’s a challenging market, yet it holds promising prospects.
Introduction to Apache Iceberg | PySpark
Introduction to Apache Iceberg | PySpark
The Story Behind a Data Lake
Do We Need the Lakehouse Architecture?
Do We Need the Lakehouse Architecture?
When data lakes and data warehouses are not enough.
AWS Glue: essential tips for enhancing ETL development and operations
AWS Glue: essential tips for enhancing ETL development and operations
Explore 12 essential tips for Data Engineers and ETL Developers using AWS Glue
Best Practices for Writing Maintainable and Testable Spark Code in Scala
Best Practices for Writing Maintainable and Testable Spark Code in Scala
Enhancing Scalability and Reliability Through Structured Spark Development Practices
How to think about Internal Data Products as a Data Engineer
How to think about Internal Data Products as a Data Engineer
Data Products are all the rage, but why?
A Closer Look Into Databricks’s Photon Engine
A Closer Look Into Databricks’s Photon Engine
Part 2 of Databricks’s Photon paper note: Vectorization
What the Heck is MLOps?
What the Heck is MLOps?
In summary, by the end of this blog, you should be able to explain to someone:
Memory Management in Apache Spark
Memory Management in Apache Spark
Apache Spark’s performance advantage over MapReduce is greatest in the use-cases involving repeated computations. Much of this performance…
Simple Real-time Sentiment Analysis with Apache Spark and Kafka
Simple Real-time Sentiment Analysis with Apache Spark and Kafka
It’s been a long time since my last article. Just an update, at the beginning of this year, finally I’ve officially become an Analytics…
What do you do if you encounter technical interview questions that stump you?
What do you do if you encounter technical interview questions that stump you?
We’ve all been there — you’re in the middle of a technical interview, and you get hit with a question that leaves you stumped. It’s a…
Quickstart on Open Data Lakehouse with StarRocks + S3 + Delta Lake
Quickstart on Open Data Lakehouse with StarRocks + S3 + Delta Lake
Delta Lake, the open-source storage layer on top of data lakes like S3, brings reliability and structure to your data. But querying that…