Comparing query performance in the different flavors of Amazon Redshift and Athena. — For a while now I have been thinking of performing some benchmark tests on Amazon Redshift. As laziness settles in you tend to ignore a few things and this was one of them. That changed when I started to read about the new offering from Amazon Redshift. …

A guide on how to make the RDS database highly available and load balance database requests using read and write request splitting. — For readers who have known me through my articles and book in this article, I am going to digress from my usual topics around data engineering, data science, or data analytics. …

Understanding some use cases of computer vision using Amazon Rekognition — What is computer vision? Can computers see like humans? As per Wikipedia computer vision can be defined as “Computer vision is an interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos.” Why is computer vision important in today’s world? Technically, computer vision…

Understanding the usage of data engineering and machine learning in the electric mobility world — We are living in the world of electric mobility. Globally, the adoption of electric cars and two-wheeler is steeply on the rise. Electric mobility devices rely on expensive rechargeable lithium-ion batteries for power. These batteries are integral for the fight against the bad effects of fossil fuels such as pollution…

An article on the evolution of big data architectures and the power of modern Lakehouse architecture. — In this article, we will travel through time and understand the evolution of big data architectures. We will also explore the power of modern Lakehouse architecture in greater detail. I just realized that I completed 10 years working in the field of big data. …

A guide to deploying and administering Hadoop Clusters like a smart administrator — Several years ago while I was still a developer I got chosen to be part of the prestigious Data Administration group. It was a dream come true because not many got that chance. …

A journey into the evolution of Big Data Compute Platforms like Hadoop and Spark. Sharing my perspective on where we were, where we are and where we are headed. — Over the past few years I have been part of a large number of Hadoop projects. Back in 2012–2016 the majority of our work was done using on-premises Hadoop infrastructure. The age of on premises clusters….. On a typical project we would take care of every aspect of the Big…

A guide to implement effective Kafka Clusters Design Strategies using Partitioning and Replication — At the lowest denomination level may be created with a single broker instance. Using a Kafka Producer Data Stream can be sent in form of messages to the Kafka Broker. …

Learn how to track real-time gold prices using of Apache Kafka Pandas. Plot latest prices on a Bar Chart. — Note from Towards Data Science’s editors: While we allow independent authors to publish articles in accordance with our rules and guidelines, we do not endorse each author’s contribution. You should not rely on an author’s works without seeking professional advice. See our Reader Terms for details.