k-modes is used for clustering categorical variables. It defines clusters based on the number of matching categories between data points. (This is in contrast to the more well-known k-means algorithm, which clusters numerical data based on distant measures like Euclidean distance etc.) The k-prototypes algorithm combines k-modes and k-means and…

What is the Tungsten Project? Tungsten is the codename for the umbrella project to make changes to Apache Spark’s execution engine that focuses on substantially improving the efficiency of memory and CPU for Spark applications, to push performance closer to the limits of modern hardware. Tungsten Project Includes These Initiatives: Memory Management and Binary Processing: leveraging application semantics to manage…

There are two type of Data Loading Full Data Loading: Incremental Data Loading: But, Why? Its Faster, much much faster Why Not? More likely to create error. Can’t rerun with errors. Timing can be troublesome. Processs Watermark: Prepare the data store, to store the watermark. (a) Select the watermark column (b) Store the watermark value…

All there are plenty of differences the way MapReduce and Spark works but we are going to see some of them Lets start In case of hadoop mapreduce(MR) data is written to HDFS in each phase (map, shuffle, reduce). This is very expensive operation but Spark does not store anything…

When we submit a spark application using spark-submit script spark-submit –master <Spark master URL> --driver-memory 1g --executor-memory 1g --executor-cores 2 --py-files file1.py,file2.py,file3.zip,file4.egg wordByExample.py Step by step process ignition: It creates a Driver Program in any of the edge nodes or data node: Edge Node are the nodes where we submit…

All though Memory Management is a fairly vast concept and there are many ways we try to mitigate it but we would talk about it in very brief, we would try to understand the ways Spark deals with it internally for garbage collection. Garbage collection tuning Some basic information about memory management in…

1 — Download and extract Kafka Run the commands below on the newly opened terminal. (You can copy the code by clicking on the little copy button on the bottom right of the code block below and then paste it, wherever you wish.) Download Kafka, by running the command below: wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.12-2.8.0.tgz Extract Kafka from the…

Objectives Use message keys to keep message streams sorted in their original publication state/order Use consumer offset to control and track message sequential positions in topic partitions Start ZooKeeper ZooKeeper is required for Kafka to work. Start the ZooKeeper server. #For linux/MacOS cd kafka_2.13-3.1.0 bin/zookeeper-server-start.sh config/zookeeper.properties #For windows system D:\software\kafka_2.13-3.1.0\bin\windows\zookeeper-server-start.bat kafka_2.13-3.1.0\config\zookeeper.properties

Install docker composer from here To deploy Airflow on Docker Compose, you should fetch docker-compose.yaml. mkdir ~/airflow-docker cd airflow-docker curl -LfO "https://airflow.apache.org/docs/apache-airflow/2.2.3/docker-compose.yaml" This file contains several service definitions: airflow-scheduler - The scheduler monitors all tasks and DAGs, then triggers the task instances once their dependencies are complete.