AI Sciences

Why Big Data

Applications of PySpark

Introduction to Instructor

Introduction to Course

Projects Overview

Why Spark

Hadoop EcoSystem

Spark Architecture and EcoSystem

DataBricks SignUp

Create DataBricks Notebook

Download Spark and Dependencies

Java Setup on Window

Python Setup on Window

Spark Setup on Window

Hadoop Setup on Window

Runing Spark on Window

Java Download on MAC

Installing JDK on MAC

Setting Java Home on MAC

Java check on MAC

Installing Python on MAC

Setup Spark on MAC

Spark RDDs

Creating Spark RDD

Running Spark Code Locally

RDD Map (Lambda)

RDD Map (Simple Function)

Quiz (Map)

Solution 1 (Map)

Solution 2 (Map)

RDD FlatMap

RDD Filter

Quiz (Filter)

Solution (Filter)

RDD Distinct

RDD GroupByKey

RDD ReduceByKey

Quiz (Word Count)

Solution (Word Count)

RDD (Count and CountByValue)

RDD (saveAsTextFile)

RDD (Partition)

Finding Average-1

Finding Average-2

Quiz (Average)

Solution (Average)

Finding Min and Max

Quiz (Min and Max)

Solution (Min and Max)

Project Overview

Total Students

Total Marks by Male and Female Student

Total Passed and Failed Students

Total Enrollments per Course

Total Marks per Course

Average marks per Course

Finding Minimum and Maximum marks

Average Age of Male and Female Students

Introduction to Spark DFs

Creating Spark DFs

Spark Infer Schema

Spark Provide Schema

Create DF from Rdd

Rectifying the Error

Select DF Colums

Spark DF withColumn

Spark DF withColumnRenamed and Alias

Spark DF Filter rows

Quiz (select, withColumn, filter)

Solution (select, withColumn, filter)

Spark DF (Count, Distinct, Duplicate)

Quiz (Distinct, Duplicate)

Solution (Distinct, Duplicate)

Spark DF (sort, orderBy)

Quiz (sort, orderBy)

Solution (sort, orderBy)

Spark DF (Group By)

Spark DF (Group By - Multiple Columns and Aggregations)

Spark DF (Group By -Visualization)

Spark DF (Group By - Filtering)

Quiz (Group By)

Solution (Group By)

Quiz (Word Count)

Solution (Word Count)

Spark DF (UDFs)

Quiz (UDFs)

Solution (UDFs)

Solution (Cache and Presist)

Spark DF (DF to RDD)

Spark DF (Spark SQL)

Spark DF (Write DF)

Project Overview

Project (Count and Select)

Project (Group By)

Project (Group By, Aggregations and Order By)

Project (Filtering)

Project (UDF and WithColumn)

Project (Write)

Collaborative filtering

Utility Matrix

Explicit and Implicit Ratings

Expected Results

Dataset

Joining Dataframes

Train and Test Data

ALS model

Hyperparameter tuning and cross validation

Best model and evaluate predictions

Recommendations

Introduction to Spark Streaming

Spark Streaming with RDD

Spark Streaming Context

Spark Streaming Reading Data

Spark Streaming Cluster Restart

Spark Streaming RDD Transformations

Spark Streaming DF

Spark Streaming Display

Spark Streaming DF Aggregations