4.0 CSV to Parquet File Conversion Example:

Below is an example of setting up a pipeline to process CSV files (In this example, we have taken an audit file that is arriving in the On Prem Hadoop Cluster on a daily basis) and we are converting them to parquet after an ETL process using Databricks Cloud Solution. Oozie is used to orchestrate this pipeline by detecting when daily files are ready for processing and and sending a final email notification.