Requirements of Spark

Let's understand the functional and non-functional requirements of Spark.

Functional requirements

The functional requirements of Spark are listed below:

• Data processing: The system needs to process a large working dataset efficiently and also be able to do it repeatedly for iterative or interactive queries.

• Latency and throughput: Our system should achieve low latency and high throughput for the tasks, like iterative data processing (where we use the same data repeatedly) and performing ad hoc queries on the same dataset. For example, we expect our system to query many terabytes of data in a few seconds. Usually, the first run is slower than the others because our system needs to load the data from the disks that involve IO operations.

Non-functional requirements

Following are the non-functional requirements of Spark.

• Fault tolerance: If a data partitionFor parallel processing data is divided into partitions called data partitions. is lost, it should be recovered effectively.

• Data locality: The system should do computations on the worker where the data resides.

• Persistent in-memory data: The data that needs ...