4.11/12 Big data and functional programming

Question 1

What is a function?

Answer 1

A mapping of values from a domain to a set of values from a co-domain. Not all of the co-domains members needs to be outputs.

Question 2

What is the domain?

Answer 2

The set from which the function’s input values are chosen

Question 3

What is the co-domain?

Answer 3

The set from which the function’s output values are chosen. Not all of the co-domains members needs to be outputs.

Question 4

What is Big Data?

Answer 4

A catch-all term for data that won’t fit the usual containers, cannot be stored/processed on a single server, and that must be processed at very high speeds.

Question 5

What are the three Vs of big data?

Answer 5

• (very large) Volume (of data)
• Velocity (at which data is generated)
• Variety (of data types in the data)

Question 6

What does Volume mean?
Why is it a problem?
How is it solved?

Answer 6

what it means
- Data is too big to be stored/processed on a single server

why it’s a problem
- relational databases don’t scale well across multiple machines
- and the processing associated with the data must be split across multiple machines

how it’s solved
- Functional programming is a solution

Question 7

What does Velocity mean?

Answer 7

The data is generated and/or processed at very high speed - need to respond in seconds or milliseconds

Question 8

What does Variety mean?

Answer 8

• The data is in many forms such as structured, unstructured, text, multimedia.
• The most difficult aspect of Big Data involves its lack of structure.

Question 9

What is the most difficult aspect of Big Data? Why?

Answer 9

Its lack of structure (under Variety). This poses challenges because:

• Analysing the data is made significantly more difficult
• Relational databases are not appropriate because they require data to fit into a row-and-column format

Question 10

What technique is used to discern patterns in data and to extract useful information?

Answer 10

Machine learning

Question 11

What is the advantage of functional programming for big data?

Answer 11

Its features make it easier to write
- Correct code
- Code that can be distributed to run across more than one server

Question 12

4 features of functional programming that make it suitable for Big Data

Answer 12

1. Immutable data structures
2. Statelessness
3. Higher-order functions
4. Programs do not specify order of execution (meaning they work well on parallel processing systems)

Question 13

What about immutable data structures makes them suitable for Big Data?

Answer 13

• Immutable data structures cannot be changed during program execution
• Same input always gives same output
• Makes parallel processing extremely easy

Question 14

What about statelessness makes it suitable for Big Data?

Answer 14

• Statelessness means there are no side-effects of computations
• so code is easy to write correctly, and it is easy to understand and predict how the program will behave

Question 15

What about higher order functions makes them suitable for Big Data?

Answer 15

• Higher-order functions take a function as an argument, return a function as a result, or both.
• Higher-order functions can be easily parallelised

Question 16

What is a fact in a fact-based model?

Answer 16

• Each fact within a fact based model captures a single piece of information.
• Each fact is immutable and timestamped

Question 17

What is a graph schema?

Answer 17

• Graph schemas can be used to capture the structure of a dataset. They can be easily extended, without impacting existing facts (because the facts are immutable)
• Nodes are used to represents the core entities in the data set
• Edges are used to represent the relationships between the nodes
• Properties are used to capture information about the nodes

Question 18

What is a first class object?

Answer 18

First class objects are objects which may:
- R - be returned in function calls
- A - be assigned as arguments
- V - be assigned to a variable
- E - appear in expressions

Functions are first-class objects in functional programming languages

Question 19

What does function application mean?

Answer 19

Applying a function to its arguments

Question 20

What does partial function application mean?

Answer 20

Parțial function application means only applying a function to some of its arguments. The result is a function.

Question 21

What is functional composition?

Answer 21

• Combining two functions to get a new function
• g*f means apply f first, then g

Question 22

Describe in words what map does

Answer 22

Applies a given function to each element of a list, returning a list of results

Question 23

Describe in words what filter does

Answer 23

Processes a list to produce a new list containing exactly those elements that match a given condition

Question 24

Describe in words what reduce or fold does

Answer 24

Reduces a list of values to a single value by repeatedly applying a combining function to the list values

Question 25

How is machine learning used in Big Data?

Answer 25

Machine learning is used to discern patterns in data and to extract useful information

Question 26

What is meant by "parallel processing" in Big Data?

Answer 26

When more than one processor can work on different parts of a large data set at the same time without changing any other part