What is big data?
Data that can’t be processed or analysed using traditional processes or tools.
What are the 3 characteristics of big data?
1) Variety: Variety of different forms of information // data may lack structure
Cannot be represented by a relational database: e.g. email messages, videos, images
2) Volume: There is a lot / high volume of data (to process as one dataset) // data will not fit on one server
hundreds of terabytes: e.g. medical datasets for diagnosis, predicting disease outbreaks
3) Velocity: The data is generated/received and processed at high velocity
Data must be processed as it is received - it cannot be batched and processed later: e.g. card payment fraud detecton, recommendation systems
What are the challenges that come with big data?
- Data cannot be stored on one server / computer.
- Not possible to process data quickly enough with one computer.
- Data cannot be represented in a table // by a relational database.
- Some forms of data / unstructured data are difficult to analyse.
How are Big Data’s challenges overcome?
- Distributed database systems
- Using functional programming can help process and analyse the unstructured data: e.g. MapReduce can be used by splitting the input data into parts then executing mapper on each part, before combing results with Reduce, functional programming makes it easier to write correct code
- Using a fact-based model can manage bigger data sets better than relational models
What are the ethical issues around big data?
Area 3: Ethical and Legal Issues
- How can data be kept securely?
- Who should have access to what data?
- Will people know what data is being stored about them?
- Where should / will the data be stored // concerns relating to data being stored in other countries.
- What rights do people have in relation to data stored about them?
- Example laws (allow two examples): Computer Misuse Act, General Data Protection Regulations / GDPR / Data Protection Act, Regulation of Investigatory Powers Act / RIPA.
- Who owns data about individuals?
Features of functional programming languages that make it easier to write code that can be distributed to run across multiple servers
- Data structures are immutable meaning the values stored in data structures cannot be changed
- Programs are stateless meaning functions do not have side-effects
- Map-reduce can be used // Higher-order functions can compose the results of processing on multiple processors
- The order of execution can be determined at runtime
In the fact based model, each individual piece of information is stored as a […]
In the fact based model, each individual piece of information is stored as a fact
In the fact based model, […] is stored as a fact
In the fact based model, each individual piece of information is stored as a fact
What is stored with each fact in a fact based model?
Timestamp of the date and time at which a piece of information was recorded
Why are timestamps stored with facts?
Facts are never deleted or overwritten and multiple different values could be held for the same attribute so the timestamps allow the computer to discern which value is most recent (immutable)
Why can traditional relational databases not be used for big data?
Due to the volume of big data, there are usually several terabytes of information which are unstructured and need to be processed extremely rapidly
How to construct graph schema?
- Solid lines with writing to show relationship between two nodes
- Dashed lines between nodes and its properties
- Rectangular boxes for properties
- Ovals to represent nodes
Why are fact-based models useful for storing Big Data?
- Facts are immutable
- Reduces risks of accidently losing data due to human error
- Does not require indexing as new data is simply appended as it is created
Big Data characteristics MS with examples
Big Data problems and solutions MS
Big Data ethics MS
-
Data Representation39
-
Image Representation10
-
Sound Representation11
-
Compression and Encryption19
-
Data Structures and Algorithms86
-
Databases and SQL47
-
Programming29
-
Communication77
-
Object-Oriented Programming33
-
Theory of Computation86
-
Computer Systems45
-
Computer Organisation and Architecture96
-
Networking102
-
Functional Programming25
-
Big Data16
-
Ethics0
