What Method did you use to combine different data sources?
Full Outer Join: keeps all the information from both tables regardless if they are matching
What other methods of joining are there?
Left Inner: keeps all the information from the left tables and bring only the matching value from the right table
Inner: keeps only the matching values from both tables
What is a join?
combines two or more tables based on a related column, allowing data to be reviewed and analysed together
What does combining data refer to?
- Process of integrating 🔄
- and merging information 🧩
- from various sources 📚
- into a unified dataset 1️⃣
- for analysis or management purposes📊
What are the risks of combining data?
- Select wrong join type❌ 🔗 could lead to data loss (e.g. data loss)
- Security risks: table w/PII & final outcome not having right protection🛡️
- Large/complex joins = affect performance
- Data Quality - consistency/terminlogy
List in order all the stages of the lifecycle
- Plan
- Data Prep
- Analysis
- Modelling
- Refine & Compare
- Communicate & Implement
Provide an example of what you have done for refine & compare for a model you created
- Error Metrics (RMSE/MAE) 📉
- Changed models and parameters to achieve lowest RMSE/MAE values🧬⚙️
- Confidence Levels📏
- Project Sponsor Feedback💬
- Domain context: knew flat line was not realistic🧭
What is Privacy by Design?
- Embeds Privacy Protection 🛏️🛡️
- as part of the design/implementation 🛠️
- of Systems, Products, Business Practices📦
- from the start, not an after thought.
🏁
How have you applied privacy by design?
- BOBI acess:🔑
- Only used data necessary for project➖
- Google Sheets -** JLP Access **Only 👥
What DQ risks did you come across? (3 examples)
- Completeness: records present in a dataset e.g. missing records against dates, missing matches between the 2 sources
- Consistency: format of PK different between sources
- Accuracy: represents the truth e.g. risk of mistypes and I removed zeros that were not genuine
How did you resolve each DQ risk example?
- Completeness: domain context meant I knew the data was meant to have gaps - weekly usage. Removed the NULLs from the join from the analysis
- Consistency: during data prep I changed the formatting so they were consistent and I could continue the join
- Accuracy: I removed the zeros from the analysis
Provide an example in which you acted logical and analytical
- S- Forecast Code Selection for Streamline
- T - Enough data📊✅, used infrequently 🌜⏳to provide best opportunity for success
- A- Analysed codes for sufficient data/similar for consolidation
- R- Selecting a code that could be consolidated, streamlining codeset and meeting project requirement
What was the conclusion of the analysis? (5 things)
- 125 - low use - removal or consolidation
- 134 - increase - not put forward for removal
- Histogram emphasised need for code review
- Bar charts- several codes not used at all - put forward for removal
- Model was not good for seasonal predictions, need more data, but did help to inform for 125 and 134
What alternative methods or tools did you suggest for the project to be successful?
- Identified low usage code,🔭
- Completed time series forecast to understand future pattern📉
- NLP📝
- Dashboard📊
What were your customer requirements and how did you define them? (2 answers)
- Met with project sponsor to go through their challenges⛰️, strategy🎯 and project objectives🎯
- Drafted KPIs and ensured clarity via email📏📧
How did customer requirements shape the project? (7 answers)
- Project scope🎯
- Sponsor updates🆕
- Analysis approach🛤️
- Bar chart: Overview📊
- Histrogram: Skew📊
- Time Series for the future trend📈
- Inform decisions on streamlining 🏞️
What are the differences between open and public data?
- Public: all data in the public domain
- But…freedom of information request & is not always machine readable
- Open data is a subset of public data that is freely accessible to everyone without restrictions to use and share, it’s provided in a common, machine readable format
- Open example: often government portals provide open data on economy or health or environment in a easy to consume format
- Public example: Office for National Statistics publishes stats about the UK like census data and certain datasets may come with restrictions about how you can use it especially if there is sensitive info
What is administrative data?
- Information collected during
- registration, 📋
- transactions 💳
- or record keeping📀
- To support organisations with regulatory or operational reasons
- Can be structured or unstuctured (docs) but often stored electronically
What is research data?
information collected or generated to validate findings in research
What data structures did you take into account?
- Structured administrative data on inventory transactions that was in a pre-defined format
- Structured reference data that was captured in a** google sheet **like label, code, definition of codes - contained qualitative free-text data (plus unstructured emails)
What different database system designs did you take into account? (What is a relational database?)
- Structured data🏛️
- organised into tables🪑
- with pre-defined relationships between them📚🔗
- to help efficient storage,🗄️
- retrieval🏈
- Management👔
How do you adapt your communication depending on the audience & situational requirements? (4 answers)
- Power interest matrix: prioritise stakeholder communication ⚡️
- Knowledge Level: Architect needed explanation🧠
- Adjust tone: Formal = less familiar 🎵
- Technical: business vs data facing💻
Can you provide 2 examples of data classification principles you’ve applied in your project?
- Usage data:
* internal only🫁
* administrative
📝
* structured/quantitative🏗️🔢
* low risk🔽🛡️ - Metadata
* internal only🫁
* reference data🔤
* structured/qualitative🏗️🔤
* low risk🔽🛡️
Can you think of a time where you need to be flexible with classification?
If the usage data was to be combined with customer data or sensitive financial data, the risk would increase as it would include PII or sensitive information
