Big data
Big Data is a term that describes large and complex data sets.
This data cannot be collected, managed and processed using traditional data processing software within a reasonable period of time.
Big data can include:
Structured
Unstructured
Semi-structured data
Each of which can be exploited to understand customer insights, leading to better decisions and actions.
Big dat sources
- media
- cloud
- web
- internet of things
- databases
- social network profiles
- social influences
- activity generated data
- data warehouse appliances
- network & in-stream monitoring technologies
- legacy documents
The four V´s of big data
- Variety- different forms of data
- Veracity- uncertainty of data
- Volume- scale of data
- Velocity- analysis of streaming data
Volume
Volume refers to the amount of data generated through various sources. On social media sites, for example, we have 2,6 billion Facebook users, 2 billion on YouTube, and 1 billion on Instagram.
Velocity
This is the speed at which data is being made available. The rate of transfer over servers and between users has increased to a point where it is impossible to control the information explosion.
There is a need to address this with more equipped tools, and this comes under the kingdom of big data.
Variety
There is structured and unstructured data
- Different formats: Pictures, videos, emails, tweets, posts, messages, etc. are unstructured.
- Sensor-collected data from the millions of connected devices (IoT) is what you can call semi-structured.
- Records maintained by businesses: transactions, storage, and analyzed previous unstructured information are part of structured data.
Sectors generating and using Big data
- banking and finance
- media &entertainment
- healthcare
- education
- government
- transportation
- insurance
- retail
Big data analytics
The massive quantities of data contributed by all these users in terms of images, videos, messages, posts, tweets etc. have pushed data analysis away from the now incapable excel sheets, databases, and other traditional tools toward big data analytics.
Big data analytics is the often complex process of examining big data to uncover information, such as hidden patterns, correlations, market trends and customer preferences. This can help organizations to make informed business decisions.
Benefits of Big data analytics
- more effective marketing
- new revenue opportunities
- customer personalization and improved operational efficiency
- competitive advantages over rivals.
4 steps of big data analytics
- Collectdata from a variety of different sources. Often, it is a mix ofsemi-structuredand unstructured data
- Data is processed: After data is collected and stored in a data warehouse, data professionals must organize, configure and partition the data properly for analytical goals.
- Data is cleansed for quality:
- Data professionals wash up the data using scripting tools or enterprise software.
- They look for any errors or inconsistencies, such as duplications or formatting mistakes, and organize and tidy up the data. - The collected, processed and cleaned data is analyzed with analytics software
Tools for analytics software
- Data mining, search of patterns and relationships
- Predictive analytics, which builds models to forecast customer behavior and other future developments
- Machine learning, which uses algorithms to analyze large data sets
- Deep learning, more advanced than machine learning
- Text mining and statistical analysis software
- Artificial intelligence (AI)
- Business intelligence software
- Data visualization tools
Big data uses and examples
- Customer acquisition and retention
- Personalized engines
- Targeted ads
- Product development
- Price optimization
- Supply chain and channel analytics
- Risk management
- Improved decision making
Big data analytics benefits
- Quickly analyzing large amounts of data from different sources, in many different formats and types.
- Rapidly making better-informed decisions for effective strategizing, which can benefit and improve the supply chain, operations and other areas of strategic decision-making.
- Cost savings, which can result from new business process efficiencies and optimizations.
- A better understanding of customer needs, behavior and sentiment, which can lead to better marketing insights, as well as provide information for product development.
- Improved, better informed risk management strategies that draw from large sample sizes of data.
Big data challenges
1- Accessibility of data. With larger amounts of data, storage and processing become more complicated. Big data should be stored and maintained properly to ensure it can be used by less experienced data scientists and analysts.
2- Data quality maintenance. With high volumes of data coming in from a variety of sources and in different formats, data quality management for big data requires significant time, effort and resources to properly maintain it.
3- Data security. The complexity of big data systems presents unique security challenges. Properly addressing security concerns within such a complicated big data ecosystem can be a complex undertaking.
4- Choosing the right tools. Selecting from the vast array of big data analytics tools and platforms available on the market can be confusing, so organizations must know how to pick the best tool that aligns with users’ needs and infrastructure.
5- Lack of internal analytics skills and the high cost of hiring experienced data scientists and engineers, some organizations are finding it hard to fill the positions.
