Hi everyone, welcome to the 7th chapter in our Tencent Cloud Solutions Architect professional course, big data applications. At the end of this chapter you'll be able to understand the basic concepts, characteristics and technology framework of big data. The offline and real-time computing modes of big data. Tencent Cloud's big data products and Tencent Cloud's EMR solution. In this chapter we'll cover four sections, overview of big data, big data modes, Tencent Cloud's big data service system and Tencent Cloud's EMR solution. This video will cover the first section, overview of big data. Subsequent videos will cover the remaining three sections. Let's get started with section 1, overview of big data. In this video, we'll cover an introduction to big data, core big data industries, the technology framework of big data and big data solutions. According to Wikipedia. Big data refers to massive amount of data that cannot be extracted, managed, processed and organized into information that can help organizations make business decisions through currently popular software tools within a reasonable period of time. Gartner, a leading big data research company, defines big data as high-volume, high-velocity and high-variety information assets that demand cost-effective and innovative forms of information processing for enhanced insights, decision- making and process automation. In other words, big data analyzes large volumes of data, helps enterprises make better decisions and integrates with businesses to create new value. The rise of big data can be attributed to the rapid growth of data volume and diversified sources of data such as web logs, audios, videos, images and location information. Additionally, the research value and economic benefits of effective utilization of massive amounts of data became widely recognized in recent years. The cost of information storage has decreased from 10,000 USD per MB in the 1960s to 1 cent per GB now. The big data revolution would not have taken place if storage costs were not reduced. The popularity of smart devices and mobile applications have also played a role in the rise of big data. The characteristics of big data include volume, velocity, variety and veracity. The volumes of enterprise data are mostly at the terabyte level or above, and data volumes and industries such as banking and telecommunications are at the PB level or above and are growing at a rate of 40% every year. For velocity analysis results must be provided within seconds. Otherwise the data will lose its value. Data types include structured data such as text, semi-structured data represented by web page data and unstructured data such as web logs, audio, videos, images and location information. For veracity we must consider how to use powerful machine algorithms to purify the value of massive amounts of data more quickly and effectively. Core big data industries consist of data, products and services. Any company can become a data company. Data flow platform providers include data sharing platforms. Open data platforms, and data trading platforms. Data sources and API providers include government data, enterprise data personal data, and APIs. Products are provided by open source vendors and traditional closed source vendors. The application software consists of analysis software, security software and more. Basic software consists of structured and unstructured data bases. Hardware products consist of servers, storage, network devices, micromodules and more. For services, mass entrepreneurship has spawned numerous services including application services for government applications, industry applications, lifestyle applications and enterprise applications. Data analysis services include data processing and analysis services, and infrastructure services include data center and network services. The five steps involved in big data implementation are data preparations, data storage and management, computing, data analysis and knowledge presentation. During the data preparation process. Data is cleansed and organized in the traditional data processing system which is called extract, transform and load, ETL. Massive amounts of data are stored at very low costs for adaptation to diversified unstructured data and for scalability in data formats during data storage and management. For computing, appropriate algorithm models are used based on the types of data to be processed and the analysis goals to quickly process data. Data analysis involves discovering patterns and extracting new knowledge from complex data. In scenarios where big data provides support for decision making, analysis results are presented to users in an intuitive way during knowledge presentation. The technology framework of big data is shown in the following diagram. The big data processing system is equipped with data import and ETL for data preparation. Data storage management is handled with SQL and NoSQL. For computing there is batch processing, interactive analysis and stream processing. Data analysis consists of data mining, including data warehouses, Olav and business intelligence. For knowledge presentation, data visualization can be achieved for the user to view. The different big data solutions include data applications, ETL scheduling, access interaction, computing frameworks, resource allocation and storage. Some of the specific solutions and services are shown in the following diagram for reference. The basic big data platform in the cloud environment includes computing services, cloud computing and networks and cloud storage resources. The big data platform in the cloud offers professional technical support, a guarantee of high computing resources, low Ops and development costs. Compute-storage separation, deep service integration, fast delivery and resource elasticity.
