This course provides a basic fundamental of big data architecture and management. Students will learn the big data processes and the current big data technologies that are available. Further, students will be exposed to the big data platform ecosystem for big data manipulation. The big data management will be explored for the best practice in managing and manipulating large amount of data. At the end of the course, students should be able to understand the architecture and management of big data and also can develop simple application of big data handling using particular platform in assignment.
- Understand the technology for managing, processing and manipulating large amount of data.
- Design big data platform demonstrating the implementation of big data applications.
- Discuss current technology that support for sustainability of the big data platform ecosystem.
- How to Become a Data Engineer in 2024
- Essential Preparations for a Successful Start in BDM Class
- Student Information
- Course Information
- Assignment
- Exercise
| No. | Content | File |
|---|---|---|
| 1. | Charting Your Path in Data and Machine Learning |  |
| 2. | Navigating the Data Science Landscape | |
| 3. | Database Types | |
| 3a. | NoSQL Database | |
| 4. | Navigating the Database Landscape | |
| 5. | Data Management | |
| 6. | The Data Journey: From Raw to Refined | |
| 7. | The Data to MLOps Journey: An End-to-End Process | |
| 8. | Data Platforms Architecture: Governance and Operations | |
| 9. | Creating Data Products to Monetize Data | |
| 10. | Revolutionizing Data and Machine Learning with DataOps and MLOps | |
| 11. | Data Science for Beginners - A Curriculum | |
| No. | Content | File |
|---|---|---|
| 1. | Unveiling Instagram's Engagement Magic through Machine Learning | |
| 2. | Unlocking Spotify's Musical Enchantment with Machine Learning | |
| 3. | Netflix | |
| 4. | Big data in healthcare: management, analysis and future prospects | |
| 5. | Introduction to Big Data Computing for Geospatial Applications | |
| 6. | 40 Stats and Real-Life Examples of How Companies Use Big Data | |
| No. | Section | Content |
|---|---|---|
| 1 | Introduction to Big Data Management | A. What is Big Data? B. The Importance of Managing Big Data C. Why Big Data Management is Important? D. The History of Big Data |
| 2 | Understanding Big Data | A. Defining Big Data B. Characteristics of Big Data C. Sources of Big Data D. Challenges in Dealing with Big Data E. The workflow of Big Data Management |
| 3 | The Role of Big Data Management | A. Managing big data B. Benefits of Effective Big Data Management C. Risks of Ignoring Big Data Management D. Industries Benefiting from Big Data Management |
| 4 | Data Collection and Storage | A. Data Collection Methods 1. Traditional Data Sources 2. Emerging Data Sources B. Data Storage and Warehousing C. Data Security and Privacy Concerns |
| 5 | Data Processing and Analysis | A. Data Preprocessing B. Data Analysis Tools and Techniques C. Real-time Data Processing D. Machine Learning in Big Data Analysis |
| 6 | Data Integration and Governance | A. Data Integration Strategies B. Data Governance Best Practices C. Ensuring Data Quality |
| 7 | Data Visualization | A. Importance of Data Visualization B. Data Visualization Tools C. Creating Effective Data Visualizations |
| 8 | Case Studies | A. Successful Big Data Management Implementations B. Lessons Learned from Failed Big Data Projects C. Real-world Examples |
| 9 | Challenges in Big Data Management | A. Scalability Issues B. Security Concerns C. Compliance and Regulatory Challenges |
| 10 | Future Trends in Big Data Management | A. The Evolving Landscape of Big Data B. Predictions for the Future C. Preparing for the Next Generation of Big Data |
| 11 | Best Practices in Big Data Management | A. Key Takeaways B. Actionable Strategies C. Tips for Effective Big Data Management |
| 12 | Other References |
| Week | Topic |
|---|---|
| 1 | Introduction to Big Data and Big Data Analytics - Fundamentals and concepts of big data |
| 2 | Big Data Processing and Technology - Batch, real-time, and streaming processing. - Scalability, storage, sourcing challenges. |
| 3-4 | - ACID, BASE, and CAP theorem - Distributed File Processing & Map Reduce Processing |
| 5 | - Lambda Architecture |
| 6-7 | Relational Database (RDBMS) - Relational Data Modelling - Database design phases |
| 8 | Relational Database (RDBMS) - SQL programming (DDL, DML, CRUD Operation) |
| 9 | Relational Database (RDBMS) - SQL programming (Subqueries, Join Tables, Aggregate) |
| 10 | No SQL Database - Introduction to No SQL database - Semi-structured data Modelling (Key Value, Column Family, Document, and Graph) |
| 11-12 | No SQL database (Document-based Database) - Document-based data modelling - MongoDB query language |
| 13-14 | Cloud Technology - Introduction to Cloud - AWS Cloud (via AWS Learning Management System) |
| 15 | Project Presentation |
Please create an Issue for any improvements, suggestions or errors in the content.
You can also contact me using Linkedin for any other queries or feedback.