Overview

This is an in-progress set of course notes for Data 101: Data Engineering, UC Berkeley.

This repository is updated semi-weekly and is meant to be used in addition to lecture. In other words, these notes contain core concepts for reference, whereas lecture contains activities and demos for internalizing the concepts.

Table of Contents

Chapters are bundled by topic, not by lecture number. This means that content can appear out of order compared to what is presented in lecture. We will try to indicate this where possible.

Navigate with the sidebar or with the links below:

SQL

• 1. SQL
  • 1.1. SQL Review
  • 1.2. String Manipulation
  • 1.3. Truncating Relations
  • 1.4. Creating Tables and Views from Data
  • 1.5. Common Table Expressions (CTE)
  • 1.6. Subqueries
  • 1.7. DML—Data Manipulation Langauge
  • 1.8. DDL–Data Definition Language
  • 1.9. Keys & Constraints

Relational Algebra

• 2. Relational Algebra
  • 2.1. Relational Algebra Overview
  • 2.2. Extended RA Operators

Query Performance

• 3. Query Performance
  • 3.1. Index Selection
  • 3.2. Query Planning
  • 3.3. Rules of Thumb

Data Modeling

• 4. Data Modeling
  • 4.1. Data Models
  • 4.2. Entity Relationship(ER) Diagrams

Data Preparation

• 5. Data Preparation
  • 5.1. Structural
  • 5.2. Numerical, Granularity
  • 5.3. Outliers
  • 5.4. Imputation and Entity Resolution

Semistructured Data

• 6. Semistructured Data
  • 6.1. Semistructured Data
  • 6.2. MongoDB

Sampling

• 7. Sampling
  • 7.1. Database Sampling

Transactions

• 8. Transactions
  • 8.1. Transactions

Parallel/distributed Computing

• 9. Parallel/Distributed Computing
  • 9.1. Parallel and Distributed Computing
  • 9.2. CAP Theorem
  • 9.3. MapReduce

Data Pipelines

• 10. Data Operations and Pipelines
  • 10.1. Data Operations and Pipelines
  • 10.2. dbt

BI

• 11. Business Intelligence (BI)
  • 11.1. Business Intelligence (BI)

Graph Databases

• 12. Graph Databases
  • 12.1.1. Property Graphs
  • 12.1.2. Triplestores/RDFs
  • 12.1.3. Ontologies

Spreadsheets

• 13. Spreadsheets
  • 13.1. The Spreadsheet Data Model
  • 13.2. Basic Spreadsheet Functions
  • 13.3. XLOOKUP
  • 13.4. HCI Research and Direct Manipulation
  • 13.5. Scalability Challenges
  • 13.6. Spreadsheet Data Model, Recap

Appendix

• SQL Style Guide
• MongoDB Style Guide

Course Acknowledgments

The content and structure of Data Engineering have been designed and developed over the years thanks to many dedicated faculty and instructors at UC Berkeley.

Valuable and essential instructional contributors (alphabetical order): Joe Hellerstein, Lakshya Jain, Aditya Parameswaran, and Lisa Yan.

Course notes in particular were developed over many semesters by many teaching assistants. Thank you!!