This course introduces the end-to-end process of modern data engineering in three parts. Part I (Lectures 2-5) covers data collection and preparation: collection, exploration, cleaning, integration, transformation, and reduction. Part II (Lectures 6-14) covers storage and processing at scale: SQL, NoSQL, Spark, data warehouses and pipelines (BigQuery, dbt, Airflow), and streaming (Kafka). Part III (Lectures 15-19) covers monitoring and updates for AI systems: ML systems and MLOps, LLM systems and retrieval-augmented generation, and agents, including data agents that read and write data systems. Through a team-based project, students design and implement a pipeline that connects raw data to intelligent decision-making support.

Students are expected to have completed the following courses, or possess equivalent knowledge:
- IMEN574 Programming for Data Science

- [DMCT] Data Mining: Concepts and Techniques (UIUC CS412)
- [DMS] Database Management Systems (Wisconsin CS564)
- [IDS] Introduction to Database Systems (Berkeley CS186)
- Lecture slides (PDF and HTML) are posted on PLMS and in the course Google Drive.

https://docs.google.com/spreadsheets/d/1oPqM5g4XYwU_PZ0DFld4QVmYYWPAdWa8m3Cyl2n_rwY/edit?usp=sharing

Classroom: Engineering Building 4, Room 302. The registration system lists Room 407, but Room 302 is the correct location for the rest of the semester.

Attendance: taken through the POSTECH electronic attendance app. At the start of each class a 4-digit verification code is announced; students enter the code in the app under "Verification Code".

Project: teams of 2-5 students. Teams and draft topics are registered in the shared project sheet.

-