Fall 2023
Data Science Seminar: Benchmarking
Lecturers: Mourad Khayati and Alberto Lerner
Teaching language: English
Level: MSc students
Academic year: Fall 2023
Overview
The data science seminar involves presentations covering recent topics in data science. The area of this year’s seminar is benchmarking. In the scope of this seminar, we investigate papers that describe various benchmarks for systems, algorithms, and data generation. The papers explore configuration mechanisms and parameterization techniques that optimize the performance of the evaluated entities when applied to large datasets.
Structure
The goal for the students is to learn how to critically read and study research papers, describe a paper in a report, and present it in a seminar. Under supervision, students will select one paper to study and compare it with related work. This seminar aims to help students gather in-depth knowledge of an advanced topic and develop the skills required to describe a complex problem from the time series field in the form of both a presentation, a written report, and an empirical evaluation.
IMPORTANT NOTE: The papers will be distributed on a first-come, first-serve basis.
Evaluation and Expectations
The final grade depends on the quality of the report, presentation, reproducibility experiments, and active participation during the seminar. Each participant prepares a self-contained report of min 6 pages and gives a presentation of 30 minutes. The report should describe the proposed benchmark in detail. The report might contain a small running example, counterexample(s), and should explore the extreme cases where the evaluated systems and algorithms would perform best and worst. The reproducibility consists of reproducing the same set of experiments introduced in the paper using a different setup (dataset, metric, parameters, etc.).
Advice on how to:
IMPORTANT NOTE: Attendance is mandatory for the two-class seminar sessions. The total number of participants will be limited to 10.
Schedule
Kickoff Meeting. Date: Tue, 26.09.2022, 15:00-16:30, room: A403
Setup and organization of the seminar and paper assignment
----------------------------------------------------------------------
Date: Tue, 07.11.2023
Report deadline Batch1
Date: Tue, 14.11.2023, all day, room: C433 or C411
Office meeting with students from Batch1
First Seminar Session. Date: Tue, 21.11.2023, 15:00-18:00, room: G414
Presentations of Batch1
----------------------------------------------------------------------
Date: Tue, 28.11.2023
Report deadline of Batch2
Date: Tue, 05.12.2023, all day, room: C433 or C411
Office meeting with students from Batch2
Second Seminar Session. Date: Tue, 12.12.2023, 14:15-17:00, room: G414
Presentations of Batch2
----------------------------------------------------------------------
Date: Tue, 16.01.2024
Deadline final Report of Batch1 and Batch2
Paper Assignment
The papers will be distributed on a first-come, first-serve basis. To select one paper from the list of papers, please use the following link.
Paper & code | Presentation Date | Presenter | Mentor |
Pollock: A Data Loading Benchmark, PVLDB’2023 | 21.11.2023 | Sophie Pfister | Alberto Lerner |
Towards Benchmarking Feature Type Inference for AutoML Platforms, SIGMOD’21 | 21.11.2023 | Adriana Moisil | Mourad Khayatii |
Series2Graph: Graph-based Subsequence Anomaly Detection for Time Series | 21.11.2023 | Majid Samar | Mourad Khayati |
ADBench: Anomaly Detection Benchmark, NeurIPS’2022 | 21.11.2023 | De Soham | Alberto Lerner |
TSB-UAD: An End-to-End Benchmark Suite for Univariate Time-Series Anomaly Detection, PVLDB’22 | 12.12.2023 | Abeer Refay | Alberto Lerner |
Exathlon: A Benchmark for Explainable Anomaly Detection over Time Series, PVLDB’21 | 12.12.2023 | Nadezhda Videneeva | Mourad Khayati |
Anomaly Detection in Time Series: A Comprehensive Evaluation, PVLDB’22 | 12.12.2023 | Deborah Schaer | Mourad Khayati |
What Is the Price for Joining Securely? Benchmarking Equi-Joins, PVLDB’2022 | Not Assigned | Alberto Lerner | |
FEBench: A Benchmark for Real-Time Relational Data Feature Extraction, PVLDB 2023 | Not Assigned | Alberto Lerner | |
Benchmarking Learned Indexes, PVLDB’21 | Not Assigned | Alberto Lerner | |
M2Bench: A Database Benchmark for Multi-Model Analytic Workloads, PVLDB’23 | Not Assigned | Mourad Khayati | |
Benchmarking Learned Indexes, PVLDB’2022 | Not Assigned | Alberto Lerner | |
Cardinality Estimation in DBMS: A Comprehensive Benchmark Evaluation, PVLDB 2022 | Not Assigned | Alberto Lerner |