This tutorial is structurally divided into the following parts:

  • Introduction: A demonstration of the performance of the framework for reliable confidence estimation on complex real-world data.
  • Background Theory: A presentation of the background theory of the Conformal Predictions framework based on concepts from algorithmic randomness, transductive inference and statistical hypothesis testing, followed by relevance in classification and regression contexts.
  • Code Simulation: A demonstration of the implementation of the framework in Matlab code.
  • Framework Application: Case study presentations of application domains where the framework has demonstrated good performance.
  • Extensions and Adaptations: Examples of extensions of the framework to active learning, model selection, anomaly detection, feature selection, change detection.
  • Interactive Discussion: Participants and organizers discuss related application domains, problems and potential synergies (including any other questions from the audience).

Syllabus

Please find the syllabus for the tutorial below (download PDF version)

  • Introduction to the Conformal Predictions Framework
    • A demonstration
  • Background Theory
    • Algorithmic Randomness, Hypothesis Testing and Transductive Inference
    • Transductive Confidence Machines
    • Inductive Confidence Machines
    • Mondrian Confidence Machines
  • Code Simulation
  • Framework Application
    • Image Data
      • Image Classification, Face Recognition, Head Pose Estimation
    • Biomedical Data
      • Microarray and proteomics pattern recognition, Gene expression recognition, Cancer diagnosis, Hypoxia recognition, Cardiac decision support, MRI analysis
    • Other Sensor Data
      • Network Traffic Analysis, Smell Classification, Electron Content Prediction, Roadside Assistance
  • Extensions and Adaptations
    • Active Learning
    • Model Selection
    • Feature Selection
    • Anomaly Detection
    • Quality Assessment
    • Change Detection
  • Questions and Discussion

Intended Audience

This tutorial is intended for graduate students, researchers, scientists, engineers, and application developers who work in machine learning, especially related to risk-sensitive applications (such as in healthcare), where a reliable estimation of confidence is very critical. We believe that both young and seasoned researchers will benefit from exposure to this recent framework. The tutorial aims at introducing concepts and open perspectives that motivate further work in this domain, ranging from fundamentals to applications and systems. While the focus of the tutorial will be technical, we will aim at giving participants a view of the broad scope of research/applications that can be achieved with this framework.

Pre-requisites

A basic understanding of machine learning and pattern recognition approaches (such as classification, clustering, regression, etc) is a pre-requisite. Prior knowledge of theories related to algorithmic randomness and Kolmogorov complexity will be useful, but not necessary.