Conformal Prediction for Reliable Machine Learning ：Theory, Adaptations and Applications

Publication subTitle ：Theory, Adaptations and Applications

Author： Balasubramanian Vineeth;Ho Shen-Shyang;Vovk Vladimir

Publisher： Elsevier Science‎

Publication year： 2014

E-ISBN: 9780124017153

P-ISBN(Paperback): 9780123985378

P-ISBN(Hardback): 9780123985378

Subject： TP Automation Technology , Computer Technology

Language： ENG

Access to resources Favorite

Disclaimer: Any content in publications that violate the sovereignty, the constitution or regulations of the PRC is not accepted or approved by CNPIEC.

Description

The conformal predictions framework is a recent development in machine learning that can associate a reliable measure of confidence with a prediction in any real-world pattern recognition application, including risk-sensitive applications such as medical diagnosis, face recognition, and financial risk prediction. Conformal Predictions for Reliable Machine Learning: Theory, Adaptations and Applications captures the basic theory of the framework, demonstrates how to apply it to real-world problems, and presents several adaptations, including active learning, change detection, and anomaly detection. As practitioners and researchers around the world apply and adapt the framework, this edited volume brings together these bodies of work, providing a springboard for further research as well as a handbook for application in real-world problems.

Understand the theoretical foundations of this important framework that can provide a reliable measure of confidence with predictions in machine learning
Be able to apply this framework to real-world problems in different machine learning settings, including classification, regression, and clustering
Learn effective ways of adapting the framework to newer problem settings, such as active learning, model selection, or change detection

Chapter

Front Cover

Preface

Book Organization

Part I: Theory

Part II: Adaptations

Part III: Applications

Companion Website

Contacting Us

Acknowledgments

Part I: Theory

1 The Basic Conformal Prediction Framework

1.1 The Basic Setting and Assumptions

1.2 Set and Confidence Predictors

1.2.1 Validity and Efficiency of Set and Confidence Predictors

1.3 Conformal Prediction

1.3.1 The Binary Case

1.3.2 The Gaussian Case

1.4 Efficiency in the Case of Prediction without Objects

1.5 Universality of Conformal Predictors

1.6 Structured Case and Classification

1.7 Regression

1.8 Additional Properties of Validity and Efficiency in the Online Framework

1.8.1 Asymptotically Efficient Conformal Predictors

Acknowledgments

2 Beyond the Basic Conformal Prediction Framework

2.1 Conditional Validity

2.2 Conditional Conformal Predictors

2.2.1 Venn's Dilemma

2.3 Inductive Conformal Predictors

2.3.1 Conditional Inductive Conformal Predictors

2.4 Training Conditional Validity of Inductive Conformal Predictors

2.5 Classical Tolerance Regions

2.6 Object Conditional Validity and Efficiency

2.6.1 Negative Result

2.6.2 Positive Results

2.7 Label Conditional Validity and ROC Curves

2.8 Venn Predictors

2.8.1 Inductive Venn Predictors

2.8.2 Venn Prediction without Objects

Acknowledgments

Part II: Adaptations

3 Active Learning

3.1 Introduction

3.2 Background and Related Work

3.2.1 Pool-based Active Learning with Serial Query

SVM-based methods

Statistical methods

Ensemble-based methods

Other methods

3.2.2 Batch Mode Active Learning

3.2.3 Online Active Learning

3.3 Active Learning Using Conformal Prediction

3.3.1 Query by Transduction (QBT)

Algorithmic formulation

3.3.2 Generalized Query by Transduction

Algorithmic formulation

Combining multiple criteria in GQBT

3.3.3 Multicriteria Extension to QBT

3.4 Experimental Results

3.4.1 Benchmark Datasets

3.4.2 Application to Face Recognition

3.4.3 Multicriteria Extension to QBT

3.5 Discussion and Conclusions

Acknowledgments

4 Anomaly Detection

4.1 Introduction

4.2 Background

4.3 Conformal Prediction for Multiclass Anomaly Detection

4.3.1 A Nonconformity Measure for Multiclass Anomaly Detection

4.4 Conformal Anomaly Detection

4.4.1 Conformal Anomalies

4.4.2 Offline versus Online Conformal Anomaly Detection

4.4.3 Unsupervised and Semi-supervised Conformal Anomaly Detection

4.4.4 Classification Performance and Tuning of the Anomaly Threshold

4.5 Inductive Conformal Anomaly Detection

4.5.1 Offline and Semi-Offline Inductive Conformal Anomaly Detection

4.5.2 Online Inductive Conformal Anomaly Detection

4.6 Nonconformity Measures for Examples Represented as Sets of Points

4.6.1 The Directed Hausdorff Distance

4.6.2 The Directed Hausdorff k-Nearest Neighbors Nonconformity Measure

4.7 Sequential Anomaly Detection in Trajectories

4.7.1 The Sequential Hausdorff Nearest Neighbors Conformal Anomaly Detector

4.7.2 Empirical Investigations

4.8 Conclusions

5 Online Change Detection

5.1 Introduction

5.2 Related Work

5.3 Background

5.4 A Martingale Approach for Change Detection

5.5 Experimental Results

5.5.1 Simulated Data Stream Using Rotating Hyperplane

5.5.2 Simulated Data Streams Using NDC

5.6 Implementation Issues

5.6.1 Effect of Various Nonconformity Measures

5.6.2 Effect of Parameter ε

5.7 Conclusions

6 Feature Selection

6.1 Introduction

6.2 Feature Selection Methods

6.2.1 Filters

6.2.2 Wrappers

6.2.3 Embedded Feature Selection

6.3 Issues in Feature Selection

6.3.1 Selection Bias

6.3.2 False Discovery Rate

6.3.3 Relevance and Redundancy

6.4 Feature Selection for Conformal Predictors

6.4.1 Strangeness Minimization Feature Selection

6.4.2 Average Confidence Maximization (ACM)

6.5 Discussion and Conclusions

7 Model Selection

7.1 Introduction

7.2 Background

7.3 SVM Model Selection Using Nonconformity Measure

7.4 Nonconformity Generalization Error Bound

7.5 Experimental Results

7.6 Conclusions

Acknowledgments

8 Prediction Quality Assessment

8.1 Introduction

8.2 Related Work

8.2.1 Conformal Prediction

8.2.2 Confidence Estimation and the Transduction Principle

8.3 Generalized Transductive Reliability Estimation

8.3.1 Typicalness

Typicalness in machine learning

8.3.2 Transductive Reliability Estimation

A formal background

Assessing the classifier's quality: the curse of trivial models

8.3.3 Merging the Typicalness and Transduction Frameworks

Simplification of transductive reliability estimation for application within the typicalness framework

8.3.4 Extension of Transductive Reliability Estimation to Regression by Means of Local Sensitivity Analysis

Bagging variance

Local cross-validation

Local error modeling

8.3.5 Testing Methodology

8.4 Experimental Results

8.4.1 Experiments on Benchmark Problems

Comparing reliability and confidence

8.4.2 Practical Applications and Considerations

8.5 Discussion and Conclusions

Acknowledgments

9 Other Adaptations

9.1 Introduction

9.2 Metaconformal Predictors

9.2.1 Classifier Performance Metrics

9.2.2 Metaclassifiers and Metaconformal Predictors

Interpreting the p-values

Generalized performance

9.2.3 Experiments

9.3 Single-Stacking Conformal Predictors

9.3.1 Metaconformity versus Single Stacking

9.3.2 Single-Stacking Conformal Predictor

9.3.3 Experiments

9.4 Conformal Predictors for Time Series Analysis

9.4.1 Time Series Analysis Methods

Autoregressive moving average (ARMA)

Fractional auto regressive integrated moving average

Aggregating algorithm

9.4.2 Conformal Predictors for Time Series Analysis: Methodology

9.5 Conclusions

Acknowledgments

Part III: Applications

10 Biometrics and Robust Face Recognition

10.1 Introduction

10.2 Biometrics and Forensics

10.3 Face Recognition

10.4 Randomness and Complexity

10.5 Transduction

10.6 Nonconformity Measures for Face Recognition

10.7 Open and Closed Set Face Recognition

10.8 Watch List and Surveillance

10.9 Score Normalization

10.10 Recognition-by-Parts Using Transduction and Boosting

10.11 Reidentification Using Sensitivity Analysis and Revision

10.12 Conclusions

11 Biomedical Applications: Diagnostic and Prognostic

11.1 Introduction

11.2 Examples of Medical Diagnostics

11.2.1 Proteomics: Breast Cancer

11.2.2 Magnetic Resonance Imaging: Clinical Diagnosis and Prognosis of Depression

11.2.3 Online Multiclass Diagnostics

11.3 Nonconformity Measures for Medical and Biological Applications

11.3.1 Support Vector Machine (SVM)

11.3.2 k Nearest Neighbors (kNN)

11.3.3 NCM Based on Other Underlying Algorithms

11.3.4 NCM Based on Feature-Wise Two-Sample T-Test

11.4 Discussion and Conclusions

Acknowledgments

12 Network Traffic Classification and Demand Prediction

12.1 Introduction

12.2 Network Traffic Classification

12.2.1 Port-Based Approach

12.2.2 Payload-Based Classification

12.2.3 Host Behavior-Based Method

12.2.4 Flow Feature-Based Method

12.2.5 Our Approach

12.3 Network Demand Prediction

12.3.1 Our Approach

12.4 Experimental Results

12.4.1 Network Traffic Classification Datasets

12.4.2 Network Demand Prediction Datasets

12.5 Conclusions

13 Other Applications

13.1 Nuclear Fusion Device Applications

13.1.1 Image Classification for a Nuclear Fusion Diagnostic Device

13.1.2 Classification of L-mode/H-mode Confinement Regimes in a Tokamak Device

13.1.3 Reliable L-mode/H-mode Transition Time Estimation

13.2 Sensor Device Applications

13.2.1 Tea (Smell) Classification

13.2.2 Total Electron Content Prediction

13.2.3 Roadside Assistance Decision Support System

13.3 Sustainability, Environment, and Civil Engineering

13.3.1 Wind Speed Prediction

13.3.2 Air Pollution Assessment

13.3.3 Pavement Structural Diagnostics

13.4 Security Applications

13.4.1 Network Intrusion Detection

13.4.2 Surveillance

13.5 Applications from Other Domains

13.5.1 Software Engineering: Cost Estimation

13.5.2 Forensic Science: Striation Pattern Identification

13.5.3 Machine Translation: Quality Estimation

13.5.4 Pharmaceutical Industry: QSAR Modeling

Bibliography

Index

The users who browse this book also browse