Uncertainty in Deep Learning - Cambridge Machine Learning

Hello everyone,
We are working with biomedical applications in our lab, and we have been using the Tiramisu architecture [1] for some time. We would like to update to a better architecture now. We have specific requirements (listed below) which makes the transition hard, since most newer architectures use a pretrained backbone [2].
The requirements are:

The number of input channels varies wildly (1-4 channels mostly, e.g. when we have multiple fluorescent channels from a microscope)
The output layer can be used for segmentation or pixel regression, and sometimes accommodate an extra layer for aleatoric uncertainty [3].
Must converge quickly, as we often don't have access to million of images since the clinical/lab experiments are costly. We do use a lot of synthetic augmentation schemes.

Should I use a pretrained backbone and feed my e.g. 1-channel image in grayscale (this paper had good success with pretrained arch. [4])? Or could I train a model with modified backbone from scratch? Has anyone tried something like this?
Is there any architecture you can recommend for this type of usage?
[1] https://ieeexplore.ieee.org/document/8014890
[2] https://paperswithcode.com/sota/semantic-segmentation-on-cityscapes
[3] https://papers.nips.cc/pape7141-what-uncertainties-do-we-need-in-bayesian-deep-learning-for-computer-vision.pdf
[4] https://papers.nips.cc/pape8596-transfusion-understanding-transfer-learning-for-medical-imaging.pdf

[R] "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?", Kendall & Gal 2017

[D] What is the current state of dropout as Bayesian approximation?

Some time ago already, Gal & Ghahramani published their Dropout as Bayesian Approximation paper, and a few more follow-up papers by Gal and colleagues about epistemic vs. aleatoric risks etc. There they claim that test-time dropout can be seen as Bayesian approximation to a Gaussian process related to the original network. (I would not claim to understand the proof in all of its details.) So far so good, but at the Bayesian DL workshop at NIPS2016 Ian Osband of Google DeepMind published his note Risk versus Uncertainty in Deep Learning: Bayes, Bootstrap and the Dangers of Dropout, where he claims that even for absurdly simple networks you can analytically show that the 'posterior' you get using MC dropout doesn't concentrate asymptotically -- which I take as saying that there's no Bayesian approximation happening, since almost any reasonable prior on the weights should lead to a near-certain posterior in the limit of infinite data.
Alas, there are still papers popping up using the MC dropout approach, without even mentioning Osband's note. Did I miss something? Is there a follow-up to Osband's note? A rebuttal? I didn't attend NIPS2016, and I am thus not aware of any discussions that might have happened there, but would certainly appreciate any pointers (-- and given that Yarin Gal was co-organizing that workshop, I am pretty sure that he has seen Osband's note).
Edit: For completeness, here is Yarin Gal's thesis on this topic and the appendix to their 2015 paper containing the proof. Additionally, the supplementary material (section A) of Deep Exploration via Bootstrapped DQN contains some more of Ian's thoughts on this issue

Using deep learning to design a 'super compressible' material

Using deep learning to design a 'super compressible' material. The system uses a less used method called Bayesian machine learning. The researcher (Miguel Bessa, Assistant Professor in Materials Science and Engineering at Delft University of Technology) thought probabilistic techniques were the way to go when analyzing or designing structure-dominated materials because they deal with uncertainties that he categorizes as "epistemic" and "aleatoric". Normal deep learning methods are non-probabilistic.
"Epistemic or model uncertainties affect how certain we are of the model predictions (this uncertainty tends to decrease as more data is used for training). Aleatoric uncertainties arise when data is gathered from noisy observations (for example, when different material responses are observed due to uncontrollable manufacturing imperfections)."
Structure-dominated materials "are often strongly sensitive to manufacturing imperfections because they obtain their unprecedented properties by exploring complex geometries, slender structures and/or high-contrast base material properties."
https://www.youtube.com/watch?v=cWTWHhMAu7I

[R] Pytorch implementation of "What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?", NIPS 2017

aleatoric uncertainty deep learning video

Aleatoric uncertainty captures noise inherent in the observations. On the other hand, epistemic uncertainty accounts for uncertainty in the model -- uncertainty which can be explained away given enough data. Traditionally it has been difficult to model epistemic uncertainty in computer vision, but with new Bayesian deep learning tools this is now possible. We study the benefits of modeling Aleatoric uncertainty accounts for noise inherent in the observations due to class overlap, label noise, homoscedastic and heteroscedastic noise, which cannot be reduced even if more data were to be collected. In X-ray imaging, this can be caused by sensor noise due to random distribution of photons during scan acquisition. Since both types of uncertainty are not constant throughout all predictions, we need a way of assigning a specific uncertainty to each prediction. That’s where the new TensorFlow Probability package steps in to save the day. It provides a framework that combines probabilistic modeling with the power of our beloved deep learning models. Aleatoric uncertainty. Aleatoric uncertainty captures our uncertainty with respect to information which our data cannot explain. For example, aleatoric uncertainty in images can be attributed to occlusions (because cameras can’t see through objects) or lack of visual features or over-exposed regions of an image, etc. It can be explained away with the ability to observe all explanatory variables with increasing precision. Aleatoric uncertainty is very important to model for: Aleatoric uncertainty is the uncertainty arising from the natural stochasticity of observations. Aleatoric uncertainty cannot be reduced even when more data is provided. When it comes to measurement errors, we call it homoscedastic uncertainty because it is constant for all samples. Input data-dependent uncertainty is known as heteroscedastic uncertainty. Introduction This post is aimed at explaining the concept of uncertainty in deep learning. More often than not, when people speak of uncertainty or probability in deep learning, many different concepts of uncertainty are interchanged with one another, confounding the subject in hand altogether. To see this, consider such questions. - Is my network's classification… aleatoric and epistemic uncertainty (Hora, 1996). Roughly speaking, aleatoric (aka statistical) uncertainty refers to the notion of randomness, that is, the variability in the outcome of an experiment which is due to inherently random effects. The prototypical example of aleatoric uncertainty is coin flipping: The data-generating process in this type of experiment has a stochastic component that cannot be reduced by whatsoever additional source of information (except Laplace’s demon 6.7 Epistemic,Aleatoric,andPredictiveuncertainties . . . . . . . . . . . . .127 7 FutureResearch133 References137 AppendixA KLcondition149 AppendixB Figures153 AppendixC SpikeandslabpriorKL159. Nomenclature RomanSymbols A matrix a vector a scalar W Weightmatrix D Dataset X Datasetinputs(matrixwithNrows,oneforeachdatapoint) Y Datasetoutputs(matrixwithNrows,oneforeachdatapoint) x n For aleatoric uncertainty, we mean the uncertainty inherent inside the randomness of data, and it can not decrease giving more training data. Epistemic uncertainty, on the other hand, comes from our lack of knowledge. In the context of modeling, it comes from the defects in model structure or weights. [5], uncertainty quantiﬁcation is a problem of paramount importance when deploying machine learn-ing models in sensitive domains such as information security [72], engineering [82], transportation [87], and medicine [5], to name a few. Despite its importance, uncertainty quantiﬁcation is a largely unsolved problem. Prior literature on uncertainty estimation for deep neural networks is dominated by Bayesian methods [37, 6, 25, 44, 45,

aleatoric uncertainty deep learning top

2019: Long-term projections of soil moisture using deep ...

Based on those translated images, the trained uncertainty-aware imitation learning policy would output both the predicted action and the data uncertainty motivated by the aleatoric loss function. PyData Warsaw 2018We will show how to assess the uncertainty of deep neural networks. We will cover Bayesian Deep Learning and other out-of-distribution dete... CUAHSI's 2019 Spring Cyberseminar Series on Recent advances in big data machine learning in HydrologyDate: April 19, 2019Topic: Long-term projections of soil... Aleatoric Music: Live Looping ... Modern Deep Learning through Bayesian Eyes - Duration: 1:00:53. ... PyData Tel Aviv Meetup: Uncertainty in Deep Learning - Inbar Naor - Duration: ...