Varying Informativeness of Inductive Bias in Gaussian Processes Regression for Small Data

In many real-world applications, data availability is limited,
posing a fundamental challenge to the performance and reliability of
machine learning models. While large neural models excel in high-data
regimes, their effectiveness diminishes with fewer observations. In con-
trast, Gaussian Processes (GPs) offer a principled and interpretable ap-
proach for small-data scenarios due to their flexibility and strong in-
ductive priors. In this work, we investigate the impact of inductive bias
informativeness on the performance of GP models by leveraging Linear
Ordinary Differential Equation Gaussian Processes (LODE-GPs), a sub-
class of GPs constrained by linear ODEs. We propose a suite of increas-
ingly informative ODE-based prior structures, derived from variations of
a linearized bipendulum system, to systematically quantify how model
performance scales with the strength and correctness of prior knowledge.
We evaluate these variants across multiple dataset sizes, noise levels,
and inductive priors, comparing their behavior in terms of MAP esti-
mates,trainingandtestMSE,ODEsatisfaction,andnoiselevelrecovery.
Our results reveal that more informative or correct inductive priors lead
to improved data efficiency and generalization, especially in low-sample
regimes,whileincorrectpriorscandegradeperformancesignificantly.No-
tably, even partial priors can approximate full-system performance given
enough data. These findings underline the importance of tailoring induc-
tive bias to task-specific structure when designing models for data-scarce
environments.