Verifying well-definedness of variational objectives for probabilistic programs

Abstract

Probabilistic programming languages offer powerful generic inference algorithms to let programmers write probabilistic models with succinct programming notations. Stochastic variational inference (SVI) is one of the most popular classes of such inference algorithms. However, many verification challenges arise to assure that a given probabilistic program runs a valid SVI. Among those, this dissertation aims at the verification of well-defined variational objectives. We define a Pyro-like probabilistic programming language and its denotational semantics using a special data structure called random database. Then, we propose a sufficient set of conditions for the bounded evidence lower bound (ELBO), which is the most fundamental variational objective. The most remarkable condition is the finite expected execution time of a program, whose verification is a well-studied research problem. Finally, we generalized our result to three other variational objectives. The next challenge is to develop an automatic program analyzer based on our current knowledge.