Tree-width and its linear variant path-width play a central role for the graph minor relation. In particular, Robertson and Seymour [J. Combin. Theory Ser. B, 35 (1983), pp. 39--61] proved that for every tree T, the class of graphs that do not contain T as a minor has bounded path width. For the pivot-minor relation, rank-width and linear rank-width take over the role of tree-width and path-width. As such, it is natural to examine if, for every tree T, the class of graphs that do not contain T as a pivot-minor has bounded linear rank-width. We first prove that this statement is false whenever T is a tree that is not a caterpillar. We conjecture that the statement is true if T is a caterpillar. We are also able to give partial confirmation of this conjecture by proving for every tree T, the class of T-pivot-minor-free distance-hereditary graphs has bounded linear rank-width if and only if T is a caterpillar; for every caterpillar T on at most four vertices, the class of T-pivot-minor-free graphs has bounded linear rank-width. To prove our second result, we only need to consider T = P-4 and T = K-1,K-3, but we follow a general strategy: first we show that the class of T-pivot-minor-free graphs is contained in some class of (H-1, H-2)-free graphs, which we then show to have bounded linear rank-width. In particular, we prove that the class of (K-3, S-1,S-2,S-2)-free graphs has bounded linear rank-width, which strengthens a known result that this graph class has bounded rank-width.