A broad-brush view of the evolution of photometric and spectroscopic properties of brown dwarfs has been in place for nearly a decade, largely thanks to the first parallax programs that allowed modest samples of brown dwarfs to be placed on color-magnitude diagrams and even smaller samples of dynamical masses barely breaching the hydrogen-fusion boundary. I will present results from our high-precision IR astrometry program at CFHT and laser guide star adaptive optics orbit monitoring at Keck that greatly expand such tests of substellar models. Our more populous sample of ultracool dwarf parallaxes provides new perspectives on the cooling of brown dwarfs: we discover a phase of rapid cloud clearing (the "L/T gap"); quantify the seemingly paradoxical brightening in J-band as objects cool from L to T dwarfs; and identify an unexpectedly large scatter in the absolute magnitudes of cold T dwarfs. With data from Keck we have more than tripled the numbered of dynamical masses for ultracool dwarfs, pushing to much lower temperatures than before. This has enabled the strongest tests of substellar models to date and revealed significant discrepancies with observed colors, temperatures, and even cooling rates, which has implications for cluster IMF determinations and the inferred masses for directly imaged planets.