The University Botanic Garden was founded by the Earl of Danby in 1621 as a physic garden, specifically for the growing of herbs and plants for use in medicine and science, and was built on the site of the medieval Jewish cemetry. The plant collection here has been expanded and nurtured ever since. The high walls, which protect tender plants, were built at the founding of the garden, but the layout of the beds dates only from the mid-nineteenth century. The first keeper, Jacob Bobart, and after him his son, also called Jacob, kept the gardens for seventy-eight years, building up a plant collection containing over a thousand species. The Bobarts established a system of plant classification and built greenhouses. A pear tree planted in the garden at the time of the Bobarts provided the wood for the Professor's Chair of today.

Jacob the elder was an excellent gardener and botanist, and in 1648 he published the first catalogue of plants in the garden. He must have been quite a character, because on feast days he was reputed to decorate his long and flowing beard with pieces of silver. He also kept a goat as a pet - surely a most unsuitable choice of animal for a gardener! Near the lily pond is a yew tree planted by him in approximately 1650; in former times it would have been clipped into various shapes.

Jacob the younger continued his father's work, and in addition created a seed list, something botanic gardens all over the world do today for purposes of mutal exchange. He warrants a menion in Zacharias von Uffenbach's travel book, Oxford in 1710: "We entered the Hortus Medicus and Professor Bobart was waiting for us. I was greatly shocked by the hideous features and villainous appearance of this good and honest man. His wife, a filthy old hag, was with him, and although she may be the ugliest of her sex he is certainly the more repulsive of the two."

One famous visitor to the gardens, in 1736, was the Swedish botanist Carolus Linnaeus, the deviser of the binomial system of nomenclature for plants. He spoke no English and conversed in Latin, and is reputed to have described the gardens as the most important in England.