In the grounds of Green College is a striking classical building that was formerly an astronomical observatory. Begun in 1773 by Henry Keene and completed by James Wyatt, this building is described by the architectural historian Nikolas Pevsner as "architecturally the finest observatory in Europe".

The building is capped by an eighteenth-century version of the ancient Tower of the Winds in Athens. Like the original, it also has zodiacal signs and an octagonal shape. Wyatt's version is topped by Hercules and Atlas holding up the Globe. The Observatory is no longer used for astronomical purposes, but must have been a lovely building from which to have worked. The place was described at the time as "a calm and refined locality".