The history of the castle Castolovice dates back to the 13th century, when it was a fortress in the marshes. During the 15th and 16th centuries the castle changed owners many times. The most important of them was king George of Poděbrad, whose first wife Kunhuta was born a Sternberg. King Rudolf II. bought Castolovice in 1577 – although there is no evidence that he ever visited it – and seven years later he sold it to the counts of Oppersdorff.

Between the years 1588 and 1615 Frederick of Oppersdorf and his wife Magdalena of Donín rebuilt the now practically derelict fortress into a Renaissance castle. In 1694 Castolovice was bought by Adolf Vratislav Sternberg, the highest burgrave of the kingdom of Bohemia and the castle has remained in the ownership of the family Sternberg to this day, with only two interruptions: The German occupation during the second world war and its confiscation during the 40 year communist rule.

In the last half of the 19th century part of the castle was rebuilt in the then fashionable Neo-Gothic style and the English Park was designed and created.
At the beginning of the 20th century the Neo-Gothic elements were removed and a huge tower was built above the main entrance.

In 1992, thanks to the law of restitution passed by the government under the first post communist president, Václav Havel, Diana Phipps Sternberg had the castle of Castolovice returned to her. She immediately planted trees, shrubs and flowers in the courtyard and park. Rose and water gardens were planted with a wide variety of flora. A game enclosure for stag and white fallow deer was built and in 2005 the Minizoo was added.

The castle had to be completely rewired and re-plumbed. Central heating was installed. A restaurant, a tea room and a kiosk with outdoor seating were created. Modern kitchens were made, the existing museum was enlarged and opened to the public.