From 1391 the castle and then already existing town belonged to the house of Lvové (i.e. Lions) of Rožmitál. Under the new masters the fortress was for the first time rebuilt in stone (the oldest parts being the tower and Rožmitálský Palace) and the surrounding marshes were changed into wide water trenches, thus giving the castle the looks of "A swan sitting in a lake". Their long rule marks the golden age of this place. Lvové were continuously rebuilding and enlarging the castle, first with the so-called Old Palace, standing separatedly over the remnants of the Romanesque Chapel. In 1523–1530, under the guidance of renowned royal builder Benedikt Ried of Piesting (or Benedikt Rejt of Pístov)
, a new palace in a mixed Gothic-Renaissance style was appended (it is named after Rejt these days).
In the second half of the 17th century, the castle was owned by the house of Šternberkové for a short time until it was bought by the Polish house of Rozdražovští of Rozdražov in 1579 who had a new Renaissance
palace built. During their rule, the castle and town were looted. The Baroque period is marked by the Hungarian house of Serényi who had the castle rebuilt (namely after the big fire in 1763) and also erected many statues in the town and its surroundings. They also completely rebuilt the church in Paštiky.
Since 1798, the castle has belonged to the Tyrolian house by origin of Hildprandtové of Ottenhausen, except the period of communist dictatorship (1948–1989) when it was confiscated by the state. In 1850–1856 the castle received its last rebuilt, in English Gothic style, which gave it its contemporary looks.