Hi Karen
There are two reasons for this behavior.
If there is something wrong with the 5th swan, i.e., sick, injured or the runt that can’t keep up with the family, the parents will try to banish it from the family. Nature is not kind. The family or flock will banish a member if there is a possibility its state of poor health or weakness could allow predators to be alerted and follow the rest of the family/flock.
The second reason could be the age and gender of the fifth swan. If the cygnets are turning white, the parents will begin to chase them from the area as they are seen as competition, especially if this cygnet is the oldest male. The Regal Swan