To remove a contact from your contact list, you can do this from the contact drop-down list when drafting an email. Hover your pointer over the contact you want to remove and select "See details".
Select "Edit" in the bottom left section of the contact details and select
"Remove contact from my practice"
(If you feel like the contact is no longer in use or inappropriate for all practices not only yours, please report the email address as invalid first)
The contact has now been removed from your directory!