Another thing to check is your email boxes. One time my server was slow, and after we went crazy trying to figure out what it was, found out I had some of my mailboxes with a ton of spam. When the server was going through gigs of email to check them, it was slowing everything down.
I have different users in directadmin, and so it would make an email
user@whateverdomain.com and I really didn't check those.