After having to go through the server logs and see daily attempts on ssh i decided to make some further changes as described here. I had already done the usual, blocked use of passwords and the rest but still valuable cpu time was being wasted checking entries and denying them as they were not on the allowed list. After the latest round of changes I was wonderfully suprised to find now such entries are just refused out right with no further time wasted trying to verify them.
The next thing now is to enable encryption on a couple remaining services to try and plug up any holes that might be there.