I don't see any trouble on the tree on this topic at this point. Seems like a healthy tree with a few old leaves.
And I do agree with Tropfrog about keeping your tree healthy to avoid any issue with an eventual attack.
I do spray copper sulfate on my trees once in a while, it helps to control/avoid fungal diseases and promote plant health.
A healthy tree never have problem with pests. They do get it from time to time but it never gets to infestation problems.
I just wanted to point out that I already had infestation cases on very healthy trees that I had to act, of course, healthy trees won't suffer much as unhealthy trees, and it may overcome, but why would I let it weaken anyways? As I live in Brazil, I believe I'm subjected to many insects most people don't, and sometimes the same well known ones, but on swarms... It's not usuall, but it happens.
I already saw leaf-cutter ants completelly defoliate a small tree in a single day, it survived, but not only because of it's health...
So I wouldn't say never...