I think the green fungus you see could be lichens.* Is this what you see?* If so it's harmless to the tree and feeds wildlife.* It doesn't grow where there is air pollution.
You mention dead limbs and declining fruit production but no spots or discoloring of the leaves. With that I would suspect either a fungal problem in the root system or possibly some type of borer insect.* Take a look at these to see if anything sounds familiar.
These three are about plum pox virus.
I will be leaving tonight and will be gone for a week without computer access.* If those sites aren't helpful and no one else comes along that can help, I would suggest you contact your local extension service.
When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.