Sounds like the Bougainvillea caterpillar - Asciodes gordialis. As I understand it can be green or brown. It's a relative of the cabbage looper and is more common than most people think. If you do nothing your plant will recover. If you want to do something, I would recommend using BT - Bacillus Thuringiensis. Many garden centers sell it now or you could get it at a site like this.
When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.