Sounds like you have either cabbage worms aka imported cabbage worms, cabbage loopers or diamondback moth larvae.
One of the best organic sprays is one with Bt aka Bacillus thuringiensis for these particular pests.* Javelin WG is a brand name for a spray.* From this site about halfway down the page.
(The bold in the quote is mine.)
The active ingredient in Javelin WG is a toxin obtained from a bacterium, Bacillus thuringiensis variety kurstaki. The toxin is active only against caterpillar pests
, and acts as a stomach poison that must be consumed by the worms to be effective.
You can also use a ground red pepper spray or use another product called Align which contains Neem.* The problem with Neem is it targets beneficial insects as well.* You can make your own pepper spray.
Garlic sray often works for these pests as well.* Here's how to make your own.
You should find these recipes helpful.
More helpful info on natural insect control.
Enjoy your collards!
When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.