The worms are good.* They digest organic material and their 'waste', called castings, is very good for your plants.* To remove the worms from your compost you can 'sift' the soil with some type of screen that you can make or purchase.* You may even have something that can be used such as a minnow bucket.* Screen that is used for windows would be too fine, but you can find 'hardware cloth' in hardware stores that you can purchase in just about any size with different sized holes. * Something with 1/4" holes or a little smaller would be good.* Click on 'soil sifters' below for pics and sites you can look at.
Here's an example of one you can purchase.
When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.