If given a choice between topsoil or composted manure as the only growing medium for veggies, I would choose the topsoil.* The best mix I can think of would be 60% topsoil and 40% compost.* If possible I would suggest you add compost.* I would also suggest a soil test.* Some veggies don't like too much nitrogen.* Corn is one that needs lots of nitrogen, but some veggies don't do well in high nitrogen soils.* Other veggies such as potatoes, cucumbers, squash, turnips, cauliflower, cabbage, broccoli, and kale don't taste good when grown in manure that hasn't been properly composted and is still 'raw'.
Maybe you can still use the composted manure and mix it with the leftover soil.* Use it to amend the veggie beds outdoors.* You might want to consider just how composted the manure is.* There can be concerns about what is in it (potential contaminates) and how well it's been composted.* I would definately urge you to be cautious here.* You might find this info helpful about manures.
Lots of info here.* It's a summary of a book.* Click on the topics on the left.
If you really want to know what's in the manure you can find a certified manure lab for your state here.
When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.