There are two basic types of peony.* One is the tree peony and has twiggy stems and the other is the herbaceous peony.* It dies to the ground in the winter.* I'm thinking you have herbaceous peonies.* Here's a great site with lots of info.* Best to move them in early fall to give their roots time to establish.* They don't do well transplanted in spring.
http://www.goldenport.com/peony/hpcare.htm
Newt
When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.