Wow!!! That is gorgeous!!** It's a Lycoris radiata aka spider lily and some folks call them surprise lilies.*
The surprise lilies that I have are Lycoris squamigera, but they are a pale pink and their flower is trumpet shaped.
More info on surprise lilies.
Do dig them after they bloom and replant them immediately.* Mine send up wide strap-like leaves in spring which disappear after about 2 or 3 weeks.* The bulbs are large and probably 10" deep.* They cost about $3.00 each, so get all you can.* They tend to like part sun, at least that's how I've seen them grown.* I've even seen them in shade gardens.* They do multiply over time.* What a gem you have found.
When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.