It's great that you've told us your hardiness zone, but the sun conditions would help as well. Is it full, part or no sun?
There are reblooming and everblooming daylilies and a couple of hardy geraniums that bloom for a long time as well.
I can't help you with the delphinium question as I've never grown them. Hopefully someone else can chime in on that one.
Let us know so we may better help you.
When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.