I found that Veronica didn't do well in my garden either and I grew several varieties.* You don't say which one(s) you have.* My garden is enclosed by a wall and is a bit shady and damp in the winter.* I also think the soil was too rich with compost.* They seem to like it a bit on the lean side.
You might find these conversations helpful too.
When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.