As with most invasives, it's where it grows that can often determine if it's invasive.* I did a google with the term:
pampas grass + invasive + Pennsylvania
(I think I remember you live in Pennsylvania) and it appears to be invasive there as it has escaped cultivation.* It's a big problem in California and probably other warm climates as well if it's invasive in Pa.* This Pa site lists it.
I think I also remember reading somewhere that once it's established it's difficult to dig out if you don't want it anymore.
When weeding, the best way to make sure you are removing a weed and not a valuable plant is to pull on it. If it comes out of the ground easily, it is a valuable plant.