What ammo are you using? Could be a weakly charged lot of .380 causing this.
One thing I've learned from my LCP is to utilize the end of my index finger pad, about 2/3 up from the joint. Otherwise, the long trigger squeeze can make my POI lower than my POA. I don't think you need to grind your sights down. I think they've been ground down enough.