After a Payment has been completely applied it may be deleted. Selecting Delete Payments provides the option to select and delete any single payment or, you may elect to delete all closed payments, i.e., Payments with no remaining unapplied balance.
There is no rule for when you should delete payments. However, closed payments will continue to appear on lists and reports until such time as you delete them, This can cause reports to look cluttered. So, it is a matter of preference when to delete closed payments.