@mybrick It is unlikely the agent can trace the accoun through the bankcard alone.
However you have another thing in your favour.When you initially authorised payment to giffgaff, it was at £5 per month.
As you did not specifically agree to the increase to £6 per month, this year, you are entitled to ask your bank to do a charge back to your account. Your Bank may at first protest but they will when you insist refund the cash.
When that happens giffgaff will stop taking payments from your bank permanently.