Refurbished phones are processed by a subcontractor. Agents are administrators not technical experts, so are probably inclined to believe what the subcontractor says.
What precisely did Apple say was the fault with the phone and did you get the statement in writing? Vague statements like "a significant fault" do not help us or the agent to understand the problem, especially if only given verbally.
Did the agent state why the fault was not covered by the warranty? If not, ask the agent again and insist that they provide an exact description or a letter of deadlock.
If the agent still refuses to provide a reason, and you're certain you've exhausted all steps that can be taken via the agents. you can invoke the official complaints procedure. This is stated at https://www.giffgaff.com/support/complaints. It will cause your problem to be examined by a different team within giffgaff and should in theory be able to identify any wrongdoing in dealing with you, whether the problem is with the agent or with the subcontractor.