In order to refund one ascendancy passive, you need five refund points left.
An Orb of Regret grants one regular refund points. For refunding one Ascendancy passive, you need five Orb of Regret.
As the example shown below, it originally has 6 Refund Points Left. After refund one ascendancy skill, it now has 1 Refund Points Left.