You have really hit the nail on the head. The difficulty occurs when one disagrees with the decision but there is no error in law: one thing I always found frustrating about administrative law (despite the merits review procedures)!
Dr Haneef was just damned lucky the Minister applied the wrong test. If he’d have applied the right test, Dr Haneef wouldn’t have had a leg to stand on.

However, it is important to note that his Honour was of the opinion that the Minister could have been entitled to cancel Dr Haneef’s visa if he had applied the proper test.

I have a reasonable amount of experience with this area of law, and in the end this remark goes to the fundamental problem from the perspective of a common law rule-of-law believer: the High Court has said that the Parliament can delegate all kinds of merits assessments to the relevant Minister/delegate who can make it freely so long as he/she swims between the flags in a jurisdictional sense. So in this case, Andrews can make an adverse assessment based on secret information from unreliable sources and it could still be valid, so long as he says he is applying the correct test.

Once you get into the realm of pure merits, i.e. you just disagree with his assessment but can’t establish a jurisdictional error in the approach taken, it’s a very high hurdle to show that a decision was so irrational/unreasonable, or alternatively was tainted by bias such that it should be disturbed by a Court.