The accuracy and reliability of expert advice is often compromised and needs to be interrogated with the same tenacity as research data to avoid weak and ill-informed policy, according to risk analysis scholars writing in Nature.

Though governments aspire to evidence-based policy, it is more often the scientization of politics outlined in books like Science Left Behind; because the researchers say the evidence on experts themselves actually shows that they are highly susceptible to "subjective influences" and
"cognitive frailties"
- from individual values and mood, to whether they stand to gain or lose from a decision.

Due to that, experts often vastly overestimate both their objectivity and the reliability of their peers.  It's found to be so.  Highly regarded experts are routinely shown to be no better than novices at making judgments.  

To resolve that, the scholars caution against letting such experts advise decision makers directly, which can skew impartiality. That means we need a straightforward framework for improving expert advice, in order to alleviate the effects of psychological and motivational bias. 

"We are not advocating replacing evidence with expert judgments, rather we suggest integrating and improving them," write professors William Sutherland and Mark Burgman from the universities of Cambridge and Melbourne respectively.  "Policy makers use expert evidence as though it were data. So they should treat expert estimates with the same critical rigor that must be applied to data.

"Experts must be tested, their biases minimized, their accuracy improved, and their estimates validated with independent evidence. Put simply, experts should be held accountable for their opinions."

Several processes have been shown to improve performances across the spectrum, they say, such as 'horizon scanning' - identifying all possible changes and threats - and 'solution scanning' - listing all possible options, using both experts and evidence, to reduce the risk of overlooking valuable alternatives. 

To get better answers from experts, they need better, more structured questions, say the authors. "A seemingly straightforward question, 'How many diseased animals are there in the area?' for example, could be interpreted very differently by different people. Does it include those that are infectious and those that have recovered? What about those yet to be identified?" said Sutherland, from Cambridge's Department of Zoology.

"Structured question formats that extract upper and lower boundaries, degrees of confidence and force consideration of alternative theories are important for shoring against slides into group-think, or individuals getting ascribed greater credibility based on appearance or background," he said.

When seeking expert advice, all parties must be clear about what they expect of each other, says Burgman, Director of the Centre of Excellence for Biosecurity Risk Analysis. "Are policy makers expecting estimates of facts, predictions of the outcome of events, or advice on the best course of action?"

"Properly managed, experts can help with estimates and predictions, but providing advice assumes the expert shares the same values and objectives as the decision makers. Experts need to stick to helping provide and assess evidence on which such decisions are made," he said.

Sutherland and Burgman have created a framework of eight key ways to improve the advice of experts. These include using groups - not individuals - with diverse, carefully selected members well within their expertise areas.

They also caution against being bullied or "starstruck" by the over-assertive or heavyweight. "People who are less self-assured will seek information from a more diverse range of sources, and age, number of qualifications and years of experience do not explain an expert's ability to predict future events - a finding that applies in studies from geopolitics to ecology," said Sutherland.

Added Burgman: "Some experts are much better than others at estimation and prediction. However, the only way to tell a good expert from a poor one is to test them. Qualifications and experience don't help to tell them apart."

"The cost of ignoring these techniques - of using experts inexpertly - is less accurate information and so more frequent, and more serious, policy failures," write the researchers.