Good scientific policy advice: the IAB’s five principles

Policy advice is an integral part of policy-making in contemporary democracies and a central task of the IAB. But what makes “good” policy advice? Even if the quality of individual advisory services cannot be measured, quality criteria can be identified and quality assurance methods can be outlined.

Reviews in the form of rankings and league tables abound. They range from the everyday and entertaining to politically significant issues with far-reaching repercussions. It turns out that in many areas, quality and performance measurement is a highly demanding, if not impossible undertaking: An example is scientific policy advice. Alongside the processing and provision of data and research in the narrower sense, policy advice is one of the three core tasks of the IAB.

What distinguishes “good” scientific policy advice? Can the quality of policy advice be measured by rating advisory services on the basis of benchmarks? In a word: no. At least not in the sense that a measure is put in place retrospectively which determines whether an advisory service was good or bad. Nevertheless, steps can be taken in advance to create favourable conditions for good policy advice. This was the conclusion of a working group at the IAB, which discussed the questions raised above.

Scientific policy advice: bridging the gap between science and politics

Scientific policy advice mediates between science on the one hand and politics on the other. It makes scientifically-based findings and assessments of politically relevant issues available to politicians and administrations, for example through meetings, expert commissions and parliamentary hearings; or in the form of lectures, assessments and reports.

The intermediary position between science and politics results in the requirement that the knowledge transferred from science to politics, the so-called advisory knowledge, “must both meet scientific standards and be politically effective”, as the policy advice guidelines from the Berlin-Brandenburg Academy of Sciences (available only in German) formulate. For scientific policy advice, therefore, there is a double quality requirement which includes both scientific and political appropriateness. The IAB guidelines on good scientific policy advice (available only in German) reflect this idea by stating that scientific advice must be “scientifically appropriate and workable within the political process”, with the word “and” explicitly emphasized.

Policy advice is “scientifically appropriate” when it reliably presents situations, contexts and relationships. This includes maintaining independence and impartiality. Advice should also be based on empirical data which has been collected and analysed in a methodologically robust way. Finally, the relevant theoretical perspectives should be taken into account, and the knowledge gained should be embedded in discussion within the scientific community.

Policy advice is “politically appropriate” if it takes the needs of the user into account and if the scientific advice provided relates to positions and values within the political debate. The advisory knowledge and the recommendations arising from it do not have to meet the approval of the recipients. They should, however, connect with their thinking, and, at least, the underlying empirical findings should be recognized as valid by them. By providing knowledge and advice to policy-makers, as well as pointing out alternative courses of action, scientific advice supports political decisions without absolving decision-makers of their responsibility.

Scientific uncertainty and political ambiguity rule out a measurement of quality

It is, however, not possible to create indicators based on the criteria for scientific and political appropriateness in order to be able to actually measure the quality of policy advice. This is due to two fundamental structural features underpinning scientific policy advice: the uncertainty of scientific knowledge and the ambiguity of political goals. While the uncertainty of scientific knowledge is reflected within the caveated nature of scientific advice, the ambiguity of political goals means that the purpose of policy advice is also ambiguous.

Insights of science are by no means as clear and definitive as sometimes presented in public discourse. According to the philosophy of science research results are always preliminary and only valid until they are refuted. In addition, factors that were not considered in a study could be responsible for an observed outcome. Finally, studies that investigate causes and effects may produce different results even with the correct application of scientific methods.

If the scientific knowledge itself is on somewhat shifting ground, the purpose of its use cannot be clearly determined. Scientific policy advice can uncover social problems, suggest ways of problem-solving and make the political discussion more informed. However, political advice can also lend technical authority to normative convictions, legitimise political decisions and give the impression that politics is actively dealing with particular issues.

The various functions of policy advice reflect a fundamental tension within policy-making that is both about solving social problems and striving for power. Both dimensions are intertwined. Anyone who has political power can, for example, define a situation that is considered undesirable as a social problem and put it on the political agenda. The definition of a social problem is thus always associated with certain values. For example, the point at which social inequality is considered a social problem depends on political positions and values.

Five principles of good scientific policy advice

Scientific policy advice eludes an indicator-based measure of quality which can claim general validity across different advisory services. But given that this is the case, should one give up trying to make a judgement on the quality of advisory services? And even without uniform indicators, is there a way to ensure the quality of advisory services? The members of the working group ”Quality of Policy Advice” at the IAB asked themselves these two questions and answered the first with “no,” and the second with “yes.”

Following a detailed discussion including the participation of the members of staff through two in-house surveys, the working group identified five principles of quality assurance, including five measures for implementing them, respectively. Recorded in a ”policy advice concept” (available in German), they form the foundation for good scientific policy advice at the IAB.

1. Transparency through publication

The policy advice concept first demands that the source of the advisory knowledge is made transparent. In addition, the advice knowledge itself should be publicly available. Both requirements can be met through publication. In this way, the advisory knowledge can be reviewed and make a contribution to public discourse. It is not necessary that the advisory service itself takes place in public. The medium in which the findings are published is also not decisive. Nevertheless, publications with their own review process contribute to quality assurance by setting minimum standards for publication.

2. Clarity through collegiate peer review

Whilst the peer review of scholarly articles is an external quality control which occurs in specific cases, collegiate peer review is a tool for the internal quality assurance of policy advice. By reviewing and discussing findings and documents, colleagues examine both the substantive argumentation and scientific approach, as well as the comprehensibility of the advisory service. Revisions should help to make the language comprehensible, graphs clear and the reasoning appropriate to the needs of the addressees of policy advice.

3. Political understanding through training

In addition to a user-friendly explanation of the advisory knowledge, the advisors should have comprehensive knowledge of the policy and the advisory process in order to be able to carry out advice in an appropriate manner. They should have a fundamental understanding of the advisory setting and the structures that both limit and enable political actors to make decisions and fulfill actions. Relevant further training measures should provide advisory staff with the knowledge and skills that facilitate confident presentation and argumentation, as well as lead on to an understanding of the perspective and priorities of the addressees of policy advice.

4. Applicability through preliminary clarification

In order to meet the requirements of the recipients, it is important to clarify the remit of the assignment. Prior discussions on the scope of the commission should ensure that the scientific advisor can actually provide the policy advice requested, and that it is useable for the recipient. Alongside the wishes of the addressees, the advisor’s scope should also be discussed. On the one hand, fundamental questions around political ambiguity and scientific uncertainty should be clarified here. On the other hand, practical questions should be answered: these relate for example to the specification of the topic of investigation, the scope and timing of advisory services, or the availability of empirical data for scientific analysis.

During the course of clarifying the commission, the actors involved can also agree to change, expand or drop the advisory assignment. In every case, a shared understanding of the commission should be established, which creates clear expectations and thus guards against disappointment on both sides.

5. Reciprocity through regular discussions with the addressees of policy advice

Finally, regular discussions with the addressees of policy advice should enrich the mutual exchange between advisors and recipients. Whilst advisory assignments are being agreed individually, regular discussions take place independently of a specific occasion. They focus on fundamental questions of policy advice and the design of the advisory relationship.

The IAB has recurring as well as permanent contacts with many recipients of its policy advice – in particular various organizational units and committees in the German Federal Employment Agency and the Federal Ministry of Labour and Social Affairs. In this context, the purpose of a mutual exchange is to achieve a common understanding on the quality of policy advice services; to learn from each other; and to create and maintain mutual trust.

Conclusion

Trust begins where control ends. This applies especially to the relationship between advisors and addressees of policy advice. Basically, the addressee, on the one hand, has to trust that the advisor will deal responsibly with scientific uncertainty and that the advisory knowledge is in line with the current state of research. The advisor, on the other hand, has to trust the recipient to deal honestly with political ambiguity and not to instrumentalise the advisory knowledge in a distorting way within the political process. Further, trust is required when it comes to the quality of policy advice. Here too, control reaches its limits because ultimately there can be no reliable indicators for assessing quality. Nevertheless, favourable conditions can be created for “good” policy advice. With this in mind, the policy advice concept of the IAB names five quality principles and measures for their implementation.

With the policy advice concept, the IAB applies quality assurance both to the prerequisites for good advice and to the advisory process. But quality assurance indicators which relate to the output or the outcome of an advisory service are not meaningful: performance measurement should be reserved for areas where clear measurement criteria exist. Indeed, uniform indicators would provide the numerous addressees of IAB’s policy advice with a fictitious objectivity and ultimately harm rather than benefit the goal of providing appropriate scientific and political advice.

Literature

Bähr, Holger (2017): Die Qualität wissenschaftlicher Politikberatung: Unbestimmtheit durch wissenschaftliche Unsicherheit und politische Mehrdeutigkeit. In: Leviathan, Vol. 45., No 3, pp. 404–427.

Berlin-Brandenburgische Akademie der Wissenschaften (2008): Leitlinien Politikberatung:  11 June 2018.

Institut für Arbeitsmarkt- und Berufsforschung (2018): Konzept des IAB zur Sicherung der Qualität wissenschaftlicher Politikberatung: 11 June 2018.

Institut für Arbeitsmarkt- und Berufsforschung (2010): Leitlinien guter wissenschaftlicher Politikberatung für das IAB: 11 June 2018.

Lentsch, Justus; Weingart, Peter (eds.) (2011): The Politics of Scientific Advice: Institutional Design for Quality Assurance, Cambridge: Cambridge University Press.

Pielke, Roger A. (2007): The Honest Broker: Making Sense of Science in Policy and Politics, Cambridge: Cambridge University Press.