Good practice criteria

This page presents a summary of the ASSERT criteria for good practices in societal impact assessment, followed by a more detailed account of each criteria.

1.      Has the SIA design the potential to reframe the project and R&D process?
– If yes, how open is the outcome of the process? What are the implications of this?
2.    Is consultation being taken seriously?
– How can this be established (agency assessment)?
– Who is in the position to define what is important and less important?
– Are the roles and responsibilities clearly defined?
3.     Is the process flexible?
– What level of flexibility is productive?
4.     Is the process iterative?
– Where in the R&D process does SIA create an impact?
5. Is the administrative burden reasonable?
– How can this be established?
6. Is the process transparent?
– Are there situations where transparency cannot or should not be achieved?
7. Are the limitations of the process explicitly stated?
– Resources, access to knowledge, temporal scope, excluded groups (ignorance)
– Communication but not marketing of the project
8. Is the prevalent understanding of societal security in a given project clearly defined?
– How does the project enhance security?
– Whose security?
9. Is societal impact clearly defined?
– Impacts (benefits, unintended consequences, harm) on individuals, households and enterprises and communities at each level
10. What kind of knowledge is being produced in the SIA procedure?
– Scientific Knowledge
– Knowledge for political and societal decision making
– Knowledge about risk
– Why is it important to define beforehand what knowledge is at stake?


SIA Good Practice Criterion 1: Does the SIA design have the potential to change the project and R&D process?

On a conceptual level, a best practice criterion for carrying out societal impact assessments in practice is to not limit them to being merely risk assessment exercises. While it is certainly important to identify risks and to take appropriate measures to pre-empt, minimise, or mitigate them, it is equally important to understand societal impact assessment as a process that broadens the range of alternatives by reframing an issue instead of sticking to “a pre-determined range of possible alternatives” (Prainsack/Ostermeier 2013: 17). The potential of the SIA design to reframe the project and R&D process is therefore the first important good practice criterion to plan and assess SIA procedures.

The extent to which there is potential to reframe the project (including the project’s goals and how they will be reached) will largely depend on how open the outcome of the process is. If very specific deliverables are promised, then the capacity to change or modify the project will be more limited than if the deliverable specification was more open, allowing for the inclusion of unanticipated findings and results. This also raises the question of whether or not there should be criteria that, if met, would lead to the abandonment of the project. While the latter is not a possibility at present in most R&D projects in the security research domain, it is important to carry out a threshold analysis at the pre-application stage to determine whether or not the inclusion of the scenario of project abandonment is necessary. Pertaining to a less ‘extreme’ scenario, and probably more important for the implementation phase of projects, is the question whether and how the core mission or stakes of and in a project can be reframed while it is under way. This question needs to be addressed both in terms of its implications for research activities as well as in terms of project management and reporting.

Closely related to the reframing of research questions and objectives is the topic of public, user, and stakeholder participation. While participative approaches in research are becoming increasingly common in both applied and basic research, reviewing the relevant literature (see Prainsack/Ostermeier 2013) one frequently encounters the concern that too often, participation is being considered not as an opportunity for genuine learning but an ‘education exercise’ to create consent and acceptance among those who may be sceptical or resistant. In other words, participation serves the purpose of reducing friction points and increasing acceptance (see also Bogner 2012). In order to facilitate opportunities for genuine learning, it is important to render power differentials visible and put them on the discussion and negotiation agenda (see also Wadhwa et al. 2014). Questions to be discussed include: Who is likely to benefit from the research that the project sets out to do? How are they likely to benefit, and at the cost of whom? Who will be empowered and disempowered by this research (see Prainsack/Toom 2010)? Who is likely to suffer adversity, and how can these asymmetries be balanced? What are the external factors of the project impacting on the research activities? The second SIA good practice criterion can therefore be identified as follows:

SIA Good Practice Criterion 2: Is participation being taken seriously?

Attempts to plan and manage the societal impact of security research projects will need to produce knowledge about how users, stakeholders, and publics can be included in the most productive manner. One way to facilitate the careful consideration of issues pertaining to the nature and format of participation is to explore the distribution of power and agency following the guidance by Prainsack (2014). In cases where such a deeper assessment cannot be carried out, it is important for SIA plans to at least make explicit who is in the position to define what is important and less important. Furthermore, the roles and responsibilities of all project participants (and users and other stakeholders that are included in the project) need to be clearly defined from the beginning on. Any kind of user, stakeholder, or public participation should be conceptualised as a process that is supposed to have an impact on security research and not to create acceptance or reducing opposition. It is in this line of thinking that “a dialogue among different individuals and groups who are considered (or consider themselves) as potentially affected by a planned project is a benefit in itself.” (Prainsack/Ostermeier 2013: 5) It needs to be emphasised, however, that it is important for SIA planners and managers to act upon the constructive ‘irritation’ that this dialogue can create (e.g. by adjusting or amending the project’s goals, methodologies, user engagements, etc.).

The conceptualization of SIA as a genuine learning process as opposed to a risk management process constitutes a major difference that allows for the identification of a number of more specific best practice criteria, reflecting reservations against an overly instrumental take on SIA. One of them is flexibility that should allow to sufficiently acommodate the impact of a SIA process.

SIA Good Practice Criterion 3: Is the process flexible?

Project managers and researchers will need to determine the level of flexibility that ensures a productive SIA process. Closely related to the need for flexibility is the requirement to design SIA processes in a way that ensures their implementation from the early stages of the research projects, and their ability to feed the results into the R&D process.

SIA Good Practice Criterion 4: Is the process iterative?

Deliverable D1.2 of the ASSERT project concluded that “In this spirit, the main outcome of ASSERT should be a resource that requires and structures critical reflection, guiding and facilitating the conduct of societal impact assessment…” (Prainsack/Ostermeier 2013: 5) The demand for an iterative process however bears the tendency to create a second layer of project planning and management. This can a burden for research budgets and another strain on the limited time that researchers have available. Taken together, the requirement of rendering processes iterative can pose a risk for the acceptance of SIA plans, and lead to their perception as a burden rather than a benefit. [D2] What every SIA can do, however, is to facilitate iterative processes wherever it is meaningfully possible (e.g. where project participants support this and see this as a benefit).

SIA Good Practice Criterion 5: Is the administrative burden reasonable?

One way to achieve the goal of keeping the administrative burden that a SIA adds to a project very limited, or – even better – to design SIA in such a way that it helps to decrease administrative burdens throughout the project, is to redesign existing assessment processes that are already in place in some institutions or to broaden/adjust the scope of, for example, ethical review procedures. ASSERT argues that good practice criteria can be helpful both for developing SIA plans that facilitate both genuine learning and the redesigning of existing assessment processes to contain or reduce administration costs (in the wide sense of the word). Another important measure is to be very explicit about why, at certain stages in the process, SIA might create extra work, and how this benefits the overall project. If the benefit is unspecific or unclear, SIA planners and managers should consider cancelling the effort. Scaling down the scope and impact of SIA during the course of a project belongs to the abovementioned flexibility requirement in the same way that scaling it up does.

The research carried out within ASSERT highlights that it is important to understand the difference between SIA on the one hand, and evaluation on the other, when planning or managing SIA processes. Although the two processes can shade into one another, their points of gravity are different: While SIA has a prospective scope, evaluation has a retrospective scope. It is therefore important to define what the focus of the prospective scope should be: unintended consequences, questions regarding who benefits, or positive outcomes? This process includes far-reaching decisions about what needs to be known, what can be left aside, and possibly even what should be concealed in SIA processes. Furthermore, it should be made clear in the SIA plan who has access to which kind of knowledge, and to highlight to what extent transparency is desirable: are there situations where transparency cannot or should not be achieved? Rappert (2012), for example, has shown how ignorance and concealment contribute to the production of order in negotiation and assessment procedures in the international security policy making domain. This approach and the literature on “ontopolitics” (Mol 2003) can make a fruitful contribution for conceptualising transparency in SIA processes by showing that transparency and accountability always involve decisions about issues that should not be made transparent.

SIA Good Practice Criterion 6: Is the process transparent?

One crucial example for the importance to consider the scope of SIA on the time scale is the question whether or not a SIA plan should go beyond the formal end of a project (e.g. after funding has ended). While there are good reasons for why this should be the case, mainly because the results of projects are likely to continue to impact on society in various ways after the project has concluded, budget limitations render it unlikely that serious attempts to manage the impact of the results will be made. Awareness and transparency about such limitations increases both the transparency of the SIA procedures as well as an understanding of its limitations. A clear statement about the societal impacts that have not been taken into account in the SIA management plan can help to avoid misunderstandings about the remit and scope of a particular SIA process. A relevant example for this is the proliferation of security technologies in countries outside of the European Union. NGOs like Statewatch have repeatedly voiced concerns that efforts to strengthen the consideration of ethical and societal issues in security research are of little value if findings and products are developed for export to markets outside of the European Union.[1] The transnational dimension of impacts tends to be neglected in the SIA literature.

SIA Good Practice Criterion 7: Are the limitations / the scope of the process clearly stated?

Besides the inevitable limitation that access to relevant data and information tend to be restricted in many contexts in the security domain, other likely limitations pertain to resources and the temporal scope. Moreover, related to user involvement, stakeholders, and publics, even the most participatory approaches will have limitations, which should be openly talked about (also with users, stakeholders, or members of the public, before they agree to participate). The latter point is also relevant regarding the objectives of the dissemination strategy, which should be aimed at communicating the results of the project and the SIA process, but not to simply ‘sell’ it. Considerable emphasis has been put (in our workshops) on the use of social media as a means to communicate with the broader public beyond the SIA processes.


Qualitative Issues of SIA

While the first seven criteria have primarily addressed issues of knowledge and power as well as conceptual issues, the final three criteria address qualitative issues of SIAs. A general requirement that can be inferred from the literature and the research carried out within ASSERT is the need to clearly define how societal security is understood and how the societal impact of a research programme, a project, or a product, is conceived. While the societal security concept lacks a coherent definition, its benefit for SIA is that it refers to the procedural dimensions of security (Buzan et al. 1998). Authors working in a project on European Security Trends and Threats in Society (ETTIS) have defined the term societal security as signifying “ultimately what different societal actors perceive as societal security. By its very nature, societal security is continuously in the making, and it harnesses a variety of stakeholder perspectives” (ETTIS 2012: 4). In general, according to the ETTIS consortium, the concept societal security is concerned with four aspects: definitions, dimensions, sources and societal security strategies & governance (ibid.). The report provides for a working definition describing societal security as “the security of societal sources of human well-being in general, and the societal sources of individual security in particular” (ETTIS 2012: 23, original emphasis).

SIA Good Practice Criterion 8: Is the prevalent understanding of societal security in a given project clearly defined?

Applying the four aspects of societal security identified in the ETTIS Report to the SIA debate, they require SIA planners to consider and make explicit (a) how societal security is being defined and delineated, (b) the scope of the concept, (c) the factors that impact on societal security, and (d) how “strategies can be devised and implemented at any level … in a coordinated and coherent manner in order to increase security” (ETTIS 2012: 7). In short, explicitly reflecting on how a project impacts on societal security (in its prevalent or relevant iteration) requires SIA planners to reflect the implicit understanding of security and how their project is related to this understanding. This opens security research to a broader understanding of both securities and societies. The concept societal security is also helpful in overcoming what has been called a technology fetishism in security research, suggesting ever more technological ‘fixes’ for inherently societal issues (see Hayes 2009 and IRISS 2012).

In Deliverable 1.2 we emphasised the need to render SIA an assessment of societal impacts, including a much broader scope of SIAs than approaches referring to social impact assessments. SIA planners and implementers should thus be asked to specify how they conceive of societal impacts of the project. Broadly speaking, societal impacts can be understood as the impacts (e.g. benefits, unintended consequences, harm) on individuals, households and enterprises and communities at each level of society. It is important to avoid limiting the scope of an SIA plan to those directly and obviously affected by a project but to include also those who may be affected indirectly. In addition, there is a need to identify the range of societal impacts that are considered and dealt with in an SIA plan. Wadhwa et al. (2014) provide for a detailed step-by-step guidance on how to identify societal impacts.

SIA Good Practice Criterion 9: Is societal impact clearly defined?

Finally, our consultations with experts and stakeholders have consistently shown that there is a need to specify the kind of knowledge that is being produced in a SIA. This is important also for the purpose of expectation management. Cashmore (2004), for example, distinguished the purpose and the role of the knowledge produced in Environmental Impact Assessments. By making this distinction, he pointed out to the contradiction that impact assessments are at the same time expected to produce objective knowledge or ‘facts’ about the projects that can at the same time be used for decision making in inherently political matters. Citing Ulrich Beck, who claimed that “scientific rationality without social rationality remains empty; but social rationality without scientific rationality remains blind” (Beck 1992), Cashmore argued that in impact assessments, “social science theory becomes more than simply an adjunct to a technical process; it becomes integrated with, and integral to, EIA theory concerning the role of science” (Cashmore 2004: 413).[2] This leads us to our final criterion.

SIA Good Practice Criterion 10:What kind of knowledge is being produced in the SIA procedure?

Projects like VALUESEC have shown that impact assessments will always need to include the gathering and analysis of qualitative data, if not solely, in addition to quantitative data.[3] SIA planners and implementers need to be aware of the differences between scientific knowledge, knowledge for political and societal decision making, and risk management knowledge. Considering the type of knowledge helps to avoid pre-determining the outcome of the SIA by using inadequate causal assumptions and definitions (Cashmore 2004: 422). Precisely for the reason that SIA cannot and should not meet academic standards in knowledge production and analysis, it is capable of accommodating different societal realities of R&D projects (ibid).

[2] The acronym EIA in this citation refers to Environmental Impact Assessment.