AgriFoSe2030: Systematic Review and Bibliometric Analysis
Section outline
-
Systematic Review Course for AgriFoSe2030
-
-
-
In short, systematic reviews are about doing research of what is in the literature, with a set of explicit methods developed and adjusted to different disciplines.
A systematic review has a clearly formulated question, pre-defined selection criteria and uses systematic and explicit methods to search for, select and critically appraise relevant research. The methods used for carrying out a systematic review must be worked out in advance and constitute what is, in the context of systematic reviews, called the protocol. In some cases, the protocol must also be published in advance, depending on the guidelines for the journal you want to publish in, or the research organisations you cooperate with. The idea behind this is that the protocol in itself can be of value for other researchers on similar subjects, and the registration or publication of it also serves to avoid duplication of almost identical investigations. Here is an example of a systematic review protocol.
The most well-known examples of systematic reviews are perhaps Cochrane reviews in medicine and healthcare. Systematic review methods were first elaborated in medicine and healthcare, but have later been developed and adapted for use in other research areas.
A full-scale systematic review takes a lot of time and resources to perform. However, in this course we want to emphasize that the principles for systematic searching can be applied also in smaller studies to make them more reliable.
The main principles (cornerstones) of systematic reviews are that they need to be comprehensive, transparent, reproducible and designed to minimize bias. But what do these principles mean?
-
Comprehensive
The main goal of a search strategy for a systematic literature research question is that it is comprehensive and exhaustive. The idea is to find as much relevant information as possible, and also to be able to estimate to what extent that goal is fulfilled. The aim is to maximize sensitivity to include all relevant results.
-
Transparent
Careful documentation is very important! You need to document the choices you have made throughout the process, explaining both what you have done and what you have chosen not to do (e.g. choices done due to time constraints that may lead to increased risk of bias). It is highly recommended to keep track of documentation via a reporting standard, e.g. the ones provided by Prisma or ROSES. More about this later in the course.
Photo by Markus Spiske on Unsplash -
Reproducible
Documentation is also important in order to be able to reproduce the review. It's not unusual that systematic reviews are updated after some years to check if new evidence is available to answer a research question, or built upon to develop new systematic reviews.
-
Designed to minimize bias
For the credibility and objectivity of the systematic review, minimizing bias is a very important part of the methodology. Below are some examples of how to reduce the risk of bias:
- Search multiple databases. Different databases cover different journals, and you will get new references if you add another database, even if they also overlap.
- Include grey literature, e.g. reports and conference material, to avoid publication bias.
- Include more than one language when searching the literature, if possible.
- Be at least two reviewers when screening publications, and preferably add a reference group with mixed expertise.
-
-
-
It is crucial for the success of your systematic work that you define and limit your research question clearly and in advance. If you try to capture everything about several aspects of your research at the same time, the result will be very cumbersome to handle.
The research question should be clear and focused - not too vague, too specific or too broad. If you have not defined it precisely, you will not be able to know how to search for it in an exhaustive way, nor what information is irrelevant in the search results you get.
Examples of research questions (from published systematic reviews):- The multifunctional roles of vegetated strips around and within agricultural fields
- What is the scientific evidence with regard to the effects of forests, trees on human health and well-being?
- What are the effects of agricultural interventions that aim to improve the nutritional status of children?
It is good practice to start by doing a scoping review– a preliminary search that can help you define your research question and its limits, and also estimate how much time and resources it will demand. You can see the assignment exercises in this course as a scoping review for your current project, in order to get a better idea about how to deal with your research question. If your project is already underway, we hope that they will still be useful for coming projects.
What kind of systematic work are you planning for? The classic full systematic review format is extensively used in medicine and other health-related disciplines, and is also supported by the organizations mentioned on the next page, Organizations that work with systematic reviews. However, it is not always the best, or sometimes not even a possible, format to choose. A great number of other, more or less well defined types of syntheses have been suggested and also published. This page contains a list of some often used types of synthesis, including links to publications that describe the ideas behind them in more detail.
To select the best format, consider these factors:
- The research question
- Is it a question that can have an answer, in a qualitative or quantitative way (that is not to say that it must be answered in your work, sometimes the result is that there is not yet enough evidence available to answer). Or is it a question with an open end, where the result could be some kind of mapping, categorization, or finding clusters and gaps in the evidence? In the first case, you can aim at a full review, in the second a systematic map is more suitable.
- Available resources
- In general, a systematic review takes more time and effort, since you are supposed to evaluate the evidence you find, not only map it or categorize it. Of course, the demand for resources also depends on the size of the search result. That is one reason that it is important to do test searches in advance of planning, in order to have an idea about how many publications you and your co-workers will have to screen and maybe also evaluate. If you find that the test searches indicate you can't finish a full systematic review, or you need to come up with a result in a shorter time than you have for the review, this is not at all a very unusual situation, and there are ideas about how to still do the best of the situation. Please consult our list below for some alternatives. You can also find many more ideas in the literature and on websites from systematic reviewing institutes.
“A systematic review attempts to collate all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research question” ( The Cochrane handbook (Links to an external site.) )
- A protocol, describing the scope and strategies for the review, should be published in advance
- The eligibility criteria contain statements of how a study has been conducted in order to be included, and all included studies are critically appraised by the review team
- The specific research question must be a question that can be answered (or could, if enough evidence was found)
- The result is often a meta-analysis or other analysis built on data extracted from the included studies. (Requires full text! And maybe even access to the underlying data!)
McKinnon et al (2016). ”What are the effects of nature conservation on human well-being? A systematic map of empirical evidence from developing countries." Environmental Evidence 5:8.Systematic mapping does not attempt to answer a specific question as do systematic reviews, but instead collates, describes and catalogues available evidence relating to a topic or question of interest. It has been frequently used to identify:
- evidence for policy-relevant questions
- knowledge gaps (to help direct future primary research)
- knowledge clusters (sub-sets of evidence that may be suitable for secondary research, for example systematic review)
https://doi.org/10.1186/s13750-016-0058-7This diagram provides an overview of the results from a very well-documented systematic mapping project.
Ten named conservation interventions has been mapped against ten dimensions of human wellbeing, and the strength of the green colour for each combination shows where gaps and clusters of evidence can be found.
In many systematic reviews, the term "scoping review" is used to describe preparatory searches, conducted in advance of a systematic review, in order to find suitable search terms, relevant databases and other sources, and to get a grip of the resources needed to carry out the review. But it can also be used almost as a synonym for systematic mapping.
“While scoping reviews may be conducted to determine the value and probable scope of a full systematic review, they may also be undertaken as exercises in and of themselves to summarize and disseminate research findings, to identify research gaps, and to make recommendations for the future research.”
Peters et al (2015). Guidance for conducting systematic scoping reviews . International Journal of Evidence-Based Healthcare 13:141–146.
https://doi.org/10.1097/xeb.0000000000000050Narrative analysis
"Narrative methods of synthesis can be used to synthesise both quantitative and qualitative studies and have been used when the experimental and quasi-experimental studies included in a systematic review are not sufficiently similar for a meta-analysis to be appropriate”
Journal of Development Effectiveness, 4:3, 409-429.
Snilstveit et al (2012): Narrative approaches to systematic review and synthesis of evidence for international development policy and practice.
https://doi.org/10.1080/19439342.2012.710641
Realist review
”Realist review has emerged as a strategy for synthesising evidence and focuses on providing explanations for why interventions may or may not work, in what contexts, how and in what circumstances”
Implementation Science, 7:33.
Rycroft-Malone, J. et al (2012): Realist synthesis: illustrating the method for implementation research
https://doi.org/10.1186/1748-5908-7-33
It is quite common that evidence is urgently needed and there is just not enough time or resources to conduct a full systematic review. How can you still do the best possible, and how can you know to what extent the results can be trusted? This is what rapid review methods are about, and also for this there is a lot of research to rely on. This example is from Cochrane:
- Rapid review: “A rapid review is a form of knowledge synthesis that accelerates the process of conducting a traditional systematic review through streamlining or omitting specific methods to produce evidence for stakeholders in a resource-efficient manner.”
Garritty, C. et al. (2020) Cochrane Rapid Reviews. Interim Guidance from the Cochrane Rapid Reviews Methods Group.
https://methods.cochrane.org/sites/methods.cochrane.org.rapidreviews/files/uploads/cochrane_rr_-_guidance-23mar2020-final.pdf [2022-08-15]
Another way of attacking lack of resources and lack of time is to do your review as systematically as is possible, and describe the work as a systematized review, including comments on what was systematic in the work and where it was not possible to fulfill all criteria. Here are two examples of how to comment such situations:
1 Author
”Because the author conducted the review as a solo project, it should not be considered a systematic review, but rather a systematized review”
Limited search
“uses methods from a systematic review, including the use of a review protocol in which the search strategy, inclusion/exclusion criteria and extraction methods are determined in advance. However, like a rapid review, the completeness of searching was determined by resource constraints”
Whether you are working with a research question from a well funded project and together with a big team with lots of resources, or running a solo project on borrowed time, your review will be more reliable and useful if you take into consideration comprehensiveness, objectivity, transparency and reproducibility, and many well defined ways exist regarding how to handle the conflicting demands of the theory of an ideal review, and the real world.
In a structured search strategy, the search terms are grouped in search blocks, each representing one of the elements or concepts that must be included in a text for it to have relevance for your research question.
The search terms within a block are separated from each other by the Boolean operator OR, and the blocks are separated from each other by the Boolean operator AND. Read more about Boolean operators in Search tips. By organizing your search in this way, you gain control of how e.g. adding or deleting terms relating to any one of the mandatory elements affects the result of the whole search. A structured block search is easier to survey, and will reduce the number of irrelevant hits, which is an important aspect since searching with the aim to be exhaustive often renders a very great number of hits. By defining your research question well, you also make it possible to create a good, relevant and structured search query.
PICO is one example of a way to structure a research question, which is very commonly used in medicine and neighboring disciplines. PICO consists of the following search blocks:
Patients, Population or Problem
Intervention or Exposure
Comparison
Outcome
When translating this into a search query, it is sometimes not possible to use all of the aspects. For example the search block C (comparison): if the comparison concept is only "not the intervention" or "no exposure" or "normal treatment", it is not useful to include in the search strategy. Words like "normal" or "usual" etc. are too vague and too common, and if you try to capture all possible expressions for normal treatment explicitly, it will commonly result in a very long search block, still missing some expressions used in publications, and in the end might leave out some important results in spite of your great effort. In this case it is better to do a search for search block P, I and O, and express the criteria about comparison as an inclusion or exclusion criterium instead. Read more about inclusion and exclusion criteria on the Screening page. Screening of results
An overwhelming part of health research questions, and also questions about social interventions, fits readily into this PICO scheme. With a little creative thinking, many other questions can be squeezed into a similar scheme as well. But some research questions definitely do not make sense for the PICO scheme, and that is why we prefer to talk about structured research questions and search queries as a wider concept rather than restricting it to the specific PICO aspects.
The film below talks about how to structure different blocks of search terms.
PICO is not applicable to the research question in the film, that's why we use other names on the search blocks.From research question to search query
The picture below shows some other ideas for structured research questions and search queries. Basically, if you wish, you can create your own acronym, the basic idea is that the blocks should reflect the mandatory concepts of the research question you have chosen for your systematic work.
Even if you perform a systematic search, it is not enough for it to be a systematic review. You also need to treat your search results in a systematic way. The screening according to defined selection criteria is where the final weeding takes place.
The very comprehensive search that you need in order to be certain that nothing important is missed means that screening the search results to find the relevant articles will be extensive and sometimes tedious. Some of the limits you have identified for your review might be risky to address in the search strategy as they could make you miss important articles.
In order to work with this in a structured and reproducible way it is recommended to define, in advance, inclusion and exclusion criteria for your systematic review, and all screeners must use them in the selection process.The selection criteria are an important part of the search strategy. They are a possibility to make sure the team agrees on how the research question or search terms should be interpreted in the study.
The following is an example of the selection criteria for a published study with the title: “ Corn yield response to winter cover crops: An updated meta-analysis”*
- Yield records came from corn following a cover crop treatment, and corn following no cover (NC)
- Yields were reported in more than one year or location
- Enough information was provided to compute study variances
- The studies were conducted in the United States or Canada
*Marcillo, GS and Miguez, FE (2017). Corn yield response to winter cover crops: An updated meta-analysis. Journal of Soil and Water Conservation 72(3): 226-239.
DOI: https://doi.org/10.2489/jswc.72.3.226
-
-
No analysis is better than the data it is based on, so it is important to find all relevant information for your research question. We will look at methods to create a structured search strategy, to find search terms, keywords and synonyms, and to test that your search strategy will retrieve relevant search results.
"Precision vs. Sensitivity - Key to effective searching (Links to an external site.) ", University of Toronto libraries
You recognize this image visualizing the differences between precise and sensitive searches from earlier in the course . Usually, when searching the literature you only need some relevant results to refer to, or to explain something - you look for a not-too-overwhelming number of articles that nevertheless capture the essence of what is known in the matter. You take care not to expand the search with unnecessary words and to limit it appropriately. That is what can be called a precise search, and is represented by the green circle in the picture above. But in order to be systematic, you want to find everything that relates to your research question, and you need to design a more sensitive search. The wider, violet circle on the picture contains a lot of green dots representing relevant results, and you want to capture these as well as the more self-evident ones in the green circle. But this also means that your search will retrieve a lot of irrelevant results to weed out.
You need to use both free-text searching and controlled vocabulary.
Use a wide variety of search terms for each concept, combining natural language and database-specific subject headings (thesauri, controlled vocabulary). In most databases today, you have to rely to a great extent on free-text searching, i.e. use words that may appear in the title, abstract or keywords of the article. Even if the database provides keywords, they are usually few, and assigned through language algorithms rather than manual work. You can use them in your search but be aware that they are only hints, not tested against the meaning of the text, and sometimes quite misleading.
There are, however, still a few databases with more ambitious indexing. Medline/PubMed, for example, is fully indexed according to a thesaurus called MeSH, and Embase according to the EmTree thesaurus. There are also databases indexed according to biological taxonomy (for example CABI and Biosis), which can be extremely useful in environmental and ecological searches. The terms in all these controlled vocabularies are ordered in hierarchical tree structures. When you select a more general term for your search from the thesaurus, the more specific (narrower) terms from the same branch will automatically be included in the search. This is referred to as “exploding” a thesaurus term. However, to perform a sensitive search, all relevant variations or synonyms of the narrower terms must also be searched as free text keywords in the title or abstract, in addition to relying on the exploded thesaurus term.
Thesauri are great tools, not only for searching, but also to browse and find search terms in a hierarchical context. But never trust that the thesaurus' terms have been used consistently in the database! For a systematic search, you should always combine with free text searching, using both the thesaurus terms and all synonyms that you can think of.
If you want to read more about this methodology, we recommend Brahmer, W.M. et al (2018). A systematic approach to searching: an efficient and complete method to develop literature searches. Journal of the Medical Library Association, 106(4): 531-541.
DOI: https://doi.org/10.5195/jmla.2018.283
Take time to consider every possible synonym and don't stop until you are sure that a publication that does not contain any of your selected synonyms can not be relevant for your question. Avoid very common words with many meanings. Be aware that some terms have different meanings in different contexts, and that the same phenomenon can have different names in different subject areas. You have to include all your synonyms in the search, keeping in mind that most databases will do exactly what you tell them, and only what you tell them, and they do not know anything about synonyms or natural language. If you write "horse" in your search strategy, it will not find ponies, racehorses, thoroughbreds, foal, mare, stallion even though all these words also relate to horses. In some databases the search engine will not even find the plural "horses".
Specify exactly how your search terms should relate to each other. There are a few quite general syntax rules that are interpreted in the same way in most databases, see Search tips. When in doubt, different databases do different things. The easiest example is if you write two terms after each other with no Boolean operators involved, for example horse therapy, one database could interpret this as meaning horse AND therapy, another could search for horse OR therapy, a third might see it as a phrase, "horse therapy". The results signify totally different concepts. Always be as specific as possible to help the database do what you expect, and to ensure that your search is interpreted in the same way in all the databases you want to use for your search.
Look out for "user-friendly" interpretations in the database functionality. Some new tools, search engines and databases with a high ambition of being user-friendly, and also databases relying on natural language processes rather than assigned terms, will try their very best to help you with creative ideas about what you would like to know. This can give you lots of inspiration in your synonym-finding work! But it is a problem for the transparency and reproducibility that you are aiming for. Worst of all in this context is that some sources keep track of what you have selected in earlier searches (eg. Google does this very efficiently) and will do a relevance sorting which shows you mainly the things that you already know. This is more or less a disaster for the objectivity needed in systematic searching.
Consider the hierarchy of your free terms. A simple term might capture what you need, and there is no need to add extensions for that term unless you are looking for the extended term only, e.g. "horse" covers both "horse therapy", "horse riding" and any other expression containing the word horse.
Iterative method: in the articles retrieved in your first search attempt, you will probably find more relevant terms in the title, abstract, and keywords. Run a new test search with these terms included. If this captures even more relevant articles, try to identify more terms in them. If you already from the start know of some publications that are essential for the research questions, so called "golden articles", check if the search retrieves them. If not, and the articles are available in the database, you need to adapt your search further to capture the research question in your search. The "golden articles" can be a great help to find important search terms. Keep testing in one of the databases you will use, note the number of hits that each new term contributes, and repeat until no new relevant publications are retrieved. The film below is showing how to test your search terms in Web of Science .
Checking your search terms in a databaseConsult subject experts - it can be as easy as asking your colleagues.
Talk to the library. Many guidelines for systematic searching explicitly recommend discussing your search query with an experienced information specialist. Databases and their search rules are constantly changing, and there are tips and tricks that are special for each one of them. Information specialists may not be as familiar with the terminology in your research area as you and your colleagues are, but have more experience when it comes to databases and search strategy design.
You need to document every step of your search while you go, not only what words and terms you used, but also how they were combined. In some databases, search history for the session is automatically shown, and is a great tool for documenting the process. But remember, it will be erased when you log out, and we recommend you save the search before you log out. Not all sources have the search history or save functionality, and then you have to keep your own records.
Remember that at the end of the day, what you want to design is a comprehensive and reproducible search query, that can be run preferably on the same date in several databases, and documented as the basis for your screening and analysis. That is why you need to start and end in controlled databases, that do exactly what you tell them, and then add any results you have found from other kinds of sources.
-
-
The landscape for publishing in scientific publications has changed during the last decade, affecting where to search for scholarly publications, including many new types of sources. On this page you will learn a bit more about some search engines and how to search for and find grey literature. It is crucial for the comprehensiveness of your study that you use other sources as well as structured databases when searching for information to include in a systematic review.
Why include grey literature in your systematic review?
The answer in short is: to avoid publication bias. Not everything that is known will get published in scientific journals, and the reasons are not always about quality. The journals want to publish new findings on topics that create interest and citations, and researchers tend to be strategic with what they publish, to get the most out of their efforts. Negative or confirming results are not likely to get accepted for publication in a scientific journal, but can nevertheless be of vital importance and sometimes be found through other sources. Authorities and organisations also produce information that is published in other channels. If you need to include and consider all findable evidence, or only that which has been published in a scientific journal, depends on subject, aim and ambition for the review, but the decision whether to do it or not should be discussed and motivated.
Reports, preprints, memoranda, theses, state-of-the-art reports, market research, conference posters and proceedings, technical specifications and standards, bibliographies, datasets, ongoing clinical trials, government documents, patents, newsletters etc.
You will find grey literature in some of the mentioned resources and databases on the Where to search - structured databases page, e.g. Google scholar and databases like Web of Science Core Collection. But there are also a lot of repositories and other resources which have specific content and publications.
ClinicalTrials.gov - Ongoing clinical trials
Open Access Theses and Dissertations - Theses and dissertations
Networked Digital Library of Theses and Dissertations - links to a repositories world wide containg thesis, e.g. for France, India, Spain.
OSF Preprints - Searches from various preprint servers, like ArXiv, AfricArXiv, bioArxiv, RePEc and many more.
re3data Registry of Research Data Repositories - Datasets
Patentscope - Patents via WIPO, World Intellectual Property Organization
South African SDG hub - a platform to search for research on the Sustainable Development Goals (SDGs) from South African and selected non-South African universities.
Connecting Africa - Gives access to African research information produced worldwide.
ASKIA The Access to Scientific and Socio-economic Information in Africa (ASKIA) - an initiative from United Nations Economic Commissions for Africa.
AfricaBib - a collection of Africana social sciences titles. Containing 7 bibliographic databases and indexes periodical titles.
WorldWideScience.org - Science gateway and searches in many national and international scientific databases and portals.There is a vast number of ways to access information. As you learned on the Where to search - structured databases page, there are many licensed platforms and databases that require subscriptions via libraries or institutions etc. Other resources are freely available. Open access and open science have affected the publication landscape as well as access to scholarly information, and more options for searching different sources via open resources is now available.
Despite a more open access to information, there can be pitfalls in this world of databases and search engines. One example is Microsoft Academic. With close to 2,6 million indexed publications in its database, it retired in December 2021. This highlights an issue with information sources - they can be shut down, moved or changed. One consequence for the users is that the reproducibility of a search disappears when the platform or database is no longer in use. The impact of the retirement and the services of the actual resources might not be lost due to search engines like lens.org, Semantic Scholar and Scinapse and the shut down of Microsoft Academic gave OpenAlex a push forward (https://openalex.org/about). The open catalogue officially launches in September 2022. (https://explore.openalex.org/)
Google Scholar is a major resource for many researchers. From a systematic review point of view, though, it has its limitations since it is not transparent enough, and it not possible to use for reproducible searches (see the comparison between Pubmed, Web of Science Core Collection and Google Scholar on the Where to search - structured databases page). For a systematic search, you need to be able to run a structured and controlled search, that retrieves results in the same way, whenever you run the search. As the LSE impact blog writes “[...]Google Scholar are neither reproducible, nor transparent. Repeated searches often retrieve different results and users cannot specify detailed search queries, leaving it to the system to interpret what the user wants" . (A broken system – why literature searching needs a FAIR revolution | Impact of Social Sciences).
Considering all new resources sprung from the shifting publishing landscape, the context of the systematic reviews faces new challenges both regarding searching and accessing literature and information. They often can provide possibilities to find new, sometimes surprising, aspects on a query, but frequently they are not possible to search in the structured way of the databases described on the Where to search - structured databases page. This makes it even more important to be transparent and document where and how a search was made, in order to be as systematic and structured as possible.
Regardless of these difficulties, you need to use less structured databases, repositories and search engines to find information published in other forms than traditional, commercial journal articles. This includes grey literature and journal articles that are not indexed in the bibliographic databases as described on the Where to search - structured databases page.
Below are some tips of resources that are free, some with limitations in functionality if your institutions do not have a licence, including resources that retrive bibliographic records from different sources and materials. Some give you a possibility to search in a structured way using fields, Boolean operators etc., but not all. Apart from the ones listed, there are many more free resources that harvest full-text open access publications, dataset, grey literature, etc. and/or with possibilites to export bibliographic records from repositories from all over the world.
Some examples of academic search engines
Core Contains open access research papers in and datasets harvested from institutional and subject repositories from all over the world.
BASE Search engine for academic web resources, e.g. journal articles, preprints, digital collections, images/videos and research data. Searches the deep web as well.
Semantic Scholar AI-powered research tool for scientific literature. Uses machine learning to identify connections within papers.
Scinapse - Uses data from from Microsoft Research and others.
Google Scholar Uses unknown search algorithms.
Duckduckgo Search engine that does not track your search history, not a academic search engine. Useful to get out of a search bubble.You may also find grey literature on the websites of organisations and professional networks, which make the documentation of projects, datasets and their publications free. Even if it can take a lot of time, asking authors, experts or colleagues directly can also give you relevant material, including older or unpublished material. Using search engines like Google and Yahoo you have to be aware that these are commercial purpose and have theie algorithms and the search result may differ due to locatoins, or what device you searching from, like mobile or laptops. So you need to use more than one search engine. (Collaboration for Environmental Evidence (2020) Section 5 Conducting a Search. https://environmentalevidence.org/information-for-authors/5-conducting-a-search/ [2022-05-18)
A friendly reminder:
You need to adapt your search strategy to the different databases and sources. Don't forget to be transparent and document your search: Where and how did you search?
-
-
This page focuses on what happens when you have run your search query in multiple databases to retrieve the references that may or may not be included in your review (more on screening later). What do you do with all these references?
You need to collate the search results from your different sources in one place, preferably a reference management software. In this course we will focus on the desktop versions of EndNote and Zotero, to collect, organise and deduplicate your search result records. If you are already using another reference manager, there is no need to change.
We would like to mention that from a deduplication point of view, EndNote is superior. This functionality is very much needed within the systematic review context, given the often large amount of records generated from multiple sources.If you are new to reference management systems in general, or EndNote and Zotero specifically, please follow the links below for more information.
Read more about reference management software in general.
Read more about Getting started with EndNote.
Read more about Getting started with Zotero.As you learned in Where to search, the searches that are part of a systematic review can and should be run in multiple, various databases. These databases allow you to download and export your search results to a reference manager, albeit in different ways. In some cases it is fairly easy, e.g. Web of Science providing direct export to EndNote. This is because the commercial company behind the database also owns the reference management system. But for all databases the RIS-format (.ris) is a format used for download/export. The RIS-format is compatible with most reference management system, not only EndNote or Zotero. The export functionality may be named differently, e.g. Send to, Export, Download, Cite or Export to RefMan, but the common denominator is the RIS-format.
Just like you were able to export your search results from a database to a reference manager, you can export the records from the reference manager itself, e.g. to a specific screening tool or other reference managers.
As when you downloaded references to your reference management system, you export from the reference manager in RIS-format. Saving a file of your reference manager library in that format can also be a backup of your saved records if anything happens to the software or similar.
In the next part of in this course, Screening of results, you will learn about and import your references to a screening tool called Rayyan, to work with the screening of your results there. Below are links with information on how to export your records from your reference manager to a second system, such as Rayyan.
Export from EndNote (Step-by-step guide from Vrije Universiteit Amsterdam how to export a ris-file to a screening tool)
Export from Zotero (Using Zotero with Large Projects: A University of Michigan Taubman Health Sciences Library Guide)It is more or less mandatory to be at least 2 persons working on a systematic review. This demands sharing of and collaboration around the references retrieved in the searches. Below we share some information about how this can be done in EndNote and Zotero. There are also many collaboration tools for screening etc., which we come back to later.
These reference managament systems enable collaboration. In EndNote desktop you can create a shared library and add up to 100 participants to only read or read and write. You can also create groups and share these with your colleagues. In Zotero, use the Groups functionality to share collections of references. There is no limit to the number of participants per group.
How to share a library in EndNote
How to set up and use Zotero GroupsA more detailed comparison between can be found at the Washington University of St Louis or Uppsala University Library which includes EndNote Basic (formerly EndNote online) and Mendeley in the comparison.
EndNote 20 Zotero Cost Full license approx. 312 USD
For this course you will get access through SLUOpen source OS Windows, Mac Windows, Mac, Linux Installation Desktop (single), EndNote web Desktop (multiple), browser extensions, web Storage Local hd + EndNote web (unlimited if licensed) Local hd + online 300 Mb (option to buy more) Word processors Word, Open office, Libre office, Pages Word, LibreOffice, Google docs + 3rd party plugins (Links to an external site.) Mobile apps iOS (3rd party for iOS & Android) Metadata PDF:s DOI + CrossRef DOI, ISBN + CrossRef, Citation styles >7000 styles (proprietary) > 9500 (CSL (Links to an external site.), open source) Collaboration Share library with up to 100 persons, travelling library & EndNote web Share groups with references and files Before reaching the Screening of results page, you will find pages with more information about EndNote and Zotero.
In any systematic search process, it is of utmost importance to document the steps your take. There are various tools and guides to support this, e.g. ROSES and PRISMA flow diagrams.
ROSES
The ROSES flow diagrams are templates for recording the flow of articles through searching, screening, coding/meta-data extraction, data extraction, critical appraisal and synthesis for systematic reviews and systematic maps. They can be tailored by reviewers as necessary, but the templates constitute a minimum level of detail needed in any schematic of a review process. An online tool is also available.
PRISMA
PRISMA, short for Preferred Reporting Items of Systematic reviews and Meta-Analyses, aims to help authors improve the reporting of systematic reviews and meta-analyses. The PRISMA flow diagrams depicts the flow of information through the different phases of a systematic review. It maps out the number of records identified, included and excluded, and the reasons for exclusions.
-
Since you have retrieved records from multiple sources, you will have several duplicate records in your collection. Before you start screening your results, you need to remove duplicates in order to avoid reviewing the same article more than once. This process is often called deduplication.
On this page we go through the steps for deduplication in EndNote Desktop and Zotero.
One at the time
Go to Library/Find duplicates
Review and keep the records you prefer. Then the next pair of records occur automatically.All in one go
It is often more efficient to remove all identified duplicates in one go.
Go to Library/Find duplicatesClick Cancel in the upper right-hand corner. All duplicates are now highlighted.
Right-click on one of the highlighted references and select “Move Reference to Trash”
NB: The default setting for bulk deduplication is to keep the oldest records. So if you prefer the bibliographic data from one particular database, make sure to import those records first. You can see what date a record was added to the EndNote in the “Added to library” column.
Tips
Some duplicates remain unindentified in this automatic process. Reasons may be that journal names are written differently in different databases, or that author names are entered in different ways. You may find these by sorting your references in EndNote on Title.
There is an simplified deduplication functionality available in EndNote online.
For more information, see EndNote 20 : Steps to remove duplicates from your EndNote library.
Advanced deduplication in EndNote
If there is need for a more accurate deduplication, depending how many records you have retrieved and you are in need to automatically remove the most of them.
You can use this step-by-step method originated from Wichor Bramer, information specialist at Erasmus MC, but simplified here from the Karolinska institutet.On this page from the University of Michigan Taubman Health Sciences Library, you find instructions on how to deduplicate in Zotero (page 3).
Zotero also provides a page specifically about finding duplicates.
In EndNote you can deduplicate a bulk of references in one go, whereas in Zotero, you need to go through the duplicates set by set. One set of duplicates could contain more than two duplicates, though, depending on where and how many databases you searched. Also, some sets of duplicates may have no conflicting fields, allowing you to merge the duplicates really quickly.
For transparancy reasons, you need to document the number of duplicates that you remove or merge into one record. You need to document both how many duplicates were detected, and how many records where left after the deduplication. ROSES and PRISMA flow diagrams are both useful for this documentation.
Before reaching the Screening of results page, you will find three pages with more information about EndNote and Zotero.
-
Download EndNote and install from SLU student share ( Login is required. You need to activate your student login before). If you dont have the rights to download and install contact SLU's IT-support to get help with to EndNote installed. More information can be found on the library web site.
Official videos from Clarivate analytics (YouTube channel):
-
Zotero can be installed on as many computers as you want and your library will be synchronized via your
account at zotero.org. With browser connectors (Firefox, Chrome and Safari) and bookmarklet (any browser, tablets and smartphones) you can save
references directly from web pages. Plugins are available for Word, Open Office and Libre office.
Uppsala University Library has a quick guide to Zotero and is a good introduction to get started
A introductory video from Hochschule München (9:58):
The SLU library has some information on this page.
On the Zotero.org homepage you can find documentation and a Forum where you can get help from the community.
- Free to use and open source
- Easy to get started
- Excellent functionality for web import from various sources (reference databases, library catalogues, Google scholar and web pages)
- Plugins for Google docs, Open office and Libre office
- Easy to share groups of references for collaboration, without limitations in number of groups or members
- Good options for exporting references/library in different formats
- Tag articles with keywords for easy filtering of references
- Better than EndNote in retrieving metadata for PDF; uses DOI (via CrossRef) and Google scholar.
- Not as polished as EndNote
- Official support not as elaborated (although a large and helpful community)
- With only 300 Mb online storage, syncing your library with attached PDF:s mean that you can quickly run out of storage space (you can sync without including PDF:s)
- No in-built PDF reader
- Compared to EndNote, less options for building indexes with term list for authors, journals and keywords, and creating groups
Read more about how to remove duplicates
Image reference
This is a job for Zotero by jazzmodeus ( CC BY-NC 2.0 )
-
Now it is time to apply the inclusion and exclusion criteria that you prepared when planning your review (see the last paragraph of Planning your review)
In all probability, you will have a number of hits in your deduplicated library that has nothing at all to do with your research question - because one or more of the search terms have other meanings, or because the database is too optimistically indexed. These can be removed as being not in scope.
All captured publication that are in scope, on the other hand, must be assessed according to the selection criteria. It can be tempting to also remove publications that are in scope, but which you find not very interesting, or badly designed or just not dependable. But that kind of evaluation of the material must wait until the final analysis of all the publications that meet the criteria, since there might turn up interesting ideas on the subject that have not been well researched or data that are not yet fully published. The screening should strictly follow the inclusion and exclusion rules, which in turn must be objective and formulated as unambiguously as possible. It might be necessary to reformulate a criterium that turns out to be too easily misinterpreted or differently interpreted. But if you do that, you also need to assess all publications according to the changed criterium, so it is time well spent to make sure from the start that the criteria are well understood by all members of your review team.
More than one person has to be involved in the screening process in each step, to avoid that the selection reflects only one person’s comprehension of the subject. Articles where the screeners/reviewers disagree on inclusion need to be singled out and discussed separately, and decided on according to the selection criteria.
How do screeners know that they agree or disagree and how do you get an overview of the screened results? As we will see at the end of this module there are several useful tools to help keep track of this.
Documentation of every step in the searching, screening and analysis is crucial for the transparency, reproducibility and usefulness of a systematic literature research. Every document found by your search and every document excluded in the screening process must be documented and accounted for! It is convenient to fill in a flow scheme like we will see below, to keep track of where they all went.
Many organisations have developed sets of reporting standards and flow charts to facilitate and quality check the documentation steps. Here are some examples of reporting standards:
PRISMA flow diagram (PRISMA)
ROSES - RepOrting standards for Systematic Evidence Syntheses in environmental research (CEE)
MECIR Standards (Cochrane)
MECCIR Reporting Standards
download (Campbell)In this section we will use ROSES (RepOrting standards for Systematic Evidence Syntheses) as our guide. This reporting flow diagram is useful even for smaller projects and suits our subject areas well. You are free to use whichever reporting standard you want, using ROSES is just an example.
-
Remove duplicates
See previous section. Record the number of records removed and the number remaining.
-
Title screening
After you have removed all duplicates, it's time to look at the titles of the remaining articles to see what can be excluded at first glance.
As you can see in the flow diagram, this can be done at the same time as the abstract screening and it's often convenient to do so.
It saves a lot of time and effort doing the screening in a program like Rayyan. More on this later.
In this step you will:
- Remove references that are clearly not relevant;
- Does not fill inclusion criteria regarding eg. document type, language, etc.
- Obvious mismatch with the search strategy
- Remember: terms can be ambiguous and mean something quite different in other contexts
- Remove ONLY those that are CLEARLY not relevant; a title can be very vague. Keep those that seem uncertain.
- Record the number of references removed and the number kept.
- Remove references that are clearly not relevant;
-
Abstract screening
- Read abstracts and discard those that are clearly not relevant
- Fulfills the predefined exclusion criteria
- Does not fulfill the predefined inclusion criteria
- Obvious mismatch with the search strategy
- Terms can be ambiguous and mean something quite different in other contexts
- Record the number of references removed and the number kept.
Again: Remove ONLY those that are CLEARLY not relevant; even an abstract can be quite vague, or not mention important information. Keep those that you are uncertain about.
- Read abstracts and discard those that are clearly not relevant
-
Fulltext screening
At this stage your crop of articles has hopefully shrunk to a somewhat manageable number. It's not until this step you will look at the actual full text.
NB that you need to separate articles that could be retrieved as full text from the ones that were potentially interesting, but where you could not access the article itself. There will always be texts that you just can’t find and which have to be excluded. This needs to be documented and noted as part of the reporting.In the flow chart above they then return to the PICO model to explain excluded articles, but this documentation needs to be made no matter which model has been used for your criteria.
In this step:
- Read full text of all remaining references
- Final assessment whether the inclusion or exclusion criteria are fullfilled, or if there is any other reason that the publication should not be included.
- Record the number of articles removed and the number kept, and document the reasons why all excluded articles were excluded! This is important for transparency, reproducibility, credibility
- The last box in this picture mentions ”pre-screened articles from other sources”. This can be articles found in reference lists, ad hoc-discoveries and so called grey literature, in short: relevant texts you have found outside your database searches.
As with reference management and documentation tools, there is a variety of screening tools to consider. For this course we will be using Rayyan, since it is free to use, and well-known.
Rayyan - an example
For our assignment you are going to use Rayyan as a screening tool. Here is a screen-shot of Rayyan when a screening is ongoing. The example shows a screening with 3 reviewers.
Looking at the first record, the publication with the title “Cracked Equivalent Beam …” in the example below, we see reviewers (arrow 1) Britt-Marie and Jonas in red boxes. The red boxes indicate that both reviewers have decided to exclude the publication by using the exclusion or inclusion-icons pictured as a green thumbs-up and a red thumbs-down.
They have also added a blue label called Title (arrow 2) and used the pre-designed reason tag "Wrong population" (arrow 3).
In the selected record in the screen-shoot example, the record's abstract is highlighted in blue. In the abstract, words are marked in red and green color (arrow 4 and 5). Those are the inclusion and exclusion criterias that you have added in Rayyan. As arrow 4 points out, the list of keywords for inclusion is listed in the column on the left, marked as green, and showing number of articles which includes the keyword. In a corresponding way, arrow 5 points out the keywords for exclusion, marked as red in the abstract, showing the related number of articles and keywords in the left-hand column.
The example shows a Blind off-mode (arrow 6). In order to keep reviewers unaware of each other's decisions, you should instead start with a Blind-on mode. You can easily change mode by clicking All Reviews in the upper right-hand corner.
After creating a review, you need to go to All Reviews and choose your review and then invite collaborators.
- Please go to Rayyan and create an account before you move on to the assignment on the next page.
- Use the step-by-step guide to get started with Rayyan
-
-
The narrative synthesis is a textual approach to synthesizing the findings. This is a necessary element in your report, regardless of whether you will develop it further into more advanced qualitative or quantitative analyses (eg. a meta analysis), or stick to the narrative approach for your whole report. Some questions or results are not suitable for a quantitative analysis, and a narrative synthesis can be seen as having advantages when dealing with broader questions and disparate outcomes. Often a narrative synthesis is the only option when faced with a pool of disparate studies of relatively high susceptibility to bias, but such syntheses also accompany quantitative syntheses in order to provide context and background and help characterise the full evidence base.
The narrative should include a presentation of and discussion about what the body of the evidence says. It must also include a discussion on the robustness of the findings, and of the methods used, to arrive at the presented conclusion.
The body of evidence can be explored and presented with the help of visualisations, for example tables or figures for main categories
The body of evidence can be explored and presented with the help of visualisations,
for example tables or figures for main categories- Intervention/exposure
- Outcomes
- Species
- Study design
- Susceptibility to bias
- Length of investigation
- Seasons
It might be meaningful to develop group studies for some parameters.
The tables and group studies prepared may make it possible to discuss for example:- Themes and trends
- Consistency of effect
- Study clusters
- Patterns of factors
When discussing the whole body of evidence, it is necessary to take into account how these factors, plus the robustness of findings and quality of the design of the separate studies might impact on the results of each study. A simple vote counting or mean value for the effect of a specified intervention might be very misleading, as it might ignore studies with non-significant findings, and give the same weight to poor methodological and biased studies as to more robust results. The figures below are examples of sets of results where only listing the number of significant positive, significant negative or no significant effect would have failed to represent the result in a meaningful way (as shown by the forest plots* below each table).
From: Haddaway: Online course in Systematic mapping and review https://systematicreviewmethods.github.io/index.html
*how to construct a forest plot is further described on the Meta-analysis page
-
A meta-analysis is a statistical analysis that combines the results of multiple scientific studies.
Meta-analyses can be performed when there are multiple scientific studies addressing the same question in a similar way, with each individual study reporting measurements that are expected to have some degree of error. Far from all systematic reviews are suitable for a meta-analysis, and there are also many meta-analyses that are (regrettably) not based on systematic searches.
In order to do a meta-analysis, you need results that are comparable
All the studies that you would like to include must have:
- estimated the same effect
- been conducted in similar ways or at least not differently in ways that can be expected to influence the outcome
- have measured the outcomes in the same way
- and all these data must be available
Advantages with meta analysis
- improvement in precision
- ability to answer questions not posed by individual studies
- opportunity to settle controversies arising from conflicting claims
Sources for misinterpretation
When conducting a meta-analysis, there is a risk of misinterpreting the data. This risk can emanate from many sources, such as
- variations in study designs
- within-study biases
- variation across studies
- reporting biases
Meta-analyses are usually illustrated using a forest plot (see example below, and two examples on the Synthesis methods page, as comparison to the vote counting method). A forest plot displays effect estimates and confidence intervals for both individual studies and meta-analyses. Each study is represented by a block at the point estimate of intervention effect with a horizontal line extending either side of the block. The area of the block indicates the weight assigned to that study in the meta-analysis while the horizontal line depicts the confidence interval (usually with a 95% level of confidence). The area of the block and the confidence interval convey similar information, but both make different contributions to the graphic. The confidence interval depicts the range of intervention effects compatible with the study’s result. The calculation of the summary result is presented as a diamond at the bottom.
By James Grellier - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=10253357
A common source for misleading meta-analyses is of course missing data, for various reasons. A list of the most obvious risks is given in the figure below.
Funnel plots have long been used to assess the possibility that results are missing from a meta-analysis in a manner that is related to their magnitude or P value (Liu et al 2011). However, they require careful interpretation (Sterne et al 2011).
A funnel plot is just a simple scatter plot of intervention effect estimates from individual studies against a measure of each study’s size or precision. In common with forest plots, it is most common to plot the effect estimates on the horizontal scale, and thus the measure of study size on the vertical axis.
Symmetrical funnel plot, indicating lack of reporting bias
Assymmetrical funnel plot, indicating presence of reporting bias
Liu, J. The role of the funnel plot in detecting publication and related biases in meta-analysis. Evid Based Dent 12, 121–122 (2011)
Meta-analyses can be very strong evidence, and there are many types of meta-analyses, and calculations. Please note that this page can only serve as a very brief introduction. If you plan to do a meta-analysis, we very strongly recommend that you first study some of the published literature on the subject. A thorough review of the meta-analysis methods used by Cochrane Reviews can be found here:
Hansen, C., Steinmetz, H. & Block, J. (2022). How to conduct a meta-analysis in eight steps: a practical guide. Management Review Quarterly 72, 1–19.
DOI: https://doi.org/10.1007/s11301-021-00247-4Forero DA, Lopez-Leon S, González-Giraldo Y, Bagos PG. (2019). Ten simple rules for carrying out and writing meta-analyses. PLoS Computational Biology. 2019;15(5):e1006922.
DOI: https://doi.org/10.1371/journal.pcbi.1006922Arya S, Schwartz TA, Ghaferi AA. (2020). Practical Guide to Meta-analysis. JAMA Surgery. 155(5):430–431.
DOI: 10.1001/jamasurg.2019.4523References to literature mentioned on this Canvas page:
Liu, J. The role of the funnel plot in detecting publication and related biases in meta-analysis. (2011). Evidence-Based Dentistry 12, 121–122.
DOI: https://doi.org/10.1038/sj.ebd.6400831Sterne JAC, Sutton AJ, Ioannidis JPA, Terrin N, Jones DR, Lau J, Carpenter J, Rücker G, Harbord RM, Schmid CH, Tetzlaff J, Deeks JJ, Peters J, Macaskill P, Schwarzer G, Duval S, Altman DG, Moher D, Higgins JPT. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. (2011) BMJ 2011; 343: d4002.
DOI: https://doi.org/10.1136/bmj.d4002
-
There are several research networks or collaboration centres that are specialized on methodology for systematic reviewing, and that perform large systematic investigations together with researchers from all over the world. The oldest, and most famous, is the Cochrane collaboration, working in the areas of medicine, pharmacology and public health. 3ie and CEE are examples of organizations working broadly in natural and environmental science.
On this page we have collated a list of well-known centres, with links to important parts of their information. They all provide guidance and detailed information on how to work systematically with information, following generally the same basic theory and methods, but adapted to different areas of research.
To follow the strict and often quite demanding guidelines is a prerequisite if you want to publish your systematic review or systematic map through these organizations. However, the guidelines also serve very well as an inspiration for conducting less ambitious reviews by picking methods to make your review more systematic. They are also good introductions to systematic reviews as a concept.
The idea with this page is to provide you with links for further study. We suggest that you take a closer look on one or two of the organizations that are of interest for your own current research.
Organization
Logo
CEE - Collaboration for Environmental Sciences
Environmental management and ecology
- Website: environmental evidence.org
- Guidelines and Standards for Evidence Synthesis in Environmental Management (Version 5.0, 2018)
Campbell collaboration
Crime and justice, education, international development and social welfare.
- Website: campbellcollaboration.org
- Campbell systematic reviews: policies and guidelines (Version 1.8, 2021)
Cochrane collaboration
Healthcare and medicine
- Website: cochrane.org
- Online training
3ie - International Initiative for Impact Evaluation
International development, social and economic development in low- and middleincome countries.
Africa Centre for Evidence, University of Johannesburg
ACE’s mission is to contribute to the reduction of poverty and inequality in Africa and South Africa through the use of evidence
- Website: https://africacentreforevidence.org/
- Evidence synthesis published at ACE
Centre for reviews and dissemination, University of York
Healthcare
Center for Evidence-Based Management
Management and leadership
- Website: https://www.cebma.org/
- CEBMa guideline for rapid evidence assessments in management and organizations (Version 1.0, 2017)
EPPI-Centre
Specialist centre for (i) developing methods for systematic reviewing and synthesis of research evidence; and (ii) developing methods for the study of the use research.
- Website: https://eppi.ioe.ac.uk/cms/
Joanna Briggs Institute (JBI)
Systematic Reviews for Animals & Food
Other guides- The production of quick scoping reviews and rapid evidence assessments [PDF] (Department of Environment food & rural affairs, UK government)
-
A research question for an evidence-finding systematic review or map is often a real world question, and thus related to more than one research discipline. This is both deeply satisfying and a challenging situation when performing systematic information research.
"What are the effects of agricultural interventions that aim to improve the nutritional status of children?"
From: Masset E, Haddad L, Cornelius A and Isaza-Castro J (2011), A systematic review of agricultural interventions that aim to improve nutritional status of children. London: EPPI-Centre, Social Science Research Unit, Institute of Education, University of London
This question could relate to:
- impacts on total household income
- agricultural methods
- choice of crops
- content of micronutrients
- nutritional status
- organization of work load
- diet composition
- technological innovations
- subventions
- etc. etc.
Each one of the disciplines in the image above have their own terminology. Are you familiar with the terminology in all disciplines that are involved in research that can have importance for your review question? The same concept can be discussed in diverse terms between disciplines or research groups. Even worse, the same term can have different meanings between disciplines or research groups. This means that doing crossdisciplinary work makes it necessary to work even more with all of the aspects that we discussed on the search strategy page. If you don't belong to a multidisciplinary research group but aim to review a question that relates to many disciplines, it might be a good idea to consult with representatives from all related disciplines. Be prepared to compare words, terms and concepts in an open-minded way, and consider how different meanings can be treated in the search strategy.
Beside the big databases with broad scopes there are also many that have a more narrow focus. The general databases are of course very good sources for crossdisciplinary work. But you also need to know if there are focused databases for any of the aspects of your review question, that you do not normally cover. Regarding sources for grey literature, they are often even more discipline-specific and need to be investigated with care.
Publication culture and traditions differ not only between disciplines but also between research groups within the same subject. Very widely, one can say that in medicine, it is customary to publish many journal articles, and to cite each other frequently. In biology and environmental science, publishing in journal articles is also common, though not as frequently as in medicine. In many social sciences, it is common to publish and discuss through conferences and conference proceedings, while researchers in the humanities tend to publish books more often than is common in other disciplines. Researchers in physics are well known for publishing many preprints. This is of course a simplified and exaggerated picture, but still something to consider in order to avoid bias resulting from not including the appropriate publication types, or from not taking differing publication frequencies and traditions into account.
-
Stakeholder analysis is an important step in the planning and performing of a systematic review, both in order to make it as comprehensive and correct as possible, and to help the results of it to be seen, and hopefully used. For a big evidence-finding project, stakeholder management can be a rather complicated commitment. But even if you are working with a smaller review or mapping project, it is always worth the time to consider possible contributors or recipients/users of the results from the work, and when to involve them.
Stakeholders can be involved in the review project (collegues, practicians, expert networks, translators) or in the publication and dissemination of the results (journals or other publishing possibilities, journalists, the readers, knowledge brokers, innovators, policy makers). Depending on the setting for the review, there might also be authorities or companies involved, sometimes with contradictory agendas.
The image below shows some possible contributions to and from stakeholders, without mentioning from or to whom. Some of these contributions have been discussed earlier in this course: for the search itself, it is very valuable with experts in the area suggesting sources, terms, and important article to consider; for the methodology, you might take help from organizations working with systematic reviews in your subject, or from other review articles; sometimes you need to involve other experts in the screening and interpretation. We have not talked about funders, but they are of course very important stakeholders. And we have hardly mentioned the readers, the policy makers, and the final users.
Haddaway et al. (2017) A framework for stakeholder engagement during systematic reviews and maps in environmental management. Environmental Evidence volume 6, Article number: 11.
https://doi.org/10.1186/s13750-017-0089-8Now and then stakeholders have opposite ideas, hopes and wishes, or maybe represent competing interests. This can be hard to handle and is one of the reasons for it being a good idea to try to think proactively. What are the possibilities that we have stakeholders on board who has ideas from the start about what the results of the review should be? Luckily, this is not so common, but if you find yourself in a situation where a stakeholder's wish could impact the objectivity of the review, what do you do? Probably the most common stakeholder conflict is that someone who has funded the review needs the results in five weeks, while your scoping review indicates that you need at least five months for the screening, interpretation and analysis. What can be done in that case? And the situation where two of the experts involved bitterly oppose each other in the interpretation of the results, this is also a stakeholder conflict that needs a solution, as objective and satisfying as possible.
It is important to plan for how to get your review published. Scientific journals, publishers and reviewers are powerful stakeholders, and journalists, social media, policy makers and innovators can be important intermediators. Think in advance about possible journals or organizations where you would like to publish for best visibility or best reach of the groups you want to impact - what are the submissions rules in these journals, and how are systematic reviews or maps presented in them? Would it be useful to write a summary in another language, or a popular report for some stakeholders? What are your possibilities to place a comment or a note about the work in social media? Would it be a good idea to alert scientific journalists to write newspaper commentaries about it, by sending out a press release?
How to handle stakeholder involvement is an area of research in itself, and too broad a subject to involve in this course. It is, however, a part of a systematic review project that is often not reported about at all, or reported in a fragmented way. In some cases, it would have been good for transparency and reproducibility to have access to information about how stakeholders have contributed, at what level and in what roles. If you are interested to read more about this, please find a link below to a scoping review about stakeholder involvement in systematic reviews.
Pollock et al. (2018). Stakeholder involvement in systematic reviews: a scoping review. Systematic Reviews volume 7, Article number: 208
DOI: https://doi.org/10.1186/s13643-018-0852-0
Create an account to get more
-
Track your progress
Review and track your learning through your profile.
-
Micro-Credentials
Completing the course, you will earn a Recognition Badge or Certificate.
-
Access all course activities
Take course quizzes and access all learning.
-
Review the course
When you have finished a course leave a review and tell others what you think.
For further information, take a look at our frequently asked questions which may give you the support you need.
Have a question?
Advance Your Future at the University of Nairobi
Built on a legacy of world-class research and academic excellence, the University of Nairobi delivers world-tier education wherever you are. Whether advancing your career, mastering a specialisation, or pursuing frontier research, your next chapter begins here. Join Africa’s premier community of leaders and innovators.
Apply now: application.uonbi.ac.ke.