Consultations may be conducted remotely (online for example), in face-to-face meetings or workshops, or a combination of these. Consultation may take the form of peer review with patient and public expert reviewers. It can also include research with patients and carers (using methods such as surveys, focus groups and interviews). Research participants are typically not expected to represent the views of other people, but to characterise their own views and experiences. Whichever approach is taken, consultation adds significantly to the time and resource requirements of guideline development and should be factored in at the outset. In most consultation processes, such as feedback on draft scoping documents and draft guidelines, patient or public consultation can occur simultaneously with professional consultation. As Cluzeau et al. (2012) concluded, for stakeholder engagement to be successful, it needs to be inclusive, equitable and adequately resourced. The box contains a summary of the main consultation approaches.

Public comment

Background

In public comment, guideline developers post guideline materials in a public forum for feedback. This typically involves posting materials online but can include an open forum for discussion. Materials shared for public comment include guideline scopes and research protocols (to obtain feedback before starting the systematic review) or draft guideline documents (to obtain feedback before final publication). Public comment can include feedback from individual professional and patient experts, but is generally considered distinct from external peer review, which is solicited.

In the US, the Institute of Medicine (now, the National Academy of Medicine) Committee on Standards for Developing Trustworthy Clinical Practice Guidelines (CPGs) includes public comment in its external review standard 7.4:

‘A draft of the CPG at the external review stage or immediately following it (i.e., prior to the final draft) should be made available to the general public for comment. Reasonable notice of impending publication should be provided to interested public stakeholders’. (Chapter 5; 2011).

Despite the fact that public comment is recommended by the Institute of Medicine, a review of guideline developer methodology manuals by Armstrong et al. (2017) found that only 6 of 101 US-based guideline developers posted protocols for guideline development at least some of the time. Only 1 organisation, the United States Preventive Services Task Force, posted a draft research plan using a public-friendly template (for example, using plain language, avoiding excessive background or technical information). Only a quarter of US guideline developers posted draft guidelines for public comment. One developer used a public hearing for public comment, while the remainder used online mechanisms. Most developers using online feedback posted materials for comment for 1 month (range 14 days to 60 days). There was no evidence that any guideline developer posted a patient-friendly version of the draft guideline for comment.

By way of comparison, Ollenschläger et al.’s (2018) assessment of all guidelines in the German national guideline registry in 2018/19 found that 58% had involved patients on the guideline group. However, only 14% (39/270) had provided plain language versions of the draft guideline for consultation.

Practical approaches for using public comment

As with other consultation approaches, guideline developers need to be intentional about using public comment approaches. Desired feedback will vary at different stages, such as between draft scope, protocols and draft guidelines, and may differ between developer types. For example, guideline developers representing national health systems or governing bodies may desire different feedback than professional organisations. Guideline developers must also consider available resources when considering public comment. Potential costs associated with public comment include developing public-friendly materials for posting, hosting a public forum or website, publicising the comment period, and allowing time to respond to public comments (including decision making, documenting comments and responses).

After choosing to use public comment as a consultation strategy, developers decide the stage(s) at which to use public comment (for example, scoping the topic, research protocol, draft guideline). To make optimal use of public comment, developers need to create materials that are likely to result in meaningful engagement and avoid tokenistic public comment. Many guidelines are aimed at professional audiences and can be hundreds of pages long. Difficulty in understanding medical terminology is one of the most common barriers to patient and public involvement in guidelines. (Jarrett et al. 2004; Légaré et al. 2011; Qaseem et al. 2012; van de Bovenkamp et al. 2009; van Wersch et al. 2001.) Thus, developers desiring meaningful feedback need to prepare patient- and public-friendly guideline documents for draft review. For developers working with patients to create patient guideline versions, this could also include preparing and posting a draft for public comment (see the chapter on how to develop information from guidelines for patients and the public for further information).

In conjunction with creating the materials for posting, developers must determine the feedback desired from respondents. For example, the U.S. Preventive Services Task Force (2017) posts 3 types of documents for public comment, as shown in table 3.

Table 3 Public comment feedback requested by U.S. Preventive Services Task Force

Type of document	Response requested
Draft research plans	Respondents to indicate level of agreement and provide free-form comments on the: analytic framework, proposed questions, proposed research approach (presented in tabular form)
Draft evidence review	Asks if the respondent: thinks the report includes all of the relevant studies, agrees with the interpretation of the evidence, has suggestions for making the findings clearer
Recommendation statements	Asks the respondent: how to make the statements clearer, if expected information is missing, whether the conclusions reflect the evidence, what associated tools would be useful, other experiences and comments

Many online public comment approaches are similar to those of the U.S. Preventive Services Task Force in that they use a web-based survey to ask the respondent to indicate his or her level of agreement (with questions, evidence synthesis, recommendations) and then allow open comments.

For meaningful feedback, developers must create a plan for notifying key public members regarding upcoming public comment periods. Potential strategies include notifying relevant professional and patient organisations regarding the public comment period and asking them to invite their members to participate. Government organisations desiring feedback may also provide advance notice to broader populations. For example, the external review standard 7.4 of the US Institute of Medicine Committee on Standards for Developing Trustworthy Clinical Practice Guidelines recommends that developers provide reasonable notice of impending publication prior to posting (Chapter 5 2011). There are no best practices for posting length, but 1 month is a typical time frame (Armstrong et al. 2017). As with other consultation strategies, guideline developers should be prepared to respond to feedback provided through public comment (see responding to consultation comments).

Consulting patient and public stakeholder organisations

The UK’s NICE uses an open consultation process, with draft consultation documents posted on its website at key stages in the guideline development process. This is similar to the public comment approach; however, to manage the volume of comments in a transparent way, NICE encourages individuals to respond through a relevant stakeholder organisation. These organisations receive a response to each of their comments, and both the comments and the developers’ responses are published on the NICE website. Responses from individuals are acknowledged and considered, but do not receive a response unless they are designated peer reviewers.

In the NICE model, all registered stakeholder organisations are invited to contribute at key stages of the guideline development process. This includes:

• Setting the scope of the guideline and the key questions.
• Circulating NICE website advertisements to their members and networks for recruitment to the guideline development group (health and social care professional and patient or public members).
• Responding to calls for evidence if the guideline developers believe that their literature search has not found all the relevant information. Such evidence could include patient surveys and other real-world evidence on the impact of the condition on people’s lives, the views of patients and carers about their treatment or care, or the difference a particular type of care or treatment might make.
• Commenting on the draft guideline.

To support stakeholder engagement, NICE maintains an extensive database of contacts for organisations representing patient and public interests and invites them to register their interest for new guideline topics. Staff in NICE’s Public Involvement Programme help identify relevant organisations and offer information and advice to support their involvement.

Identifying and reaching patient and public groups

Not all guideline developers have the structure and resources needed for the NICE model. The following suggestions may be helpful in identifying relevant patient and public groups (organisations and individuals) and inviting them to take part in consultations.

Networks of patient advocacy groups and charities may provide a useful avenue for reaching relevant patient or public stakeholders. For example, SIGN’s Patient and Public Involvement Network members are notified of involvement opportunities when a new guideline is being developed.

Other sources for identifying relevant patient or public stakeholders include health professionals and their organisations, patient organisations that are already known to guideline developers, the internet and social media. In addition, if the guideline development group has been convened, it may be fruitful to work with patient and public members to identify key organisations and individuals with the desired perspectives and experiences.

Consider contacting national and international patient or public groups, because they can be a useful source of contacts and advice, as well as an avenue for collaboration. Examples include:

• National groups, such as Consumers United for Evidence-based Practice (CUE) in the US and Foro Español de Pacientes in Spain
• International groups, such as G-I-N Public (Guideline International Network’s Public Working Group), CCNet (the Cochrane Consumer Network), and the Health Technology Assessment international’s (HTAi) subgroup on Patient and Citizen Involvement in Health Technology Assessment.

Social media can be an excellent way to promote a consultation, by posting details about it and tagging in patient and public advocacy groups from the guideline’s topic area. If the consultation is open to the public, this can also be an effective way of reaching a wider audience of people beyond the usual patients the guideline developers may work with. Increase the reach by using hashtags that are commonly used by patients or public in the topic field and post details of the consultation with relevant advocacy groups on social media or online patient forums. NICE has found social media helpful in building relationships with key patient and public stakeholders and supporting their involvement with NICE guidelines. NICE also uses social media to promote published guidelines, working with key stakeholders and communities to ensure the main messages reach the public. 

Examples of consultation at key stages

Setting the scope of the guideline

It is important to include patient and public perspectives from the beginning of the guideline development process. With this end in mind, SIGN and NICE consult patient and public groups on the scope of a new guideline before the first meeting of the guideline development group. GuíaSalud in Spain also include consultation with patients at this preparatory stage of guideline development. For example, they used focus groups and interviews with patients to inform the scope and key questions for 2 guidelines on anxiety and insomnia (Díaz del Campo et al. 2011).

Four months before the first meeting of a new guideline development group, SIGN invites patient and carer organisations to highlight the issues they think the guideline should address. A form is supplied to enable them to structure their feedback in a useful way and to indicate the source of their suggestions (such as telephone helpline data, surveys). SIGN then summarises the information received and presents it to the guideline group at its first meeting. When published evidence is scarce and there is inadequate feedback from patient organisations, SIGN may seek patient and public views through direct contact with users of the service. This has been achieved using focus groups with patients in different regions of Scotland, attendance of SIGN staff at patient support group meetings, and SIGN-organised meetings for patients and members of the public. The information obtained from these approaches is reported to guideline groups to influence the development of key questions underpinning the guideline. (SIGN 100 2019; SIGN 50 2019.)

NICE involves patient organisations and other stakeholders in the scoping process in 2 ways: participation in a meeting and online consultation. All organisations that have registered an interest in a new guideline project are invited to attend the scoping meeting. This gives patient organisations and other stakeholders an opportunity to become familiar with the guideline development process and to take part in detailed discussions about the scope. It sets out what the guideline will and will not cover, defines the aspects of care that will be addressed, and outlines the key research questions. A draft scope is then produced, and stakeholders are invited to comment on it during a 4-week online consultation. This online process is designed to ensure openness and transparency, because all written comments receive a formal response from guideline developers, and both comments and responses are published on the NICE website. NICE encourages patient organisations to comment on the draft scope and provides prompting questions in its guide for stakeholders (NICE 2018). The purpose of the prompts is to seek their views on key issues (such as whether the identified outcome measures are in line with what matters to people with the condition or people using services), and to ask what should be included or excluded.

Some developers have used surveys to inform the research plan or protocol, as part of a strategy to incorporate evidence on patients’ values and preferences in guideline development. For example, the German National Disease Management Guidelines Programme found a benefit in surveying patients with anal cancer to obtain their feedback on the relative importance of a range of health outcomes (Werner et al. 2020. In the survey, they asked patients (n=37) and members of the guideline group (n=25) to rate the relative importance of outcomes in different clinical situations using the GRADE scale. For example, they found that agreement between the expert and patient ratings was fair for stage I-II anal cancer, but low for stage III anal cancer. In another example, whereas patients rated some adverse effects (such as early morbidity, proctitis or urge, radiodermatitis) as critical, experts rated these as important but not critical. The survey results informed the development of the guideline and helped with the trade-off between desired and undesired effects of interventions when making recommendations.

The draft guideline

Consulting patients and the public on draft recommendations helps ensure the range of their values and preferences has been integrated into the recommendations. As noted by Kelson et al. (2012), such feedback can include desired outcomes, the ways in which people weigh up risks and benefits, preferred treatment and management options, and whether the draft recommendations have real-world applicability.

Patient or public stakeholders can make an important contribution at this stage. For example, Chambers and Cowl (2018) analysed documentary evidence of comments from consumer organisations on the draft recommendations from 7 NICE maternity guidelines. Their aim was to assess the levels of engagement, along with the impact of that engagement. For each of the 7 guidelines, comments from consumer organisations resulted in 5 or more changes to the wording or meaning of the recommendations. For a more detailed look at the impact of consumer organisation comments see the Slideshare presentation on NICE maternity services evaluation.

SIGN combines open consultation on the draft guideline with a later period of peer review. During the open consultation, SIGN may hold a national open meeting with professionals, patients and the public to discuss the draft recommendations. Draft guidelines are presented on the SIGN website and through social media. Anyone can respond to the online consultation and particular efforts are made to ensure all equality groups with a potential interest in the topic are made aware of the opportunity to comment.

NICE follows a similar online consultation process, inviting stakeholder organisations to comment on the draft guideline during a set period, using email, social media and other promotional channels to encourage responses. Consultation usually lasts for 6 weeks, during which stakeholders can review the draft recommendations and supporting information. 

In NICE’s experience, some patient or public stakeholders find it helpful to have questions or a checklist to guide their response. NICE encourages patient organisations and other stakeholders to consider issues such as:

• How well do the recommendations:
  • cover the issues in the guideline scope that patients, their families, and carers consider important?
  • reflect what the evidence says about treatment and care
  • take account of the choices and preferences of people affected by the guideline, and the information and support they need
  • consider the needs of different groups (for example, children and young people, and people from black, Asian and minority ethnic groups)
  • use wording that is clear, easy to follow and respectful.
• Do the recommendations include anything that people affected by the guideline might find unacceptable?
• Is there any other evidence that should be included?
• Do the research recommendations cover key gaps in the evidence about important areas of patient and public experience? (NICE 2018)

Patient and public expert reviewers

When peer review by external individuals is a routine part of the process of guideline development, patients, members of the public or advocates should be included as expert reviewers. This inclusive approach to external review is recommended by major standard-setting agencies, such as the Institute of Medicine (2011; now the National Academy of Medicine). So, for example, all SIGN guidelines are reviewed in draft form by independent experts including at least 2 patient or public reviewers (SIGN 50 2019). At NICE, external review is mainly conducted through consultation with stakeholder organisations (2014). However, guideline developers may also consider arranging additional expert review of part or all of a guideline. Expert reviewers may include patients, members of the public and advocates, as well as health professionals. This review may take place during guideline development or at the final consultation stage. Expert reviewers are required to complete a declaration of interests form (NICE 2014; SIGN 50 2019).

Consulting patients and the public using online engagement methods

As discussed earlier in this chapter, public commenting is typically conducted online. Some guideline developers have used other online methods such as Delphi processes, voting tools, Wikis and discussion forums. Discussions could also be facilitated through social media channels, like Twitter, Facebook or an online patient forum. This kind of approach may be particularly useful for topics in which consultation with patient organisations might be limited and so a range of patient or public views is needed. It also allows the important flashpoints for patients, that appear in the guideline, to be framed in language that is easily understandable and relatable for members of the public.

Online methods can be particularly useful for engaging a lot of people who are geographically dispersed. This includes those who have difficulty attending face-to-face meetings because of illness or disability, and people who prefer a more anonymous method of contributing. Grant et al. (2018) examined the potential advantages and disadvantages of online engagement as part of a project to create a protocol that patients with Duchenne muscular dystrophy (DMD) and their carers could use to rate the perceived patient-centredness of guideline recommendations. From a rapid review of the literature on patient involvement in guideline development, the authors found that online methods can facilitate greater openness and honesty by patients, as well as having the potential to reflect the diversity of patient views. This can increase the utility of guideline products. The challenges of using online methods may include the extra time, skill and resources needed for patient engagement, and also the potential difficulty of involving specific patient populations. The authors concluded that online methods are most likely to be useful when guideline developers wish to engage a large, diverse and geographically dispersed group of patients, and have the required resources. The authors also suggest that online methods are particularly suitable when patients seek anonymity in order to share their views, and they are able to use online technology.

Khodyakov et al. (2020) suggest that an online modified-Delphi approach combining rounds of rating, anonymous feedback on group results, and a moderated online discussion forum is a promising way to involve large and diverse groups of patients and carers. They offer guidance on using such online approaches to facilitate engagement with patients, carers and other stakeholders in the guideline development process. The authors outline 11 practical considerations covering the preparation, implementation, evaluation and dissemination stages. Their first step is to co-develop an engagement approach with relevant patient representatives, such as a key patient advocacy organisation. The complete set of considerations proposed by Khodyakov et al. are reproduced below:

• co-develop an engagement approach with relevant patient representatives
• mirror methods used for expert and stakeholder engagement
• pilot-test the engagement approach
• recruit patients with diverse perspectives
• assemble a panel of adequate size and composition
• build participant research and engagement capacity
• build 2-way interaction
• ensure continuous engagement and retention of patients
• conduct scientifically rigorous data analysis
• evaluate engagement activities
• disseminate results.

Consulting individual patients and the public using research techniques

Guideline developers may undertake consultation using research techniques with individual patients and others, either to inform the scoping, review questions or development stages, or to test the relevance and acceptability of draft recommendations. This work typically uses methods such as focus group discussions, interviews and surveys. Some guideline developers use surveys as part of, or alongside, a routine public comment consultation process. Other developers use research techniques with patients and carers to supplement gaps in one or more of the following areas:

• important gaps in the evidence base on patient views, values, preferences and experiences  
• insufficient involvement or feedback from patient organisations (for example, for some guidelines or topics there may be no patient organisation with a focus on the topic)
• gaps in membership of the guideline development group in terms of patients’ perspectives (for example, a broader range of experience is required or the guideline covers a population not directly represented on the group, such as children and young people)
• gaps in information on the perspectives of seldom heard patients who are not part of an organised group or who don’t have an organisation to advocate for them, or potentially excluded groups, such as people from certain minority cultures or ethnic groups.

Before considering such work, it is important to check whether the information that the guideline developers are looking for might already be available. There may be relevant information on the views and experiences of patients and the public in the grey literature or from real-world evidence, including surveys conducted by advocacy organisations. For example, in the US the Listening to Mothers surveys are good examples of population-level resources about women’s experiences of care, their knowledge and preferences, with coverage of topics from before pregnancy to well into the postpartum period. These Childbirth Connection surveys have been developed in concert with multi-stakeholder advisory groups, including consumer representatives.

Consulting patients and the public using research techniques is an exceptional option requiring additional human and financial resources. Guideline developers need to consider the recruitment strategy and choice of methods carefully, including the methods for analysing data to ensure the data generated produces robust evidence to feed into work on the guideline. Group-based methods and interviews are best for exploring how people feel and exploring topics in detail. Surveys or questionnaires are useful for quantifying the extent to which people hold beliefs, values and attitudes, and how much they vary between groups of people, for example. 

Guideline developers need to ensure that those conducting this type of consultation have the relevant knowledge and skills, including expertise in research methodology and ideally expertise in conducting research with the relevant population. NICE commissions such work using a tender process. This involves interviewing prospective contractors to ensure they have appropriate expertise, policies and procedures for ensuring the safety and welfare of participants, as well as following best practice and the country’s legal requirements for working with the affected population. Consent, incentives, and other ethical issues should be considered, including whether formal ethical approval is required from the relevant research governance body. Ethical approval can take time, in some cases many months, and this should be considered in the timelines. Researchers and guideline developers should also consider how participants will receive feedback about their impact during and on completion of the work, including how they will be acknowledged.

Techniques for eliciting people’s views need to be tailored to the age, cognitive ability, and culture of participants. Materials and activities should be adjusted to suit participants and take into account any adaptations needed for people with physical or sensory impairments. In the UK, the National Children’s Bureau has produced guidance on how to conduct research with children and young people, as well as advice on involving them more actively in the research process (Shaw et al. 2011). Also in the UK, the Alzheimer’s Society’s toolkit provides information on how to recruit adults with dementia and gain their consent for research.

---