Physician Trainee Mistrust of the E-Portfolio and Workplace Based Assessment Process

Andrew Tomkins, Department of Health, Manchester University Hospitals NHS Foundation Trust, UK, Email: andrewtomkins@nhs.net

Received: 04-Aug-2021 Published: 25-Aug-2021

Introduction

Since its inception in 2005, the Electronic Portfolio (ePortfolio) has been utilised across all postgraduate medical training programmes across the UK, Europe and North America [1]. Defined as an ‘electronic tool for doctors to store and record a collection of evidence that demonstrates their learning achievements and abilities,’ it allows trainees in the UK to demonstrate that they comply with standards of competence, care and conduct in line with General Medical Council (GMC) Guidance [2,3]. An ePortfolio includes a variety of information that includes skills assessment, involvement in presentations and audit and reflective practice [1]. In 2018, UK national guidelines were published in order to support postgraduate trainees in reflective process and engagement with the ePortfolio [4]. Their release was prompted following growing concerns of repercussions of ePortfolio documentation, due to a highly publicised UK medico legal case involving a paediatrics trainee [5]. She was initially convicted for gross negligence manslaughter following the death of a six year old boy in hospital. Whilst the decision to remove the doctor from the GMC register was overturned, there was widespread misreporting that the doctor’s ePortfolio reflections were used as evidence against her. A statement released by the GMC dispelled this; however the doctor’s reflections had been seen by expert witnesses involved in providing evidence in the case.

Previous research specifically exploring trainee trust of the ePortfolio process has focused on General Practice trainees who have different ePortfolio requirements to physician trainees [6]. In this study, we aimed to address the paucity of research which exists into factors which impact upon physician trainee engagement with their ePortfolio. It was conducted at a time when the medico legal case was in the appeals process, prior to the release of specific reflective practice guidance.

Methods

As part of a dual phase mixed methods study into trainees’ perceptions of workplace based assessments (WPBA), two focus groups were conducted with fourteen higher specialty trainees across Genitourinary, Endocrinology and Geriatrics medical specialities in a training region in North West England. Appropriate Ethical and Research Governance approval was granted (Health Education England Research Governance approval; dated 11th May 2017 and Edge Hill University Research Ethics approval; reference FOHS200; dated 21 March 2018). In phase one we conducted an online, anonymous questionnaire involving 25 respondents; which is reported previously [7]. Participants were selected as they had experience both of engaging with the ePortfolio for several years and completion of assessments for other colleagues. We utilised purposive sampling. Participants were contacted two weeks prior to the focus groups, via email (which included a Participant Information Sheet and consent form) and focus groups were conducted following regional specialty training in May 2018. Participants could withdraw consent at any stage. Grounded Theory methods were applied, thus allowing themes to be generated through analysing the data [8]. Audio recording devices were used during the focus groups and data was transcribed verbatim, with data analysis occurring simultaneously with data collection. Line by line coding and memo writing was used until theoretical saturation was reached. Preliminary analysis was undertaken by the lead researcher (AT), with all members of the research team reviewing, discussing and agreeing the coding, and with consensus achieved regarding ultimate themes that were generated.

Results

Participants reported a heightened mistrust of the ePortfolio and reflective process since the high profile medicolegal case. This related to concerns of permanency of documentation and the potential for this to negatively impact upon career progression and the identification of inherent biases present in the assessment process. Illustrative quotations are presented throughout the text. Table 1 summarizes key themes and subthemes.

Theme 1: Concerns of documentation permanency and negative training implications

Participants reported a concern of the permanency of the recording of knowledge gaps in the ePortfolio and the perceived potential negative repercussions of recording suboptimal performance. Concerns were expressed that training issues would not be actioned upon until the end of the training year, which may impact upon progression.

‘I have concerns about the action taken following less than satisfactory SLEs^*…. I think it’s that continual review that’s needed, somehow, so it doesn’t get to the end of the year and then “by the way I’m going to have to say you’re not a good enough trainee”.’

^*SLE=Supervised Learning Events

There was an identified tension between utilising reflective practice to learn from mistakes and a concern that this may be detrimental to career progression. Several participants reported having experienced issues because of reflection recorded on their ePortfolio.

‘So, a personal example was a complaint. I reflected on it on my ePortfoio straightaway; what I’d written was a direct description of events and then my thoughts on it. And what I got told in the ARCP* was, “has reflected but very defensive.” And the criteria for me moving on from the ARCP were to reflect again and not make it defensive.’

^*ARCP=Annual Review of Competency Progression.

A general mistrust of the anonymity offered by seemingly anonymous multi rater assessments was reported. Several participants had experienced a breakdown of anonymity when completing assessments for colleagues, leading to fractured working relationships. Often trainees were expected to complete multi rater assessments for senior colleagues who may influence future career opportunities, thus creating a challenge for the individual in providing objective feedback.

‘You shouldn’t put anything in a form that you wouldn’t be prepared to say to someone’s face… so how do you create that environment to feed back that is safe and supported?’

‘With seniors, if you’re looking for jobs in the future and things, you just don’t want to be critical and some people are open to criticism better than others.’

Participants felt that biases may be introduced to assessment outcomes depending on the quality of working relationships they have with their Educational Supervisors. The multi assessor nature of the ePortfolio was cited as offering some protection if there were poor Trainee Supervisor relations.

‘It encourages an Educational Supervisor to be more unbiased if there are these other things recorded. Because if they suddenly write a really bad report then it’s, “hold on a minute, what’s going on here? There’s something a bit weird.” Whereas, if they know it’s just them, actually they can write anything they like.’

The recent widely reported UK medico legal case was felt to have heightened participant mistrust of ePortfolio and honest reflection.

‘I mean, I’m referring to the doctor involved in the medicolegal case yes so they say… “Don’t mention anything.” Because the thing which put her in trouble was her portfolio and her reflection on that event which her Supervisor forced her to do…’

The case was cited as being a reason for participants to change their own reflective practice to avoid negative training implications. Tension existed between ensuring the ePortfolio contained sufficient information to allow progression of training yet did not include potentially incriminating information.

‘It’s really tricky. It becomes a political exercise basically, reflecting, because you’ve got to avoid incriminating yourself, because of that case…I’ve actually stopped doing it.’

Whilst the majority of participants reported being more cautious with their reflective practice documentation, a proportion felt its use should not be hampered by a self-editing process, rather should be used to develop self-reflective practice. It is apparent from the focus groups that participants seemed to have varying understanding of the purpose of the ePortfolio.

‘Your reflection is your reflection that is my thoughts, if you want to think differently it’s in the Educational Supervisor Report or some other Consultant Report but those are my reflections.’

Theme 2: Inherent biases of ePortfolio

Participants reported perceived inherent biases existing within the ePortfolio. It was felt to play a role in disabling individuals to raise safety concerns within their organizations for fear of negative repercussions on their own progression.

‘The portfolio suppresses all the weaknesses in that Trust^* or area, you can’t be criticizing anything, you’re not allowed.’

^*A hospital Trust is a hospitals organization which provides secondary healthcare services within the English National Health Service.

Several participants expressed that raising concerns relating to organizational issues may lead to negative training outcomes and may hinder their own career progression.

‘If it turns out that you have to raise some really serious issues when you’re in a hospital, then you are not going to get a good report. And it might be that actually potentially you’re a very good doctor, but you have nothing else to fall back on.’

This led to some individuals feeling complicit in organizational issues in order to safeguard their own career.

‘So the only way to do it is to comply and when you have these scandals, I know it’s a very political thing, but this is the system that keeps that going. That’s the problem, because everybody is intelligent, they know how to stay alive and that’s unfortunately the way to do it.’

Additionally, participants reported witnessing colleagues manipulate ePortfolio outcomes by preferential selection of lenient assessors who are likely to provide positive feedback on performance.

‘ I think there’s a bias with the portfolio because you are going to send your assessments to people who you know will fill it in well for you, so are you making the portfolio look at the positive side of you?’

Participants identified that they may be at a disadvantage by using their own ePortfolio objectively, rather than preferentially selecting assessors as per the reported cultural ‘norm’.

‘Statistically it’s scientifically disadvantaging yourself, because basically everyone’s sending MSF^* to the people they thought would give them good feedback’

MSF=Multi Source Feedback

Participants reported flaws in the current assessment process due to the possible challenges in assessors lacking objectivity, particularly if working in an environment where concerns have been raised by the trainee.

‘Unless you can take the assessment system out of the local place, in my opinion, it’s much fairer, and more unbiased. The person concerning who’s assessing you has never met you; if you can do it like that…that’s where I see the inherent problem is with work place based assessments.’

Participant suggestion for improvement

Participants had the opportunity to suggest improvements of the existing ePortfolio and reflective process. Proposed suggestions included ensuring clarity on how best to reflect upon clinical incidents or training deficiencies in the ePortfolio. Participants felt that earlier identification of training difficulties would be beneficial, rather than at the end of a training year, which would allow timely provision of support and may negate training issues. Participants felt that a greater onus being placed on trainees’ supervisors to monitor engagement with the ePortfolio and progression would be beneficial. Additionally, participants felt that improvements in transparency within hospital organizations would lead to enhanced trust in the ePortfolio process. Finally, participants felt that an enhancement of the confidentiality of seemingly anonymous, multi rater assessments would enhance confidence in their use.

Discussion

This study highlights several issues which may impact upon physician trainee engagement with their ePortfolio. Mistrust is mainly driven by concerns of negative training repercussions. The identified fragility of trainees’ relationships with their ePortfolio may threaten the educational and professional development it aims to support [9].

Transferability of findings may be limited by geographical and specialty factors. Our research was conducted in one geographical region in the UK; there may therefore be differences in the level of training and clarity of expectation communicated across other training regions. Our research focused on physician trainees; there may be different requirements for non-physician specialties. Additionally, we acknowledge the small sample size which may not be representative of the wider trainee opinions.

Previous research into factors relating to physician trainees identified several advantages to utilising a reflective ePortfolio, providing it is correctly implemented [10]. However, possibly due to the timing of when our study took place which coincided with the high profile legal case, the focus group focused substantially on negative aspects that caused them concern.

Conclusion

As our study was conducted prior to the release of national guidance on reflective practice and the ePortfolio, it is yet to be established if this has provided sufficient reassurance for negate these concerns, which future research may wish to explore. Additionally, further exploration into trainee perceptions of the purpose of the ePortfolio, including reflective practice, may be of benefit.

Conflict of Interest

The authors declare no conflicts.

References

1. Heeneman S, Driessen EW. The use of a portfolio in postgraduate medical education - reflect, assess and account, one for each or all in one? GMS J Med Educ. 2017; 34 (5): 57.
2. NHS Health Careers, NHS.
3. General Medical Council. Generic Professional Capabilities Framework.
4. Academy of Medical Royal Colleges and Conference of Postgraduate Medical Deans of the United Kingdom. Academy and COPMeD Reflective Practice Toolkit. 2018.
5. Dyer C. Hadiza Bawa-Garba wins right to practise again. BMJ. 2018; 362:k3510.
6. Rouse J, Green C. ‘What Factors Influence General Practice Specialist Trainees’ Engagement with their E-portfolio?’. MedEd Publish. 2018; 7(3): 35.
7. Tomkins A, Sherratt C, Patel M. Do Supervised Learning Events reflect clinical competency and support ‘Trainee in Difficulty’ identification? Physician Trainees’ perceptions. J Contemp Med Edu. 2019; 9(4): 75-86.
8. Charmaz K. Constructing grounded theory: A practical guide through qualitative analysis. Sage Publications. 2006.
9. Tochel C, Haig A, Hesketh A, Cadzow A. The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12. Medical Teacher. 2009; l31(4): 299–318.
10. Tailor A, Dubrey S, Das S. Opinions of the ePortfolio and workplace-based assessments: a survey of core medical trainees and their supervisors. Clinical Medicine. 2014; 14 (5): 510-6.