In April last year, we announced that we had awarded a grant to Liverpool School of Tropical Medicine to allow them to develop and trial an SMS referral system aimed at providing young people with better access to sexual health services. The idea was that patients would visit an informal drug store in their village, whereupon the drugstore owner would send an SMS referral to a health clinic/formal dispensary. They would then receive an automated reply with a unique code to be given to the patient. The patient would then travel to the health clinic or dispensary where they would present their code and receive a fast-track, professional service. As the project got underway, the principal researcher – John – noticed a high rate of errors in SMS sent and received. I asked him why this was:
I agree, the error messages at the health facility level (dispensary/health centre) is still too high. At the start, errors in drug shop messages were similarly high, but we worked hard to reduce them.
However, although we want dispensaries/health centres to send correct messages as much as the drug shops, the fact is that dispensary SMS errors are not as detrimental to the patient or to the data itself, because it is last in the communication chain. I will explain the reasoning: If you look at the messages with errors closely, you will see that what constitute errors are either a space, a full stop, an apostrophe, or a comma misplaced or shouldn’t be there. These don’t distort the message interpretation. When I look at that data I don’t discard these ‘incorrect’ messages. At analysis stage, these data will be cleaned and analysed together with the ‘correct’ messages. That doesn’t mean we’re not worried about the rate of errors at dispensary though.
What increases the number of error messages at the dispensary/health centre levels is that their message is harder (with random patient IDs) and longer (with diagnosis, treatments and advice codes). Also, I am assuming that clinic staff are busier and have little time to concentrate on texting while other patients are waiting. Our assumption at the start of the intervention was that clinical officers at dispensary/health centre level would be less prone to making errors, owing to their higher level of medical education in comparison to drug store attendants. But the evidence is showing that that assumption doesn’t hold. We have conducted two major training sessions in the field in addition to a third pilot training which all pointed to errors among health clinic staff. Another observation is that when you look at texting patterns, you realise that there are fewer patient referrals and an increase in errors the longer it is since the training was held. This was our basis for concluding that we need quarterly re-orientations, to keep constant feedback with them in the field. We have also designated a Master’s Student who will be going to Mwanza in May/June 2013 to explore reasons for patients’ non-attendance of health facility after referral.
These are the sorts of challenges that you can anticipate before a project, but there’s very little you can do at that stage to prevent them. It’s also a good example of the way in which human error can be the undoing of structured data formats. As far as a computer is concerned, a missing comma or pair of parentheses can make messages look like incomprehensible gobbledygook, thereby churning out an error message. A human brain, on the other hand, can very quickly tell the difference between a nonsensical message and a message with a missing comma, but it can’t perform millions of calculations a second like a computer can. So while it may be a bit time consuming for John and his team to go through message after message, it’s a necessary evil. It’s also sound evidence of the need for repeated training and exposure, which is necessary to ensure that any project enjoys the sort of long-term engagement necessary.
As a result of this feedback, we have awarded an additional grant of £4,695 to LSTM for their work. This will enable John and his team to host reorientation meetings with local district medical officers (DMOs). The hope is that this top-up training will enable the DMOs to run refresher courses without the need for LSTM involvement, thereby ensuring the long-term sustainability of this project.
You can read more about the project here. With thanks to John Dusabe and the LSTM for providing much of the text for this post.