Abstract
Background: The University of Arizona Emergency Medical Services (UAEMS) is a collegiate EMS agency that responds to 911 medical calls on the University of Arizona campus in a basic life support (BLS) non-transport capacity. Objective: This study assesses UAEMS’ patient evaluations when referenced to Tucson Fire, the municipal agency they respond in tandem with. Methods: In this study, agreement between Tucson Fire and UAEMS regarding final patient dispositions is examined across multiple categories with Cohen’s kappa test and a sensitivity/specificity analysis to evaluate how similar UAEMS is in their patient evaluations. Results: When compared to Tucson Fire, UAEMS’ evaluation for transport necessity shows high sensitivity and moderate specificity. Conclusion: UAEMS demonstrates a more conservative evaluation of patient transport necessity and provides a case-study showing the value of Collegiate-Based Emergency Medical Services.
Introduction
Emergency Medical Services (EMS) agencies in the United States play an important role in ensuring public health and safety by responding to medical emergencies1. EMS agencies can vary both in the level of care provided (basic life support [BLS] and advanced life support [ALS])2 and by who runs them (commercial, municipal, etc.).3 The field of EMS, like other healthcare sectors, is facing an efficacy challenge as a result of healthcare reform as part of the evolution of evidence-based medicine4. Because of the variability inherent in EMS, a significant amount of research is needed to re-analyze the effectiveness and preparedness of individual agencies, and the system as a whole.4,5
Within the emergency healthcare system, it is common for EMS agencies, especially those in rural locales, to assist or request assistance from other organizations that are part of neighboring jurisdictions.6,7 The city of Tucson has one such system in place, a mutual-aid agreement created between Tucson Fire Department (TFD) and two nearby fire agencies, that helps prioritize emergency response regardless of jurisdiction.8 In total, this agreement includes more than 40 fire stations, spans 640 square miles, and is expected to reduce response times by 15 percent.8 Such agreements and understanding between EMS agencies are important for the delivery of effective EMS care.9,10
Collegiate-based emergency medical services (CBEMS) are EMS agencies that are created to help contribute to healthy and safe communities on college or university campuses.11 The capabilities of CBEMS agencies vary across universities; some organizations strictly function during school events, while others operate 24/7, 365 days a year.12 While some organizations have the ability to transport their patients, other agencies serve solely in a rescue unit capacity, necessitating they work in tandem with another EMS agency to provide transport.12 The University of Arizona Emergency Medical Services (UAEMS) is a CBEMS agency that was established at the University of Arizona in 2012.13 UAEMS responds to emergency medical calls on a 24/7, 365-day basis within the bounds of the University of Arizona’s main Tucson campus, while additionally providing standby services for special events hosted within the university’s jurisdiction.13
Currently, UAEMS provides emergency medical services in a BLS non-transport capacity.14 In the event of a 9-1-1 emergency medical call on the university campus, UAEMS responds jointly with a TFD unit;15 both work in unison to provide emergency medical care. This cooperation is made possible due to the integration of UAEMS into Tucson Fire’s Computer-Aided Dispatch system, allowing UAEMS to be dispatched as if they were a response unit with Tucson Fire. For ALS-level acuity calls, an ALS-level transport unit is dispatched to respond and for BLS-level acuity calls, a non-transport unit (usually a fire engine staffed with a paramedic fire captain) is dispatched to respond.
Responding ALS transport units are usually quicker to arrive to campus than BLS transport units as there are about four ALS medic units within a few mile radius of campus while BLS transport units are about six miles away. Regardless of whether the call was deemed BLS or ALS, patients transported by an ALS transport unit were considered as an ALS disposition for Tuscan Fire.
This study’s primary purpose is to examine UAEMS’ capability to determine patient acuity and transport necessity in comparison to Tucson Fire units by comparing the overall patient disposition determined by both agencies. Due to the nature of collegiate EMS and the high turnover for providers due to graduation, this study will also assess the evaluation capabilities of individual supervisors to determine if significant variation exists. Additionally, as little literature exists regarding comparisons of collegiate and municipal EMS agencies, a tertiary objective is to provide a case study about collegiate EMS effectiveness and evaluation capabilities with municipal EMS as a reference.
Materials and Methods
A total of 599 call records were generated for the calendar year of 2020 by the University of Arizona Emergency Medical Services. Data from these call records was extracted using the Report Writer functionality of ImageTrend, an interface that allows for the generation of reports containing the minimal information requested. The data collected did not include identifiable markers, and thus access to Personal Health Information (PHI) was avoided using this method. A waiver of consent for the data used in this study was obtained from the University of Arizona Institutional Review Board (see Appendix A).
Of these 599 call records, 368 were complete Patient Care Reports (PCRs), 160 were designated as missed calls (a result of situations where UAEMS does not make patient contact due to cancellation or other reason), 35 were designated public assists (low-acuity calls that do not require full PCRs to be generated), 23 were designated late arrivals (PCRs which are potentially incomplete due to arrival after Tucson Fire’s evaluation has begun) and 13 are uncharacterized (Figure 1).
Figure 1: A visual representation of the total call categorization
Every PCR includes a section for the final patient disposition (ALS transport/BLS transport/patient refusal) as independently determined by both UAEMS and TFD; this information will be used to compare both agencies’ assessment of the same patients. While only complete PCRs containing Tucson Fire and UAEMS’s patient dispositions were used to analyze the effectiveness of UAEMS’ medical evaluations, an additional 37 PCRs were excluded due to inaccurate data for patient disposition. Upon retrospective review, one supervisor was found to have conflated final patient disposition and UAEMS operating capacity, resulting in “BLS” being selected for every patient disposition. This was confirmed upon review with said supervisor, and verbal confirmation was made with every other supervisor who responded to a 911 call in 2020 to ensure this mistake was isolated to one supervisor only. With the exclusion of these 37 reports, a total of 331 PCRs were used for this statistical analysis.
Each PCR used for this study was sorted into seven distinct call types: Assault, Behavioral, Ethanol/Overdose, Falls, Medical, Motor Vehicle Accident (MVA) and Other Trauma; a call type of Not Applicable (N/A) was also included for any call that does not fall under the preceding types. Each PCR was also sorted by the level of resources dispatched (ALS or BLS) and by “primary impression”, or what was determined to be the main cause of the emergency.
To determine statistical correlation between Tucson Fire and UAEMS patient disposition decisions, Cohen’s kappa test was performed, using Tucson Fire’s disposition as the baseline for comparison. The kappa result of each test will be between -1 and 1, with 1 implying perfect agreement and any negative number showing imperfect agreement, more than what would be expected by chance.16 The sensitivity and specificity of UAEMS’ judgment was also evaluated to determine where the bulk of any disagreements lie. For this analysis, a “false positive” is when UAEMS has a transport disposition compared to TFD’s non-transport disposition; when comparing ALS vs. BLS transport decisions, a “false positive” is when UAEMS has an ALS disposition compared to TFD’s BLS disposition.
Results
Figures 1 and 2 present the kappa value, sensitivity, and specificity of UAEMS’ decision of transport vs. non-transport in each call category; the values for UAEMS’ ALS and BLS decisions are also shown for comparison. Note that the “Assault” call type is not shown on the κ value graph, as the small sample size prohibited the usage of Cohen’s kappa test. UAEMS’ decision-making between ALS and BLS by specific call type is likewise not shown above, also due to small sample size concerns. While the transport decision sample includes all 331 calls from above, only calls in which UAEMS and TFD both agreed on transport were utilized for the level of care comparison. As a result, the ALS vs. BLS analysis has a sample size of 157 calls. Figures 3 and 4 illustrate an evaluation of transport decision-making for each of the 11 UAEMS shift supervisors who responded to more than 10 included calls over the 2020 calendar year. Figures 5 and 6 show the kappa value, sensitivity, and specificity of UAEMS’ decisions regarding transport and level of care for calls separated by initial dispatch acuity. The sample used for these two figures contains only 312 calls, as only calls with recorded and valid dispatch codes were included. Additionally, only 140 of the 312 calls contained a transport disposition for both TFD and UAEMS, and these calls solely were a part of the sample used to evaluate patient acuity decisions.
Finally, an evaluation of UAEMS’ transport decisions was made for each primary impression used for a total of 10 calls or more, shown in Figures 7 and 8. While all 331 calls were included in this analysis, only 6 primary impressions were indicated in 10 or more calls, due in part to a total of 64 different impressions being selected across all calls included.
Discussion
Recent research suggests that, for medical research particularly, kappa values of κ ≥ 0.8 and κ ≥ 0.9 should be considered strong and “almost perfect” agreement respectively.17 As such, agreement regarding transport decision-making was shown to be “almost perfect” when considering calls of a Medical, MVC/MVA and unknown (N/A) nature (although the “N/A” call type suffers from a relatively small sample size); further, agreement between UAEMS and TFD regarding whether to transport a patient was considered strong for every call type evaluated.
Most of the variation in UAEMS and TFD transport decisions resulted from having a dissenting evaluation of “transport” for a patient. This is supported by the near-perfect sensitivity rates for each type of call, even when looking across all dispatch levels, primary impressions, and call types evaluated. Looking specifically at calls separated by primary impression, while kappa values fluctuate between 0.6 and 1 (representing “moderate” and “near perfect” agreement respectively), sensitivity is a perfect 100% for all but one patient disposition.
This trend is mirrored in the evaluation of individual supervisors; while the values for the comparison of primary impressions decrease compared to TFD impressions when grouped by individual supervisor, the sensitivity of “transport” decisions is a perfect 100% for 13 of the 15 supervisors included in this analysis. The two supervisors with sub-100% percentages (sensitivities of 87.5% and 90%) had one call each where they gave a “non-transport” disposition for a patient TFD determined should be transported.
One potential confounding factor central to this analysis that should be addressed is the independence of UAEMS’ evaluations. While TFD’s transport and patient acuity determinations are largely independent, UAEMS is cognizant of TFD’s decisions when inputting their final patient assessments in the post-call PCR, even if TFD’s evaluation has no physical bearing on UAEMS’ evaluation. With respect to individual supervisor evaluations, 6 of the 15 supervisors examined gave identical “transport” decisions as TFD; however, 3 of these 6 supervisors disagreed with TFD in regard to “patient acuity” evaluations, leaving only 3 supervisors with no disagreements. While UAEMS consistently demonstrates high sensitivity for “transport” evaluations across all categories, the disagreement in “patient acuity” evaluations demonstrates that UAEMS shows independent evaluation skills when assessing a patient’s condition.
The low specificity of UAEMS’ patient acuity evaluation (75%) is likely a result of several factors. One specific factor is the difference in BLS transport standing orders between UAEMS and TFD before Fall 2020 (See Limitations). Additionally, UAEMS is staffed exclusively by EMTs, while TFD’s “BLS’’ non-transport response units are occasionally staffed by paramedics7; this means that while TFD will send paramedics to evaluate situations dispatched at an ALS priority, they sometimes also send paramedics to evaluate BLS priority situations. As such, it is possible that a predilection exists for acuity decisions to match the level of training of the evaluator, causing UAEMS’ evaluations to lean towards BLS, and TFD’s to lean towards ALS. No firm conclusions can be made at this juncture without further investigation.
While transporting patients not requiring ambulance transport to an ED isn’t desirable, it is preferable to situations where a patient requiring ambulance transport to an ED is deemed stable enough to not necessitate emergency transport. Based on the consistently high sensitivity rates, it can be concluded that UAEMS provides a statistically similar patient assessment when compared to assessments by TFD.
Limitations
Some limitations were encountered during this experiment that should be addressed. While UAEMS’ yearly call volume is normally around 900 calls, the COVID-19 pandemic kept many students at home and reduced the population on campus, which in turn reduced the number of calls available for statistical analysis. In addition, the data used for this study is second-hand information, whose accuracy is reliant on the provider writing the PCR. Finally, UAEMS and TFD possessed different standing orders until Fall 2020 when UAEMS adopted TFD’s standing orders (See Appendix A); as such, some disposition differences can be attributed to differences in standing orders regarding patient disposition and not differing evaluations of the same patient.
Conclusion
With TFD’s evaluation set as the standard, UAEMS’s evaluations of a patient’s transport necessity has been demonstrated to possess near-perfect sensitivity. When paired with a lower (but still high) specificity, UAEMS demonstrates a more conservative evaluation of patient transport necessity. This case study has also shown, albeit with one example, that collegiate emergency medical service agencies can provide safe patient evaluations. Additionally, shift supervisors were generally consistent in sensitivity, indicating the effectiveness of the training of supervisors.
Acknowledgements
The authors would like to thank Sophia von Hippel for their contributions to this research study. The authors would also like to acknowledge the University of Arizona Emergency Medical Services for their continued dedication to the safety and well-being of the University of Arizona campus.
References
- Van Mulligan M, Mitchell III JP, Tucker J, Arkedis J, Caravalho D. An Analysis of Prehospital Emergency Medical Services as an Essential Service and as a Public Good in Economic Theory. Washington, DC: US Department of Transportation National Highway Traffic Safety Administration; 2014. Report No. DOT HS 811 999a.
- Cone DC, Brice JH, Delbridge TR, Myers JB. EMS Personnel. In: Emergency Medical Services: Clinical Practice and Systems Oversight. Hoboken, NJ: Wiley-Blackwell; 2021.
- Freeman VA, Slifkin RT, Patterson PD. Recruitment and retention in rural and Urban EMS. Journal of Public Health Management and Practice. 2009;15(3):246-252. Doi:10.1097/phh.0b013e3181a117fc
- National EMS Advisory Council. EMS Makes a Difference: Improved Clinical Outcomes and Downstream Healthcare Savings. https://www.ems.gov/NEMSAC-advisories-and-recommendations/2009/NEMSAC_Position_Statement_EMS_Makes_a_Difference_Dec_2009.pdf. Published December 2009. Accessed December 30, 2021.
- Dawson DE. National Emergency Medical Services Information System (NEMSIS). Prehospital Emergency Care. 2006;10(3):314-316. doi:10.1080/10903120600724200
- Hanfling D, Altevogt BM, Viswanathan K, Gostin LO. Prehospital Care Emergency Medical Services (EMS). Crisis Standards of Care: A Systems Framework for Catastrophic Disaster Response. Washington, DC: National Academies Press; 2012. https://www.ncbi.nlm.nih.gov/books/NBK201058/. Accessed December 30, 2021.
- Knott A. Emergency medical services in rural areas: The supporting role of state EMS agencies. The Journal of Rural Health. 2003;19(4):492-496. doi:10.1111/j.1748-0361.2003.tb00587.x
- Sayers J. Fire agencies say mutual aid agreement will help cut response times in emergencies. Arizona Daily Star. https://tucson.com/news/local/fire-agencies-say-mutual-aid-agreement-will-help-cut-response-times-in-emergencies/article_24f200f1-e661-5f6b-8cce-fb432c24101e.html. Published January 23, 2020. Accessed January 3, 2022.
- Narad RA. Coordination of the EMS system: An organizational theory approach. Prehospital Emergency Care. 1998;2(2):145-152. doi:10.1080/10903129808958860
- Boyd DR. Emergency Medical Services Systems Development: A national initiative. IEEE Transactions on Vehicular Technology. 1976;25(4):104-115. doi:10.1109/t-vt.1976.23630
- Bartho MJ, Goode T, Gan G. Determinants of Volunteer Collegiate-Based Emergency Medical Service Budget Size. The Journal of Collegiate Emergency Medical Services. 2021;4(2):22-27.
- Fisher J, Ray A, Savett SC, Milliron ME, Koenig GJ. Collegiate-Based Emergency Medical Services (EMS): A Survey of EMS Systems on College Campuses. Prehospital and Disaster Medicine. 2006;21(2):91-96. doi:10.1017/s1049023x00003411
- History. universityems. https://www.ems.arizona.edu/history. Accessed May 11, 2021.
- Bartsch J. UA Emergency Services offers 24/7 health care, among other providers. The Daily Wildcat. https://www.wildcat.arizona.edu/article/2020/07/sc-rg-emergency. Published July 15, 2020. Accessed May 11, 2021.
- Emergency operations. Official website of the City of Tucson. https://www.tucsonaz.gov/fire/emergency-operations. Published February 11, 2020. Accessed January 1, 2022.
- Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;20(1):37-46. doi:10.1177/001316446002000104
- McHugh ML. Interrater Reliability: The kappa statistic. Biochemia Medica. 2012:276-282. doi:10.11613/bm.2012.031
Supplementary Materials
Appendix A: This research project was approved by the University of Arizona’s Institutional Review Board, project ID# 2103647633. A waiver of consent was obtained for the data in this study. Due to the patient population studied herein, specific permission for inclusion of patient records of minors was sought and approved by the IRB board.
Appendix B: Tuscan Fire Standing Orders – Excerpt (Click to Download PDF)
Appendix C: UAEMS Standing Orders prior to Fall 2020 – Excerpt (Click to Download PDF)
Appendix D: Supplemental Material (Click to Download PDF)
Author & Article Information
Hans Bradshaw, MD is an Assistant Professor of Emergency Medicine and Pediatrics and serves as the Emergency Medicine Clerkship Director at the University of Arizona College of Medicine-Tucson. Dr. Bradshaw currently also serves as the Medical Director for the University of Arizona Emergency Medical Services. Brittany Wright, BHS, BS, EMT-B is a recent graduate from the University of Arizona and previously held the position of Continuous Quality Improvement Officer at the University of Arizona Emergency Medical Services. Bianca Carrasco, EMT-B, is the current Chief and Executive Director at the University of Arizona Emergency Medical Services and a senior at the University of Arizona. Jenna Wise, EMT-B, is a current Continuous Quality Improvement Officer at the University of Arizona Emergency Medical Services and a junior at the University of Arizona. Timothy Gustafson, BS, EMT-B, is a current graduate student at the University of Arizona and previously held the position of Chief and Executive Director at the University of Arizona Emergency Medical Services. Jacob Robishaw-Denton, BS, EMT-B is a first-year medical student at the University of Arizona College of Medicine-Tucson and previously held the position of Deputy Chief of Operations at the University of Arizona Emergency Medical Services.
Author Affiliations: From University of Arizona College of Medicine, University of Arizona Emergency Medical Services – in Tucson, AZ (H.B., J.R.). From University of Arizona Emergency Medical Services – in Tucson, AZ (B.W., B.C., J.W., T.G.).
Address for Correspondence: Jacob Robishaw-Denton | Email: jkdenton16@email.arizona.edu | Mailing Address: University of Arizona Emergency Medical Services, 1715 E 7th Street, Tucson, AZ 85719, USA
Conflicts of Interest/Funding Sources: By the JCEMS Submission Declaration Form, all authors are required to disclose all potential conflicts of interest and funding sources. All authors disclosed their affiliation with the University of Arizona Emergency Medical Services and declared no other conflicts of interest. All authors declared that they did not receive funding to conduct this research and/or associated with this work.
Authorship Criteria: By the JCEMS Submission Declaration Form, all authors are required to attest to meeting the four ICMJE.org authorship criteria: (1) Substantial contributions to the conception or design of the work; or the acquisition, analysis, or interpretation of data for the work; AND (2) Drafting the work or revising it critically for important intellectual content; AND (3) Final approval of the version to be published; AND (4) Agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
Submission History: Received May 14, 2021; accepted for publication December 20, 2021
Published Online: February 25, 2022
Published in Print: February 25, 2022 (Volume 5: Issue 1)
Reviewer Information: In accordance with JCEMS editorial policy, Original Research manuscripts undergo double-blind peer-review by at least two independent reviewers. JCEMS thanks the anonymous reviewers who contributed to the review of this work.
Copyright: © 2022 Bradshaw, Wright, Carrasco, Wise, Gustafson & Robishaw-Denton. This is an open access article distributed under the terms of the Creative Commons Attribution 4.0 International (CC BY 4.0) License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. The full license is available at: https://creativecommons.org/licenses/by/4.0/
Electronic Link: https://doi.org/10.30542/JCEMS.2022.05.01.04
