Skip Navigation

Internal Reviews For New Programs

Policy

It is the policy of the GMEC at Emory University School of Medicine to assure that residency programs and fellowships are compliant with ACGME requirements and that all residents receive outstanding educational experiences.    The institution will assure compliance by conducting internal reviews scheduled mid-cycle between RRC site visits, communicating with each program regarding ACGME requirements and mentoring Program Directors regarding preparation for RRC site visits. 

GOALS FOR INTERNAL REVIEWS

The GMEC Review Panel will review current and historic program documents, and interview program faculty and residents, to assess:

  1. The residency program's compliance with ACGME Institutional, Common and specialty/subspecialty-specific Program Requirements pertaining to the program, including:

    - Professionalism, personal responsibility, and patient safety
    - Transitions of care
    - Alertness management/fatigue mitigation
     -Supervision of residents
    - Clinical responsibilities
    - Teamwork
    - Resident duty hours (including ongoing and effective monitoring system)
  2. The program's educational objectives and effectiveness of the program in meeting its objectives;
  3. The adequacy of educational and financial resources provided to support the program;
  4. The effectiveness of the program in addressing areas of noncompliance and/or concern in previous ACGME accreditation letters and, if necessary, in previous internal reviews;
  5. The effectiveness of the program in using appropriate evaluation tools and dependable outcome measures to assess competence in each of these areas;
  6. The effectiveness of educational outcomes in the ACGME general competencies;
  7. Annual program improvement efforts in:
    • resident performance using aggregated resident data;
    • faculty development;
    • graduate performance including performance of program graduates on the certification exam;
    • program quality, including:
      • Residents and faculty annual confidential, written evaluations of the program
      • The program’s use of the results of residents’ assessments of the program together with other program evaluation results to improve the program.
      • Effectiveness of implementation of the program’s written plan of action emanating from the last annual program review
      • Any other issues or concerns which may properly come before the GMEC.

GENERAL PROCESS FOR INTERNAL REVIEWS

  1. The GMEC will conduct the internal review within 2 months of the midpoint of the program’s cycle.   The cycle length is the time between the effective date and the date of the next site visit, both of which are defined by the ACGME and found in the program’s most recent letter of report.  The midpoint is defined as the date of the GMEC meeting at which the Review Panel’s report is presented to the GMEC for its decision. 
  2. The GME office will notify the Program Director of the need for the internal review 3 months prior to the review, and, working in conjunction with the Program Director will establish the date, time and location for the review. 
  3. The Program Director must send the documents to be used for the internal review to the GME office at least 2 weeks prior to the review. 
  4. The members of the Review Panel will be appointed primarily from the GMEC, although faculty members and residents at – large may serve on review panels.
  5. The Associate Dean reserves the option of reassembling the Review Panel and the program’s representatives or of speaking with members of the department if questions remain after the review. This occurrence shall be noted in the final report of the internal review.
  6. The Panel will develop and write the report within one week of the date of conducting the internal review.
  7. The report will be presented at the next possible meeting of the GMEC that is within 2 months of the midpoint. 
  8. The GMEC could accept and endorse or add/delete/alter the recommendations of the Review Panel.  The GMEC will always offer the final recommendation.
  9. After the GMEC meeting, the DIO will send the final report to the Program Director and to the Chair of the Department. 
  10. 10.  The GMEC will expeditiously follow up with the program to assure that the Program Director is developing an action plan to resolve the issues and could request one or more progress reports.
  11. In the event that the ACGME schedules a survey earlier than originally anticipated thereby changing the midpoint, the GMEC will schedule an internal review to meet the new midpoint. 
  12. In the event that a program receives a cycle of 2 years or less, the GMEC will ask the DIO immediately to meet with the program’s residents or fellows and leadership to begin an intense oversight process to include an internal review at the midpoint and other meetings as necessary to resolve areas of noncompliance.  The GMEC will notify the Chair of the department and the Dean about these letters of support. 
  13. If the program due for an internal review has no residents enrolled at the midpoint of the accreditation cycle, then the GMEC will conduct a modified internal review at the ACGME defined midpoint to ensure that the program has all required elements in place prior to enrolling a resident, has maintained adequate faculty and staff resources, clinical volume and other necessary curricular elements to comply with the institutional, common and program requirements.  A second internal review then will be completed within the second six month period of the resident’s first year in the program.  

MEMBERS OF THE INTERNAL REVIEW PANEL

  1. The DIO or a School of Medicine faculty designee, including the Assistant Dean of GME will chair the Review Panel.
  2. Assistant Dean for GME or faculty designee as the administrator.
  3. At least one additional faculty member from the GMEC or faculty at-large.
  4. Other faculty members as appointed by the DIO.
  5. One or more residents from the GMEC or residents at-large.
  6. An administrative assistant from the GME Office.
  7. Panel representatives must never be members of the training program under review.  Likewise, if the DIO is a member of the department under review, then the DIO must appoint another faculty member to chair the review. 

PROGRAM INTERVIEWEES

  1. The Program Director must attend.
  2. The Associate Program Director when one has been appointed must attend.
  3. The Department Chair or designee is welcome to attend.
  4. If the program utilizes more than one hospital, then a faculty member from each major teaching location listed on the updated ACGME WebADS should attend. 
  5. Extremely small programs that use only one location could have only a Program Director representing the program. 
  6. At least one, peer-elected resident from each level of training must attend. 

 CONDUCT OF THE INTERNAL REVIEW

  1. Members of the Internal Review Panel and representatives from the program will receive identical information before the review. 
  2. The data sources include copies of:
    1. The most recent ACGME accreditation letter of report, progress reports sent to the RC and correspondence concerning the program received from the RC,
    2. Previous internal reviews, recommendations and follow-up if the Chair deems necessary,
    3. Current Program Requirements, Common Program Requirements and ACGME Institutional Requirements in effect at the time of the review,
    4. Written report of all annual program evaluations, and resultant action plans, conducted since the last ACGME program survey,
    5. Results from resident surveys. 

Documentation from the Program Director

  1. The completed Internal Review Document to assist the GMEC Review Panel in conducting the review,
  2. A written curriculum that incorporates the teaching of the general competencies as specified in the specialty’s program requirements,
  3. Samples of all evaluation tools, 
  4. Evidence that the program has developed and used dependable outcome measures to assess resident performance in the competencies,
  5. Evidence that the program is effective in linking educational outcomes with program improvement (may be part of the annual program evaluation).
  6. Program Letters of Agreement (PLA), and Memoranda of Understanding (MOU),
  7. Conference schedule with documentation of resident and faculty attendance.
  8. Written program policies/resident manual.
  9. Sample of the duty hour monitoring process, and most recent duty hour summaries.  The information will include but will not be limited to
    1. The ACGME resident survey,
    2. A duty hour report from the database,
    3. Ongoing monitoring systems and
    4. Education about chronic fatigue,
    5. The members of the review panel will
      1. Document whether the Program Director has addressed each citation,
      2. Document whether required follow-up responses to the ACGME are up to date,
      3. Review policies, evaluation processes and PLA’s and MOU’s,
      4. Document that the program has written criteria and processes in compliance with institutional and RRC requirements,
      5. Discuss the ACGME survey,
      6. Document duty hour compliance and review the monitoring process,
      7. Document evidence of didactic and clinical curricula with goals and objectives linked to the core competencies,
      8. Document the use of assessments and outcome measures to evaluate residents’ attainment of the competencies,
      9. Evaluate the status of the program’s link between educational outcomes with program improvement,
      10. Discuss issues with the residents/fellows and
      11. Develop a report for the GMEC’s consideration.
      12. Faculty members representing the program will be excused, and the panel members will discuss the current condition of the program with the residents.
      13. The members of the panel will remain after residents leave to discuss the findings of the review and to develop a report that will be sent to the GMEC.
      14. The Chair of the Review Panel will write the report to the GMEC.  This report will include the name of the program, date of the review, names and titles of internal review panel members, description of the process, list of interviewees and reviewed materials, sources of documentation and discussion related to program compliance with ACGME requirements, resolution of past concerns from site visits.  The GMEC will review the report and add the date presented to the GMEC and final actions and recommendations of the GMEC.  The Department Chair and the Program Director will receive copies of the final report.

 

INTERNAL REVIEW DOCUMENT FORM 

When completing this form, please complete your answers in bold immediately after each question.

BASIC INFORMATION

  1. Name of program:
  2. Departmental Chair:
  3. Program Director :
  4. List the faculty members/titles and residents/level of training who will attend the review.  At least one resident from each level of training must attend the Internal Review.
  5. Have there been administrative changes in the program or in the department since the last site visit?
  6. List hospitals and training sites currently used and changes since last site visit.  Explain differences between your list and the list available on WebADS.
  7. List the current number of residents by level of training and compare this list with the ACGME approved number.  Explain differences.

CITATIONS, AREAS OF CONCERN AND RESPONSES

  1. List your citations from the last ACGME site visit.
  2. Report your efforts to address each citation in bold after stating the citation

EVALUATIONS (Attachment A: provide each type of evaluation form)

  1. How often do residents evaluate the faculty?  Are these evaluations anonymous?
  2. How often do residents evaluate the program?  Are these evaluations anonymous?
  3. Resident Evaluates Rotation (Required if stated in Program Requirements)?
  4. How often do faculty members evaluate residents?
  5. How often do faculty members evaluate program?
  6. How often does the Program Director formally meet with residents to provide feedback and counseling?
  7. Is there a 360-evaluation?
  8. Is there a final, written evaluation of each resident in the resident’s file? 
  9. Is there a 1-year and 5-year follow-up system in place?

 SCHOLARSHIP (Attachment B: list of departmental publications for the last year and underline your residents/fellows names)

  1. Do you have a committee that includes faculty members and residents/fellows who review the program including at least resident/fellow issues, goals and objectives, assessments, metrics, program improvement, policies and evaluations? 
  2. Does this committee meet at least annually? 
  3. When did this committee last meet to discuss the current program format? 
  4. Did you take minutes and record attendance for this meeting?   
  5. State your residents’ pass rate on board exams. 
  6. What kinds of practices do your residents enter?
  7. Do your residents/fellows have access to IRB approved basic and clinical research and other scholarly activities?
  8. Do your residents/fellows have opportunities to present at regional and national meetings?
  9. Do your residents/fellows present at departmental conferences?
  10. 10.  Is there a mentoring system for their presentations? 

DIDACTICS (Attachment C: list of conferences, presenters, topics for the last year)

  1. Provide a statement concerning the program’s compliance with conference and didactic lecture schedules required by the ACGME. 
  2. Explain differences between RRC requirements and your list.
  3. Is attendance required for residents/fellows and faculty members? 

POLICIES (Attachment D: copy of each policy)

  1. Does the program have the following program specific policies? 
    1. Recruitment and selection,
    2. Promotion,
    3. Grievance and due process,
    4. Warning, probation and dismissal,
    5. Graded responsibility and supervisory lines of responsibility for patient care,
    6. Moonlighting and
    7. Duty hours 
  2. Are these policies individualized for your program?  Each policy must be on a different sheet of paper with the title of the policy, the name of the training program and the date of the last review of each policy at the top of each page. 

DUTY HOURS, PATIENT SAFETY, AND LEARINING ENVIRIONMENT (Attachment E: raw data from latest monitored period)

  1. How often does your program monitor duty hours?
  2. Are you in compliance with every duty hour guideline?
  3. If you are out of compliance, which guidelines are problematic for your program?
  4. What steps are you taking to bring the program into compliance? 
  5. Provide information regarding in-hospital hours worked when residents are on call from home.
  6. What is your backup system should residents become overwhelmed with clinical work?
  7. Indicate ways that you educate residents to recognize the signs of fatigue and sleep deprivations?
  8. What percentage of your residents participates in patient safety programs during an academic year?
  9. What percentage of your residents participates in interdisciplinary clinical quality improvement programs to improve health outcomes?
  10. What methods do you use to insure that hand-over processes facilitate both continuity of care and patient safety?

GOALS/OBJECTIVES AND ASSESSMENTS (Attachment F: two representative sets of goals and objectives for major rotations organized by rotation then by competency.  Assessments must be associated with each objective)

METRICS AND PROGRAM IMPROVEMENT (Attachment G: provide plan for program improvement)

  1. Present your plan for program improvement using metrics.
  2. Explain how you are linking the metrics to improvement initiatives. 

RESIDENT/FELLOW ACGME SURVEY

  1. Explain your initiatives to address each question the residents answered in a negative sense. 

ATTACHMENTS   

  1. Evaluation forms
  2. Publications (underline resident/fellow names)
  3. List of conferences, presenters, subjects (please define your year: academic, calendar)
  4. Policies
  5. Duty hour data
  6. Goals, objectives and assessment methods from two representative, major rotations
  7. Plan for program improvement
 
COMPETENCY DESCRIPTIONS
Practice-Based Learning and Improvement
  1. Describe one learning activity in which residents engage to identify strengths, deficiencies and limits in their knowledge and expertise; set learning and improvement goals; identify and perform appropriate learning activities to achieve self-identified goals.
  2. Describe one example of a learning activity in which residents engage to develop the skills needed to use information technology to locate, appraise and assimilate evidence from scientific studies and apply it to their patients’ health problems.  Organize the answer in the following sequence:           
    1. Locating information
    2. Using information technology
    3. Appraising information
    4. Assimilating evidence information from scientific studies
    5. Applying information to patient care
  3. Give one example and the outcome of a planned quality improvement activity or project in which at least one resident participated in the past year that required the resident to demonstrate an ability to analyze, improve and change practice or patient care.  Describe planning, implementation, evaluation and provision of faculty support and supervision that guided this process. 
     
  4. Describe how residents:
    1. Develop teaching skills necessary to educate patients, families, students and other residents;
    2. Teach patients, families and others;
    3. Receive and incorporate formative evaluation feedback into daily practice. 

Interpersonal and Communication Skills

  1. Describe one learning activity in which residents develop competence in communicating effectively with patients and families across a broad range of socioeconomic and cultural backgrounds, and with physicians, other health professionals and health related agencies.
  2. Describe one learning activity in which residents develop their skills and habits to work effectively as a member or leader of a health care team or other professional group.  In the example, identify the members of the team, responsibilities of the team members and how team members communicate to accomplish responsibilities.
  3. Explain (a) how the completion of comprehensive, timely and legible medical records is monitored and evaluated and (b) the mechanism for providing residents feedback on their ability to competently maintain medical records. 

Professionalism 

  1. Describe at least one learning activity, other than lecture, by which residents develop a commitment to carrying out professional responsibilities and an adherence to ethical principles. 
  2. How does the program promote professional behavior by residents and faculty members?
  3. How are the lapses in these behaviors addressed?

 Systems-Based Practice

  1. Describe the learning activities through which resident achieve competence in the elements of systems-bases practice: work effectively in various health care delivery settings and systems; coordinate patient care within the health care system; incorporate considerations of cost-containment and risk-benefit analysis in patient care; advocate for quality patient care and optimal patient care systems; and work in inter professional teams to enhance patient safety and are quality
  2. Describe an activity that fulfills the requirements for experiential learning in identifying system errors. 

REVIEW OF YOUR PROGRAM REQUIREMENTS

  1.  The questionnaire used by the site visitor will match your program’s requirements.  You should review your requirements and determine if you can answer “yes” to each “must” or “should” statement.  Please see the ACGME website and review the document Program Directors Guide to the Common Program Requirements.

LIST OF ASSESSMENT TECHNIQUES FOR USE IN THE NEXT TABLE

  1. Clinical performance ratings
  2. Evaluation committee
  3. Focused observation
  4. 360 evaluations
  5. Structured case discussions
  6. Stimulated chart recall
  7. Review of case or procedure log
  8. Review of patient chart
  9. Standardized patient
  10. OSCE
  11. High tech simulators
  12. Role play or simulations
  13. Formal oral exam
  14. In-training exam
  15. In-house exams (pre- and post-tests)
  16. Multimedia exams
  17. Resident project reports (portfolio)
  18. Resident experience narrative (portfolio)
  19. Other portfolio
  20. Audits of drug prescribing and patient outcome
  21. Other (define)

 LIST OF POTENTIAL EVALUATORS FOR USE IN THE NEXT TABLE

  1. Patients
  2. Family members
  3. Faculty supervisors
  4. Faculty members
  5. Chief residents
  6. Junior or senior residents
  7. Medical students
  8. Program Director
  9. Chair
  10. Clerical staff
  11. Others as you see fit

COMPETENCY

ASSESSMENT METHOD

EVALUATOR(S)

Patient Care

Patient Care

Patient Care

Patient Care

Patient Care

Medical Knowledge

Medical Knowledge

Medical Knowledge

Medical Knowledge

Medical Knowledge

Practice Based Learning and Improvement

Practice Based Learning and Improvement

Practice Based Learning and Improvement

Practice Based Learning and Improvement

Practice Based Learning and Improvement

Interpersonal and Communication Skills

Interpersonal and Communication Skills

Interpersonal and Communication Skills

Interpersonal and Communication Skills

Interpersonal and Communication Skills

Professionalism

Professionalism

Professionalism

Professionalism

Professionalism

Systems Based Practice

Systems Based Practice

Systems Based Practice

Systems Based Practice

Systems Based Practice

Send the completed document and attachments to the GME Office at least 2 weeks in advance of the date of the review.