Online Guide to EBM: 
Solving a clinical problem using a Clinical Prediction "rule"
Notes on using the lecture

1) What is a clinical prediction "rule"?
2) Why should I use one?
3) Ask The Question
4) Finding the Evidence: choosing a source
5) Finding the Evidence: constructing a search strategy (OVID)
6) Reading the Study: Is it Valid?
7) What are the Results?
8) Will the results help me care for my patient?
9) Will the results change my management?
case scenario: follow along with a sample case
 

 
1)What is a Clinical Prediction Tool (CPT)?

A clinical prediction tool has been defined as 3 or more variables from the history, physical examination or simple ancillary tests1. When input into a prediction model, these variables yield an estimate of the probability of disease presence, or of a specific event occurring. Clinical Prediction models take several forms.
 
2) Why use a clinical prediction tool?

Many of the clinical decisions we make are based upon dogma passed down through generations of house officers.  Few are based on validated evidence.  This does not imply that most decisions are wrong.  However, utilizing medical dogma (all patients with new onset CHF must be admitted), gestalt, and personal observation is sub-optimal.  Clinical prediction tools allow clinicians to make more precise estimates of the probability of disease presence, absence, or of a specific event occurring.
 
3) Ask the Question
follow along with an example case

This is similar to posing a question for a diagnostic test search, except the test we are evaluating is a clinical prediction tool.  The question has 4  parts: 
  • What is the disease or condition of interest
  • What is the test of interest?: a  clinical prediction rule
  • What is the comparison test (gold standard comparison), if any
  • What do we want to know about the test, e.g. what is the "test"  related outcome of interest?: how we want to measure the value of the CPT

    •  
    4)  Finding the Evidence: Choosing a source.
    Link to Primary Sources PageSearch OvidSearch Grateful MedSearch PubMedLink to Secondary Sources pageTurning Research Into Progress: The TRIP DatabaseGlobal Emergency Medicine Archives: GEMANational Guideline ClearingHouse

    As usual, the best yield is through one of the NLM databases.  Again, the TRIP database is a good place to start if you are short on time.  Several sites provide online, interactive, clinical prediction tools.  However, to critically evaluate these tools you must review the original report on the tool.Click here for more info.
     
    5)  Finding the evidence: Constructing a strategy (OVID)
    Searching 101MeSH BrowserOnline Statistical Text'sClinical Calculator linksfollow along with an example case

    If you have not already done so, this is a good point  to stop and read more about literature searching.

    Try the following:

    • Search for the test (search filter for CPT's).

      • There is no MeSH heading "clinical prediction" Enter the following in the OVID search box:
      exp logistic models/ or exp models, theoretical/ or exp probability/ or exp multivariate analysis or *decision support techniques
    • Search the disease of interest
    • Combine 1 AND 2
    • If you came up with too many citations go on and Search the outcome: Enter   exp "sensitivity and specificity"/
    • Combine 3 (product of 1 AND 2) and 4  (the outcome search)
    • See an example of this strategy Click here for more info.
      If you don't find what you want here are some additional search tips:Click here for more info.

     
    6) Reading the study: is it valid (based on standards of Laupacis et al1)
    Searching 101MeSH BrowserOnline Statistical Text'sClinical Calculator linksclick here to see an analysis of our example article

    The question of validity can be divided into 5 separate questions
    • Are the predictors valid? potential predictor variables  should be prospectively defined, clearly defined, and should be reliable.  During model development or validation, the presence or absence (or value of) the predictors under investigation should be prospectively recorded by individuals blinded to outcome data.
    • Is the decision tool outcome valid? The outcome which the decision model predicts should be clinically relevant, and  be clearly defined.  Persons determining the outcome should be blinded to other clinical information and predictor values.
    • Methods and reporting. The   mathematical techniques used to derive the rule should be reported.   The demographic characteristics of the population should be clearly reported. Without this information readers cannot assess the applicability of results to other populations (issues of spectrum bias).
    • Have the results been validated?  Many apparently well designed studies do not perform well in validation studies.  Prior to widespread implantation of a clinical prediction model, it is important to establish the models "transportability" to other settings. Several methods are used for validation, but a prospective validation in a different setting is most important. 
    • Did they include too many predictor variables in the final rule? The number of outcome events (fractures, MI's) in the derivation population should be at least 10 times the number of predictor variables included. Click here for more info.

     
    7) What were the result (how does the model perform) ?
    Searching 101MeSH BrowserOnline Statistical Text'sClinical Calculator linksclick here to see an analysis of our example article

    This can be divided into 4 separate issues
    • Categorization: How does the model categorize patients?  For the model to be of value it must divide patients into meaningful groups. 
    • Discrimination: How accurately does the tool categorize patients? This is often expressed as sensitivity, specificity, predictive values, likelihood ratios, and by ROC analysis.  If these values are not reported, try to calculate them. 
    • Calibration: How well does the model fit the data? Look for tabular or graphic data showing predicted vs actual results for the patient population under investigation. This is particularly important when the a tool uses continuos or multiple risk stratification's. Click here for more info.
    • Precision: Are confidence intervals on the results reported? If not, try to  Calculate   these. 
    8) Will the results help me care for my patient?
    click here to see an analysis of our example article
    • Is the tool easily usable? Some models, particularly those using regression equations in the final model, require complex calculations or a computer. Most models employing regression equations in the final form require a computer.
    • Do the results apply to my patient? Derivation and/or validation populations should be similar to yours.  Ask the question: would my patient have been eligible for entry into the study? 
    • Are the results meaningful? If the decision support tool doesn't  categorize patients into meaningful groups, or doesn't does this with adequate accuracy, then the model will not be of help.
    9) Will the  results change my management? 
    click here to see an analysis of our example article
    • Has the model been shown to improve outcome or decrease costs? Few models survive to outcome level investigations 
    • Does the model perform better than the alternatives: Although valid and accurate, if a model doesn't perform better than alternative tests, or physician judgment , it may be of little value for the experienced clinician. 
    • Based on the above considerations, weigh the risks and benefits of using the rule. This requires consideration of your own, and you patients, values.
          •  

    Back to the top