Reporting
      Back to home
      1. Administrator Knowledge Base
      2. Admin Application Guides
      3. Reporting

      Form Analysis Guide

      The Form Analysis Report provides a comprehensive statistical overview of exam forms, helping users evaluate the reliability, consistency, and difficulty of each form.

      Overview

      The main objective of form analysis is to obtain form-level statistics (e.g., mean total score, reliability) to help understand the properties of the test and monitor their effectiveness. This report cannot analyze randomized forms (forms with dynamic sections).

      The Form Analysis page automatically analyzes the exam response data. This involves four steps:

      1. Specify the test data used for item analysis
      2. Select the criteria to exclude invalid exam results
      3. View the form analysis results
      4. Apply any filter, choose the columns to show, and export the data

      Specify test data

      Specify the data you would like to use for form analysis through the drop-down lists at the top of the page. Click “RUN REPORT” to start analyzing the data.

      • Organization: The organization which owns the exam.
      • Exam: The name of the exam.
      • Forms (Assessments): The test form(s) of an exam. Check the box in front of one or more forms to select a single form or multiple forms under the same exam.
      • Data Range: the time range of administration. Select from one of the following:
      • Last Month: The previous calendar month.
      • Last Quarter: The previous three-month period. A year is typically divided into four quarters – Q1 (January to March), Q2 (April to June), Q3 (July to September), and Q4 (October to December).
      • Last Year: The previous calendar year.
      • Past 30 Days: The time period that spans the previous 30 consecutive days, including today.
      • Past 12 Months: The time period that spans the previous 12 consecutive months, including this month.
      • Other: Other specific time period. If “Other” was selected, you need to select the “Start Date” and “End Date” by typing in the date (in format mm/dd/yyyy) or selecting the date from the drop-down calendar.

      Tip: When multiple forms are selected in form analysis, each form will be analyzed separately. The form analysis results will display each form in a row.

      Exclude invalid exam results

      The system allows you to exclude invalid exam respondents from the analysis if the pattern of their responses indicate inattentiveness or low response quality. You can choose from one or more of the following three criteria:

      • Incomplete Data: Missing a lot of the responses may indicate an atypical respondent who should not be included in the analysis. Apply this criterion will remove any result that has more than or equal to 30% omit/not reached responses.
      • Quick Responders: Extremely short administration time usually indicates that the respondent was rushing through the exam and not being attentive. Applying this criterion will remove any result whose Administration Time was less than 15 minutes.
      • Chance Responders: Test total score lower than chance level indicates the respondent was not inattentive that their score was lower than simply selecting a random response for each item (assuming the exam consists of multiple-choice-single-selection items with four options, therefore, the chance level responding is defined as correctly answer 25% of the item for this exam). Applying this criterion will remove any result with less than 25% of the items answered correctly.

      Run the form analysis without any data cleaning, the number of respondents affected by each criterion will show up on the right of each criterion. Check the box in front of one or more data cleaning criteria to remove invalid exam response from the form analysis. Click “RUN REPORT” to see the results with specified criterion/criteria applied.

      Tips:

      • The default data cleaning criteria has proved to be effective for most of the exams. The system does not support customizing the data cleaning criteria at the moment. If you wish to apply another set of data cleaning criteria, please export the response data and conduct the data cleaning manually.
      • Cases are never deleted from the dataset. You can check/uncheck the boxes for data cleaning criteria and hit “RUN REPORT” and the data analysis is regenerated automatically.

      Form analyses

      Once the test data is selected and the data cleaning criteria applied (optional), the form analysis results will show on the bottom half of the page. The results include the following columns:

      • Exam Name: (hidden by default) The official title or name of the exam.
      • Exam Code: The unique identifier or code of the exam.
      • Exam Form Name: The name of the specific form.
      • Sample Size: The number of test-takers included in the analysis who took the form.
      • Mean Score: The average raw total score that was achieved by the sample included in the analysis. This is a measure of form difficulty where higher scores indicate easier forms.
      • Median Score: The middle raw total score when all the exam scores are arranged from lowest to highest; half the total scores are above the median, and half are below it.
      • Standard Deviation: The average distance between all raw test total scores and the average raw total score (i.e., Mean Score). This measures the dispersion of the test score distribution where higher values indicate more dispersed distribution of test scores. The system reports the unbiased estimation of standard deviation which is estimated in the sample to indicate the value for a population of test-takers.
      • Minimum Score: The lowest raw total score achieved by any test-taker included in the analysis sample.
      • Maximum Score: The highest raw total score achieved by any test-taker included in the analysis sample.
      • Cut Score Type: The type of cut score on the form. This can be one of the following:
      • Percent Score: The percentage of marks achieved out of the total marks on the form.
      • Raw Score: The number of marks achieved out of the total marks on the form.
      • Scaled Score: A scaled score is a score that has been mathematically transformed from a raw score for a variety of reasons, such as to account for variation in difficulty between different test versions or to align scores on a common scale to make them directly comparable.
      • Test Part Scoring: Tests with Test Part Scoring assigns a separate score for different parts or sections of a test.
      • Cut Score: The minimum score required to pass the exam. Test scores higher than or equal to the Cut Score are considered passing while those lower than the Cut Score failing.
      • Pass Rate: The percentage of test-takers included in the analysis who achieved at least the Cut Score and thus passed the exam.
      • CoefficientAlpha: Cronbach’s Alpha estimate of reliability, which should be higher than 0.70 for most exams ideally > 0.80). The calculation method uses Allen & Yen (1979; pp. 124-125) equation 6.6 to estimate alpha from item statsitics.
      • CoefficientAlphaN: (hidden by default) The sample size used to calculate the CoefficientAlpha. This should be equal to the Sample Size of the form.
      • SEM: This stands for standard error of measurement, a statistical measure of how far an observed score is expected to differ from the true score. For example, if a single test-taker were to take the test form repeatedly (assuming no new learning taking place between tests and no effects from remembering the questions from the previous test), the average score they achieve across tests would be their true score. Then the standard deviation of their repeated test scores would be the standard error of measurement (SEM). It is calculated as:

      SEM = Standard Deviation * sqrt(1 – CoefficientAlpha)

      Other functions

      Filter function:

      Enter a value in the “Filter” textbox below the header row to show form(s) with the specified values. Click on the filter button under each header row to set more complex filters using that column. To set a filter using multiple columns, click on “FILTERS” on the right edge of the page.

      Manage columns:

      To mange the columns, click on “COLUMNS” on the right edge of the page. Check the box in front of a column to make it appear and uncheck the box to hide it.

      Export Item and Option analyses results:

      To export the analysis results, click on the download button located at the top right corner of the page. The exported file will show exactly what you see on the page with visible columns only and any filter applied.

      Contact Us

      If you have any questions or need additional assistance, please contact us by either emailing support@certiverse.com or by submitting a ticket from this article.