PART 462—MEASURING EDUCATIONAL GAIN IN THE NATIONAL REPORTING SYSTEM FOR ADULT EDUCATION Authority: 29 U.S.C. 3292, et seq., Source: 73 FR 2315, Jan. 14, 2008, unless otherwise noted. Subpart A—General § 462.1 What is the scope of this part? The regulations in this part establish the— (a) Procedures the Secretary uses to determine the suitability of standardized tests for use in the National Reporting System for Adult Education (NRS) to measure educational gain of participants in an adult education program required to report under the NRS; and (b) Procedures States and local eligible providers must follow when measuring educational gain for use in the NRS. (Authority: 29 U.S.C. 3292) [73 FR 2315, Jan. 14, 2008, as amended at 81 FR 55551, Aug. 19, 2016] § 462.2 What regulations apply? The following regulations apply to this part: (a) The Education Department General Administrative Regulations (EDGAR) as follows: (1) 34 CFR part 76 (State-Administered Programs). (2) 34 CFR part 77 (Definitions that Apply to Department Regulations). (3) 34 CFR part 79 (Intergovernmental Review of Department of Education Programs and Activities). (4) 34 CFR part 81 (General Education Provisions Act—Enforcement). (5) 34 CFR part 82 (New Restrictions on Lobbying). (6) 34 CFR part 84 (Governmentwide Requirements for Drug-Free Workplace (Financial Assistance)). (7) 34 CFR part 86 (Drug and Alcohol Abuse Prevention). (8) 34 CFR part 97 (Protection of Human Subjects). (9) 34 CFR part 98 (Student Rights in Research, Experimental Programs, and Testing). (10) 34 CFR part 99 (Family Educational Rights and Privacy). (b) The regulations in this part 462. (c)(1) 2 CFR part 180 (OMB Guidelines to Agencies on Governmentwide Debarment and Suspension (Nonprocurement)), as adopted at 2 CFR part 3485; and (2) 2 CFR part 200 (Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards), as adopted at 2 CFR part 3474. (Authority: 29 U.S.C. 3292) [81 FR 55551, Aug. 19, 2016] § 462.3 What definitions apply? (a) Definitions in the Adult Education and Family Literacy Act (Act). Adult education, Eligible provider, Individual of limited English proficiency, Individual with a disability, Literacy. (b) Other definitions. Adult basic education (ABE) Adult education population (1) Who have attained 16 years of age; (2) Who are not enrolled or required to be enrolled in secondary school under State law; and (3) Who— (i) Are basic skills deficient; (ii) Do not have a secondary school diploma or its recognized equivalent, and have not achieved an equivalent level of education; or (iii) Are English language learners. Adult secondary education (ASE) Content domains, content specifications, or NRS skill areas Educational functioning levels English as a Second Language (ESL) Guidelines Implementation Guidelines: Measures and Methods for the National Reporting System for Adult Education www.nrsweb.org. Local eligible provider State Test Test administrator Test publisher (Authority: 29 U.S.C. 3292, et seq., [73 FR 2315, Jan. 14, 2008, as amended at 81 FR 55551, Aug. 19, 2016] § 462.4 What are the transition rules for using tests to measure educational gain for the National Reporting System for Adult Education (NRS)? A State or an eligible provider may continue to measure educational gain for the NRS using tests that the Secretary has identified in the most recent notice published in the Federal Register Federal Register (Authority: 29 U.S.C. 3292) [81 FR 55551, Aug. 19, 2016] Subpart B—What Process Does the Secretary Use To Review the Suitability of Tests for Use in the NRS? § 462.10 How does the Secretary review tests? (a) The Secretary only reviews tests under this part that are submitted by a test publisher. (b) A test publisher that wishes to have the suitability of its test determined by the Secretary under this part must submit an application to the Secretary, in the manner the Secretary may prescribe, by October 1, 2016, April 1, 2017, October 1, 2017, April 1, 2018, October 1, 2018, and by October 1 of each year thereafter. (Authority: 29 U.S.C. 3292) [73 FR 2315, Jan. 14, 2008, as amended at 81 FR 55552, Aug. 19, 2016] § 462.11 What must an application contain? (a) Application content and format. (1) Include with its application information listed in paragraphs (b) through (i) of this section, and, if applicable, the information listed in paragraph (j) of this section; (2) Provide evidence that it holds a registered copyright of a test or is licensed by the copyright holder to sell or distribute a test. (3)(i) Arrange the information in its application in the order it is presented in paragraphs (b) through (j) of this section; or (ii) Include a table of contents in its application that identifies the location of the information required in paragraphs (b) through (j) of this section. (4) Submit to the Secretary four copies of its application. (b) General information. (2) The name, address, e-mail address, and telephone and fax numbers of a contact person to whom the Secretary may address inquiries. (3) A summary of the precise editions, forms, levels, and, if applicable, sub-tests and abbreviated tests that the test publisher is requesting that the Secretary review and determine to be suitable for use in the NRS. (c) Development. (1) The nature of samples of examinees administered the test during pilot or field testing, such as— (i) The number of examinees administered each item; (ii) How similar the sample or samples of examinees used to develop and evaluate the test were to the adult education population of interest to the NRS; and (iii) The steps, if any, taken to ensure that the examinees were motivated while responding to the test; and (2) The steps taken to ensure the quality of test items or tasks, such as— (i) The extent to which items or tasks on the test were reviewed for fairness and sensitivity; and (ii) The extent to which items or tasks on the test were screened for the adequacy of their psychometric properties. (3) The procedures used to assign items to— (i) Forms, for tests that are constructed prior to being administered to examinees; or (ii) Examinees, for adaptive tests in which items are selected in real time. (d) Maintenance. (1) How frequently, if ever, new forms of the test are developed; (2) The steps taken to ensure the comparability of scores across forms of the test; (3) The steps taken to maintain the security of the test; (4) A history of the test's use, including the number of times the test has been administered; and (5) For a computerized adaptive test, the procedures used to— (i) Select subsets of items for administration; (ii) Determine the starting point and termination conditions; (iii) Score the test; and (iv) Control for item exposure. (e) Match of content to the NRS educational functioning levels (content validity). (1) Whether the items or tasks on the test require the types and levels of skills used to describe the NRS educational functioning levels; (2) Whether the items or tasks measure skills that are not associated with the NRS educational functioning levels; (3) Whether aspects of a particular NRS educational functioning level are not covered by any of the items or tasks; (4) The procedures used to establish the content validity of the test; (5) The number of subject-matter experts who provided judgments linking the items or tasks to the NRS educational functioning levels and their qualifications for doing so, particularly their familiarity with adult education and the NRS educational functioning levels; and (6) The extent to which the judgments of the subject matter experts agree. (f) Match of scores to NRS educational functioning levels. (1) The standard-setting procedures used to establish cut scores for transforming raw or scale scores on the test into estimates of an examinee's NRS educational functioning level; (2) If judgment-based procedures were used— (i) The number of subject-matter experts who provided judgments, and their qualifications; and (ii) Evidence of the extent to which the judgments of subject-matter experts agree; (3) The standard error of each cut score, and how it was established; and (4) The extent to which the cut scores might be expected to differ if they had been established by a different (though similar) panel of experts. (g) Reliability. (1) The correlation between raw (or scale) scores across alternate forms of the test or, in the case of computerized adaptive tests, across alternate administrations of the test; (2) The consistency with which examinees are classified into the same NRS educational functioning levels across forms of the test. Information regarding classification consistency should be reported for each NRS educational functioning level that the test is being considered for use in measuring; (3) The adequacy of the research design leading to the estimates of the reliability of the test, including— (i) The size of the sample(s); (ii) The similarity between the sample(s) used in the data collection and the adult education population; and (iii) The steps taken to ensure the motivation of the examinees; and (4) Any other information explaining the methodology and procedures used to measure the reliability of the test. (h) Construct validity. i.e. (1) The extent to which the raw or scale scores and the educational functioning classifications associated with the test correlate (or agree) with scores or classifications associated with other tests designed or intended to assess educational gain in the same adult education population as the NRS; (2) The extent to which the raw or scale scores are related to other relevant variables, such as teacher evaluation, hours of instruction, or other measures that may be related to test performance; (3) The adequacy of the research designs associated with these sources of evidence ( see (4) Other evidence demonstrating that the test measures gains in educational functioning resulting from adult education and not from other construct-irrelevant variables, such as practice effects. (i) Other information. (2) Additional guidance on the interpretation of scores resulting from any modifications of the tests for an individual with a disability. (3) The manual provided to test administrators containing procedures and instructions for test security and administration. (4) A description of the training or certification required of test administrators and scorers by the test publisher. (5) A description of retesting (e.g., re-administration of a test because of problems in the original administration such as the test taker becomes ill during the test and cannot finish, there are external interruptions during testing, or there are administration errors) procedures and the analysis upon which the criteria for retesting are based. (6) Such other evidence as the Secretary may determine is necessary to establish the test's compliance with the criteria and requirements the Secretary uses to determine the suitability of tests as provided in § 462.13. (j) Previous tests. (2) For a test first published five years or more before the date it is submitted to the Secretary for review under this part, the test publisher must provide documentation of periodic review of the content and specifications of the test to ensure that the test continues to reflect NRS educational functioning levels. (3) For a test that has not changed in the seven years since the Secretary determined, under § 462.13, that it was suitable for use in the NRS that is again being submitted to the Secretary for review under this part, the test publisher must provide updated data supporting the validity of the test for use in classifying adult learners with respect to the NRS educational functioning levels and the measurement of educational gain as defined in § 462.43 of this part. (4) If a test has been substantially revised—for example by changing its mode of administration, administration procedures, structure, number of items, content specifications, item types, forms, sub-tests, or number of hours between pre- and post-testing from the most recent edition reviewed by the Secretary under this part—the test publisher must provide an analysis of the revisions, including the reasons for the revisions, the implications of the revisions for the comparability of scores on the current test to scores on the previous test, and results from validity, reliability, and equating or standard-setting studies undertaken subsequent to the revisions. (Authority: 29 U.S.C. 3292) [73 FR 2315, Jan. 14, 2008, as amended at 81 FR 55552, Aug. 19, 2016] § 462.12 What procedures does the Secretary use to review the suitability of tests? (a) Review. (2) The Secretary reviews and determines the suitability of a test only if an application— (i) Is submitted by a test publisher; (ii) Meets the deadline established by the Secretary; (iii) Includes a test that— (A) Has two or more secure, parallel, equated forms of the same test—either traditional paper and pencil or computer-administered instruments—for which forms are constructed prior to administration to examinees; or (B) Is an adaptive test that uses computerized algorithms for selecting and administering items in real time; however, for such an instrument, the size of the item pool and the method of item selection must ensure negligible overlap in items across pre- and post-testing; (iv) Includes a test that samples one or more of the major content domains of the NRS educational functioning levels of ABE, ASE or ESL with sufficient numbers of questions to represent adequately the domain or domains; and (v) Includes the information prescribed by the Secretary, including the information in § 462.11 of this part. (b) Secretary's determination. (2) For tests that contain multiple sub-tests measuring content domains other than those of the NRS educational functioning levels, the Secretary determines the suitability of only those sub-tests covering the domains of the NRS educational functioning levels. (c) Suitable tests. (1) Notifies the test publisher of the Secretary's decision; and (2) Annually publishes in the Federal Register www.nrsweb.org (d) Unsuitable tests. (2) The test publisher may resubmit an application to have the suitability of its test determined by the Secretary under this part on October 1 in the year immediately following the year in which the Secretary notifies the publisher. (i) An analysis of why the information and documentation submitted meet the criteria and requirements in § 462.13, notwithstanding the Secretary's earlier decision to the contrary; and (ii) Any additional documentation and information that address the Secretary's reasons for determining that the test was unsuitable. (3) The Secretary reviews the additional information submitted by the test publisher and makes a final determination regarding the suitability of the test for use in the NRS. (i) If the Secretary's decision is unchanged and the test remains unsuitable for use in the NRS, the Secretary notifies the test publisher, and this action concludes the review process. (ii) If the Secretary's decision changes and the test is determined to be suitable for use in the NRS, the Secretary follows the procedures in paragraph (c) of this section. (e) Revocation. (i) The information the publisher submitted as a basis for the Secretary's review of the test was inaccurate; or (ii) A test has been substantially revised—for example, by changing its mode of administration, administration procedures, structure, number of items, content specifications, item types, forms or sub-tests, or number of hours between pre- and post-testing. (2) The Secretary notifies the test publisher of the— (i) Secretary's decision to revoke the determination that the test is suitable for use in the NRS; and (ii) Reasons for the Secretary's revocation. (3) Within 30 days after the Secretary notifies a test publisher of the decision to revoke a determination that a test is suitable for use in the NRS, the test publisher may request that the Secretary reconsider the decision. This request must be accompanied by documentation and information that address the Secretary's reasons for revoking the determination that the test is suitable for use in the NRS. (4) The Secretary reviews the information submitted by the test publisher and makes a final determination regarding the suitability of the test for use in the NRS. (5) If the Secretary revokes the determination regarding the suitability of a test, the Secretary publishes in the Federal Register www.nrsweb.org (Authority: 29 U.S.C. 3292) [73 FR 2315, Jan. 14, 2008, as amended at 81 FR 55552, Aug. 19, 2016] § 462.13 What criteria and requirements does the Secretary use for determining the suitability of tests? In order for the Secretary to consider a test suitable for use in the NRS, the test or the test publisher, if applicable, must meet the following criteria and requirements: (a) The test must measure the NRS educational functioning levels of members of the adult education population. (b) The test must sample one or more of the major content domains of the NRS educational functioning levels of ABE, ASE or ESL with sufficient numbers of questions to adequately represent the domain or domains. (c)(1) The test must meet all applicable and feasible standards for test construction and validity provided in the 1999 edition of the Standards for Educational and Psychological Testing, http://www.archives.gov/federal_register/code_of_federal_regulations/ibr_locations.html. (2) If requested by the Secretary, a test publisher must explain why it believes that certain standards in the 1999 edition of the Standards for Educational and Psychological Testing (d) The test must contain the publisher's guidelines for retesting, including time between test-taking, which are accompanied by appropriate justification. (e) The test must— (1) Have two or more secure, parallel, equated forms of the same test—either traditional paper and pencil or computer administered instruments—for which forms are constructed prior to administration to examinees; or (2) Be an adaptive test that uses computerized algorithms for selecting and administering items in real time; however, for such an instrument, the size of the item pool and the method of item selection must ensure negligible overlap in items across pre- and post-testing. Scores associated with these alternate administrations must be equivalent in meaning. (f) For a test that has been modified for individuals with disabilities, the test publisher must— (1) Provide documentation that it followed the guidelines provided in the Testing Individuals With Disabilities section of the 1999 edition of the Standards for Educational and Psychological Testing; (2) Provide documentation of the appropriateness and feasibility of the modifications relevant to test performance; and (3)(i) Recommend educational functioning levels based on the information obtained from adult education students who participated in the pilot or field test and who have the disability for which the test has been modified; and (ii) Provide documentation of the adequacy of the procedures used to translate the performance of adult education students with the disability for whom the test has been modified to an estimate of the examinees' standing with respect to the NRS educational functioning levels. (Authority: 29 U.S.C. 3292) [73 FR 2315, Jan. 14, 2008, as amended at 81 FR 55552, Aug. 19, 2016] § 462.14 How often and under what circumstances must a test be reviewed by the Secretary? (a) The Secretary's determination that a test is suitable for use in the NRS is in effect for a period of seven years from the date of the Secretary's written notification to the test publisher, unless otherwise indicated by the Secretary. After that time, if the test publisher wants the test to be used in the NRS, the test must be reviewed again by the Secretary so that the Secretary can determine whether the test continues to be suitable for use in the NRS. (b) If a test that the Secretary has determined is suitable for use in the NRS is substantially revised—for example, by changing its mode of administration, administration procedures, structure, number of items, content specifications, item types, forms, sub-tests, or number of hours between pre- and post-testing—and the test publisher wants the test to continue to be used in the NRS, the test publisher must submit, as provided in § 462.11(j)(4), the substantially revised test or version of the test to the Secretary for review so that the Secretary can determine whether the test continues to be suitable for use in the NRS. (Authority: 29 U.S.C. 3292) [73 FR 2315, Jan. 14, 2008, as amended at 81 FR 55552, Aug. 19, 2016] Subpart C [Reserved] Subpart D—What Requirements Must States and Local Eligible Providers Follow When Measuring Educational Gain? § 462.40 Must a State have an assessment policy? (a) A State must have a written assessment policy that its local eligible providers must follow in measuring educational gain and reporting data in the NRS. (b) A State must submit its assessment policy to the Secretary for review and approval at the time it submits its annual statistical report for the NRS. (c) The State's assessment policy must— (1) Include a statement requiring that local eligible providers measure the educational gain of all students who receive 12 hours or more of instruction in the State's adult education program with a test that the Secretary has determined is suitable for use in the NRS; (2) Identify the pre- and post-tests that the State requires eligible providers to use to measure the educational functioning level gain of ABE, ASE, and ESL students; (3)(i) Indicate when, in calendar days or instructional hours, eligible providers must administer pre- and post-tests to students; (ii) Ensure that the time for administering the post-test is long enough after the pre-test to allow the test to measure educational functioning level gains according to the test publisher's guidelines; and (iii) Specify a standard for the percentage of students to be pre- and post-tested. (4) Specify the score ranges tied to educational functioning levels for placement and for reporting gains for accountability; (5) Identify the skill areas the State intends to require local eligible providers to assess in order to measure educational gain; (6) Include the guidance the State provides to local eligible providers on testing and placement of an individual with a disability or an individual who is unable to be tested because of a disability; (7) Describe the training requirements that staff must meet in order to be qualified to administer and score each test selected by the State to measure the educational gains of students; (8) Identify the alternate form or forms of each test that local eligible providers must use for post-testing; (9) Indicate whether local eligible providers must use a locator test for guidance on identifying the appropriate pre-test; (10) Describe the State's policy for the initial placement of a student at each NRS educational functioning level using test scores; (11) Describe the State's policy for using the post-test for measuring educational gain and for advancing students across educational functioning levels; (12) Describe the pre-service and in-service staff training that the State or local eligible providers will provide, including training— (i) For staff who either administer or score each of the tests used to measure educational gain; (ii) For teachers and other local staff involved in gathering, analyzing, compiling, and reporting data for the NRS; and (iii) That includes the following topics: (A) NRS policy, accountability policies, and the data collection process. (B) Definitions of measures. (C) Conducting assessments; and (13) Identify the State or local agency responsible for providing pre- and in-service training. (Approved by the Office of Management and Budget under control number 1830-0027) (Authority: 29 U.S.C. 3292) [73 FR 2315, Jan. 14, 2008, as amended at 81 FR 55552, Aug. 19, 2016] § 462.41 How must tests be administered in order to accurately measure educational gain? (a) General. (b) Pre-test. (1) Administer a pre-test to measure a student's educational functioning level at intake, or as soon as possible thereafter; (2) Administer the pre-test to students at a uniform time, according to the State's assessment policy; and (3) Administer pre-tests to students in the skill areas identified in the State's assessment policy. (c) Post-test. (1) Administer a post-test to measure a student's educational functioning level after a set time period or number of instructional hours; (2) Administer the post-test to students at a uniform time, according to the State's assessment policy; (3)(i) Administer post-tests with a secure, parallel, equated form of the same test—either traditional paper and pencil or computer-administered instruments—for which forms are constructed prior to administration to examinees to pre-test and determine the initial placement of students; or (ii) Administer post-tests with an adaptive test that uses computerized algorithms for selecting and administering items in real time; however, for such an instrument, the size of the item pool and the method of item selection must ensure negligible overlap in items across pre- and post-testing; and (4) Administer post-tests to students in the same skill areas as the pre-test. (d) Other requirements. (2) A local eligible provider may use the results of a test in the NRS only if the test was administered in a manner that is consistent with the State's assessment policy and the test publisher's guidelines. (Approved by the Office of Management and Budget under control number 1830-0027) (Authority: 29 U.S.C. 3292) [73 FR 2315, Jan. 14, 2008, as amended at 81 FR 55553, Aug. 19, 2016] § 462.42 How are tests used to place students at an NRS educational functioning level? (a) A local eligible provider must use the results of the pre-test described in § 462.41(b) to initially place students at the appropriate NRS educational functioning level. (b) A local eligible provider must use the results of the post-test described in § 462.41(c)— (1) To determine whether students have completed one or more educational functioning levels or are progressing within the same level; and (2) To place students at the appropriate NRS educational functioning level. (c)(1) States and local eligible providers are not required to use all of the skill areas described in the NRS educational functioning levels to place students. (2) States and local eligible providers must test and report on the skill areas most relevant to the students' needs and to the programs' curriculum. (d)(1) If a State's assessment policy requires a local eligible provider to test a student in multiple skill areas and the student will receive instruction in all of the skill areas, the local eligible provider must place the student in an educational functioning level that is equivalent to the student's lowest test score for any of the skill areas tested under § 462.41(b) and (c). (2) If a State's assessment policy requires a local eligible provider to test a student in multiple skill areas, but the student will receive instruction in fewer than all of the skill areas, the local eligible provider must place the student in an educational functioning level that is equivalent to the student's lowest test score for any of the skill areas— (i) Tested under § 462.41(b) and (c); and (ii) In which the student will receive instruction. (Approved by the Office of Management and Budget under control number 1830-0027) (Authority: 29 U.S.C. 3292) [73 FR 2315, Jan. 14, 2008, as amended at 81 FR 55553, Aug. 19, 2016] §§ 462.43-462.44 [Reserved]