|
|
|
In order to provide high quality and cost-effective services to its clients, MI is divided into four main functional service units:
A fifth functional service unit, Financial and Administrative Services, is concerned primarily with internal corporate operations.
In order to ensure maximum efficiency, staff members participate in tasks across units when necessary. This organizational flexibility allows MI to coordinate project activities more effectively by allocating human resources to the areas in which they will be most productive.
Research and
Product Development
Over the past twenty years, Measurement Incorporated has earned a
solid reputation for quality test and item development and associated
services. All the tests we develop are custom designed specifically
to meet the needs of our clients. We have no off-the-shelf test
program to promote, and our project managers have complete freedom to
deliver exactly what the client wants. To date, we have developed
tests in a variety of subjects from primary grades through high
school and for several private organizations and certification
agencies. Our primary clients have been state departments of
education, including the following states:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
We have built long-term relationships with many of these state clients, completing numerous projects or serving on a single project for many years. For example, we have conducted eight test development projects with the Florida Department of Education since 1980; we have been the contractor for the development of the Georgia High School Graduation Tests since 1990; and we have been the developer of New Jersey's eighth- and eleventh-grade statewide assessments since 1993.
While the needs of each client vary, we have delivered a wide array of services, including the following:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
We work particularly well with groups of teachers and other client staff, listening carefully to stakeholders and making sure their needs are addressed in the final product.
MI has been a leader in the development of "authentic assessment" instruments since before the name was applied. We have developed multiple-choice, short-answer, and extended-response test items, as well as performance assessments, oral assessments, rating forms, and observation checklists, as clients' needs dictated. We have pioneered the application of generalizability theory to the assessment of test reliability to account for multiple sources of score variability.
The R&D division of Measurement Incorporated operates with a staff of about 50 professionals including psychometricians, project directors, content specialists, editors, graphic artists, and support staff. While in-house staff create many of the test items themselves, we have also trained a pool of hundreds of test item writers who create test items for review by our content specialists and project directors. We maintain long-term consultant relationships with a select group of nationally respected experts in technical, legal, and other fields with direct bearing on assessment and evaluation issues.
Program Evaluation
Measurement Incorporated provides a
full range of program evaluation services. Typically, projects last
one to two years and involve consultation, planning and design,
instrument development, survey administration, on-site
investigations, data analysis, and reporting. Project subjects have
ranged from alternative education to vocational education programs.
Principal investigators are recognized in their fields for their
content expertise and their ability to focus the evaluation on
questions that can be answered and answers that can guide policy
makers to real-time decisions that affect learners.
Certification
Testing
The newest specialty area within MI's R&D department is
certification and credential examinations. Measurement Incorporated
is a sustaining member of the National Organization of Competency
Assurance (NOCA)
and is actively engaged in NOCA programs. Employing the same
expertise and psychometric rigor that we apply to other high-stakes
assessments, we work closely with clients to identify their testing
needs, conduct a thorough job analysis, translate the results of
those analyses into test blueprints and item specifications, create
test items and other forms of assessment, conduct pilot and field
tests, select items, construct and print tests, administer and score
the tests, and report results to candidates and reporting agencies.
Additional activities include standard setting, registration and
scheduling, program management, development of instructional
materials, and technical consultation. For test registration,
scheduling, and administration, we participate in a nationwide
network of test administration sites to augment our own twelve
locations in seven states.
Current work includes a certification examination for judiciary interpreters and translators. This project has required development of written and oral questions in English and Spanish, pre-recording of oral items by professional voice talents, and both large-group (written) and individual (oral) test administration. Every project is different, however, and we do not try to force any examination program into a preset design.
Writing and
Performance Assessment Scoring
One of the primary services provided by Measurement Incorporated is
the scoring of written essays and performance-based tests for state
DOEs, local education agencies, and other assessment units. The test
responses are typically evaluated using criteria established by the
client in consultation with educators, administrators, curriculum
supervisors, and measurement experts. MI has scored tests in a wide
range of subject areas (writing, reading, mathematics, science,
social studies, and others) and across all levels, kindergarten
through college and at the professional level. Most of our
handscoring activities take place in the eleven scoring
centers we
have established in seven states. We also have a great deal of
experience leading scoring efforts within the client's home state,
where we recruit local educators and train them to recognize varying
degrees of quality in student responses. Often these projects entail
the development of training materials which we use in training
leadership teams who then train hundreds of educators to score items
from thousands of students. We then continue to provide technical and
logistical support, supervision, and monitoring.
Whether acting as the prime contractor or as a subcontractor, MI handles all the procedures associated with handscoring. Our responsibilities involve cooperating with agency staff to plan the project, recruiting and hiring scoring personnel, training scoring leaders and scorers, coordinating shipping and receiving, handling the responses, providing and maintaining security, supervising the scoring sessions, and providing storage space for test materials. Our experience in these areas is unparalleled within the industry. As the nation's leading writing and performance assessment firm, we recognize the importance of these assessments both as indices of instructional effectiveness and as tools for instructional enhancement. We uphold the quality standards demanded by our clients through a combination of thorough scorer training, continuous scorer monitoring, and experienced supervisory personnel.
Reader Training and
Qualifying
Before being permitted to score any actual responses, all employees
involved in the scoring process, including scoring directors, team
leaders, and scorers, must participate in comprehensive training
sessions involving detailed discussion of the scoring rubric and
completion of a battery of training sets. Most clients also require
that all scorers complete a series of qualifying rounds in which the
scorers must correctly assign a certain percentage of scores in exact
agreement with the scores already established for these papers by the
testing body. Scorers who are unable to meet qualifying standards
after repeated attempts are not allowed to participate in the scoring
project. This provision ensures that all scorers have achieved a full
grasp of the scoring rubric and criteria before any actual responses
are scored.
Performance
Monitoring
To ensure consistent accuracy in scoring throughout the project,
supervisors conduct careful monitoring of scorer performance. They
identify scoring trends of individual scorers during the initial
training process and use various methods as necessary to track scorer
performance through the course of the project. MI routinely employs
computer-generated status reports which display the number of papers
scored, the agreement rate among scorers, the number and percentage
of papers requiring resolution readings, and score point
distribution. Daily and cumulative totals are provided for both
individual scorers and for the entire scoring group. Attentive
monitoring of these statistics enables supervisors to conduct
recalibration and retraining as necessary to ensure smooth project
operation and adherence to client specifications.
Project
Supervision
The key to the ultimate success of any handscoring project lies in
the knowledge, conscientiousness, and experience of the
project
staff. Our
supervisory personnel (project leaders, scoring directors, team
leaders, etc.) are exceptionally well-versed in the subtleties of
virtually all major scoring systems, including four- and six-point
holistic scales, multiple-category evaluation, domain-specific
assessment, primary and secondary trait scoring, and numerous
varieties of analytic scoring. They are also highly experienced in
dealing with unanticipated approaches and unusual responses to
prompts or test items and make every effort to ensure that all
responses are scored fairly and reliably. Our scorers all have at
least a four-year degree, usually in a language-related or social
science field, and many of them return to score additional projects,
giving them valuable experience with many different scoring systems
as well. Our combination of broad-based experience and
detail-oriented approach has helped to establish Measurement
Incorporated as the nation's leading writing assessment and
performance assessment firm, with a current annual volume of tests
from over four million students, many of whom remit multiple
responses.
Data
Processing and Score Reporting
Data processing and analysis and subsequent score reporting tasks are
required by nearly all of our clients. The intensive statistical
calculations entailed by these tasks are handled by our Information
Technology staff through the aid of in-house computer systems with
assistance from outside consultants and contractors when
necessary.
Systems and
Peripherals
MI's primary computing equipment is housed within our company
headquarters in Durham, North Carolina. We continue to upgrade
equipment and expand our expertise to meet growing requests for
sophisticated reporting and analysis from clients. A range of
IBM-compatible PC's interconnected through networks within the
organization allows us to process and analyze large databases and
provide refined graphics and reports. MI's equipment includes the
latest Intel Pentium technology in the PC line as well a number of
high output laser printers and state-of-the-art scanning equipment.
At our headquarters, three NCS Opscan 21 scanners rated between 7,000
and 10,000 sheets per hour provide bar code reading capability, which
facilitates entry of student identification information. Smaller
scanners are installed permanently or temporarily at some of our
other scoring centers. Through expanded telecommunications, MI is
able to transmit data to and from clients and transfer jobs from
remote mainframe locations through a PC platform to print on our
high-speed line printers.
Programming and
Analysis
MI conducts statistical research and analysis on an ongoing basis and
works to ensure that diagnostic instruments and assessment materials,
whether individual items or entire tests or batteries of tests, are
both psychometrically sound and legally defensible. Full-time staff
who are highly experienced in all aspects of psychometrics,
statistics, and computer programming conduct or supervise all of MI's
data analysis operations.
Analysis activities range from simple frequency distributions for a single small data set to multivariate analyses involving many large data sets. Included in this array of services are data analysis design, data management, descriptive statistics, inferential statistics, univariate analyses, multivariate analyses, tests of association, Rasch item calibration, and model building.
For many activities, MI programmers must develop original source code to meet unique client specifications. For others, data analysts employ various packaged programs such as SAS, BIGSTEPS, GENOVA, and SYSTAT. Particularly for large projects, it is not uncommon for MI to use a combination of both in-house and prepackaged programming tools to tailor a specialized and efficient overall data management and reporting system. Our in-house computing and programming resources allow us to exercise this cost-effective flexibility.
Score
Reporting
Measurement Incorporated routinely performs a variety of score
reporting services for its clients, including the production of class
rosters, individual score reports, reports by class, grade, school,
and district, summary reports, student record labels, and performance
data tapes. Many of our score reporting activities are handled
through our in-house computing equipment and high-speed linotronic
and laser printers. For other projects, MI electronically transmits
assessment data to other contractors for reporting. Additional
printing and publishing capabilities are provided by Measurement
Incorporated Printing Services. (See Printing
and Publishing,
below).
Printing
and Publishing
Measurement
Incorporated acquired a printing company in 1984. Now known as
Measurement Incorporated Printing Services, it serves MI's project
needs and performs outside commercial printing. Its staff has
combined experience of over 70 years in offset printing.
The Printing Services provides short turnaround for high volume photocopying with two Kodak 235 copy machines. Both copiers have sorting and finishing capabilities. Recently upgraded typesetting and graphic computer software have increased design capabilities for in-house and commercial printing requests. MI Printing Services also provides complete darkroom services, up to three color offset printing, binding, and finishing of all printed materials.