Compass Assessment Software

Project Results

Compass Version 1.1 is Here!

We have just released Compass 1.1, which includes support for Windows Vista, an improved Switch test, French test materials, and more. You can download Compass here, and it is also available through various resellers. Visit the Products page for more information about how to purchase Compass.

Compass Version 1.0 is Complete

After addressing the comments of the beta testers, and performing extensive quality testing, the first commercial version of Compass was released in 2006.

Summary of Beta Test Results

Methods

Twelve active clinicians in the area of computer access assessment participated in the Compass beta test during April 2005. Testers were mailed a packet containing: the Compass software, a written set of tasks to perform, feedback forms, survey questions, and a self-addressed stamped envelope for returning the responses. Testers were given one month to complete the beta test. They performed 10 scenario-based tasks with Compass, and were asked to comment on each task after completion. This included a try-to-break-it task, where testers were encouraged to try a variety of AT devices and settings within the software. No specific instructions on how to do the tasks were provided.

Results

No tester experienced any problems installing or running Compass. No program crashes or severe errors were reported. All testers completed all ten tasks independently and successfully. Approximately 50 suggestions for changes or new features were generated, and most of these have already been addressed.

The overall usability rating across averaged 4.6 on a scale of 1 – 5. The 95% confidence interval was [4.4, 4.7]. The overall usability rating was based on responses to 10 survey questions regarding ease of learning, efficiency of use, etc.

Examples of Tester Comments

“I WILL USE this software to assess my students' alternative computer input needs. It is EXACTLY what I need. Thank you!”

“I am very impressed with the software. I like the ease of configuring the tests, copying them for multiple trials, and the summary of results. Very thorough data, yet not overwhelming. The program is laid out in a logical way and navigation is easy with keyboard or mouse. Client files are easy to save and retrieve. The Help system is excellent, particularly the Screen Tips. I like the Menu and Drag tests, and the available customization. Good quantitative data that's easy to interpret. Great job!”

“Easy to use/follow/understand… Piece of cake. You could run this without ever reading the instructions… Simple to use, no problems… easy to follow …Self-explanatory - buttons and directions are clear.”

“LOVED the help. Accessed it many times and only once was not able to immediately find exact info sought.”

“I like the detailed information in the reports! For example, during one student assessment, we were comparing several different on-screen keyboards and I found the detailed data very helpful!”

“This system makes it easier for OT's and teachers or anyone else working with students to pass this information on to receiving educational environments… I will recommend [Compass] to our OT department.”

Summary of Pre-beta Usability Studies

Development of Compass has been guided by extensive user-centered research activities. System requirements were defined and refined through a survey of 103 clinicians and in-depth discussions with 25 more. In two formal usability trials and one alpha test trial, a total of 24 clinicians were able to perform a series of representative Compass activities with no external help and without major errors. The overall usability rating across these three studies averaged 4.2 on a scale of 1 – 5. 95% confidence interval was [4.0, 4.4]. The overall usability rating was based on responses to 10 survey questions regarding ease of learning, efficiency of use, etc. Our goal now is to complete the software to a Beta test level, with plans for the first commercial release in the 4th quarter 2004.

Prototype Alpha Test Complete

In February 2004, alpha versions of Compass were sent to 20 clinicians across the U.S. We received very positive feedback, from both a usability and technical standpoint.

Initial Usability Testing Complete

Usability testing of the Visual Basic system prototype was completed in August 2002. Results overall were excellent, and all usability targets were met. A presentation on the results will be given at the RESNA 2003 conference, and a paper describing the results is available.

Visual Basic Prototype Complete

An initial prototype was developed for illustration purposes in Visual Basic. This allowed us to mock up and test the user interface relatively easily, without the extra development time required to build software in Java.

User Survey Results -- March 2002

Potential users were asked to rate the importance of some tests that may be included in Compass and some clinician parameters that are being considered. Combined with direct discussions with potential Compass users, these survey results formed the basis for defining the requirements of the Compass software.

There were 91 total respondents with the largest groups being represented by 31 occupational therapist, 11 special ed consultants, 8 rehab engineers, 7 AT specialists, and 7 special ed teachers. Most of the respondents work in multiple settings. 62 people work in schools, 24 in the community, 19 in in-patient settings and 15 in out-patient facilities. Most of you do fewer than 25 evaluations a year and have fewer than 50 clients per year.

The test areas that were rated as most important were the single word keyboarding test, pressing a switch and scanning in an array, clicking, and aiming, double clicking within the pointer use tests. Testing the user’s ability to use menus and perform standard functions such as managing windows and moving icons were high in the application skills group. Vision tests for acuity, discrimination, tracking, and contrast were also important features. A test to determine the potential to use speech recognition was also rated highly.

When it comes to configuring tests, the ability to control the font size and color, object size and color were important parameters. The number of trials per test, the ability to have visual or auditory feedback, and adjusting the grade level for tests also came in high.

Important reporting outputs are to have both detailed and summary reports, and you want to be able to control the format. Most people want Compass to offer some guidance as to types of systems and accommodations based on the test results.

Phase I Results

During Phase I of the project, we developed and evaluated an initial prototype of software for measuring the keyboarding and pointing device skills of people with disabilities. The following goals were achieved:

Specify system requirements, design user interface and modular architecture to support a battery of skill tests.
Develop a prototype which implements modules for text entry and pointing skills assessment.
Evaluate the prototype in educational and clinical settings for technical feasibility, usability, and accuracy.
Refine the prototype design based on iterative feedback during the evaluation process.

Regarding clinical feasibility, survey results clearly indicated a high degree of interest in the use of software as a tool for computer access assessments. This confirmed our original expectations and motivation for developing the initial prototype. Perhaps more importantly, clinicians liked our specific approach to developing such a tool. While opportunities for improvement were certainly identified, the overwhelming response was that Compass would be a highly valuable tool for their assessment work. Further, objective measures of learning and evaluation time indicated that Compass is sufficiently easy and fast to use in practical clinical situations where time is at a premium.

Papers resulting from Phase I work are listed on this web site (Koester and McMillan, 1997; Koester and McMillan, 1998). [News & Publications]