.jpg){kind=logo}
This article is also available as a PDF file
Contents
Abstract
Introduction
Background
Technical Infrastructure
Security
Privacy and Legal Matters
Marketing
Issues with Multiple Choice Questions
Conclusions
References
Appendix A: Guidelines for Writing MCQs
Appendix B: Activity on the gasboys.net web-site from inception until January 2008
Appendix C: Geographic location of gasboys visitors June 2006 to January 2008
Abstract
This paper briefly explores some of the practical, technical, security, privacy, legal and pedagogical issues addressed during the implementation of gasboys (http://gasboys.net), an Internet website dedicated to anaesthetic education. It also critically appraises the site and discusses the problem of evaluating such a novel educational resource.
Introduction
For some years medical educators have recognised the exciting new opportunities the Internet provides.[1] In non-medical contexts, Internet-based formative assessment using multiple choice questions (MCQs) has been demonstrated to improve student performance[2] and randomised controlled trials have established the potential efficacy of computer-aided medical education.[3]
This paper briefly explores some of the practical, technical, security, privacy, legal and pedagogical issues addressed during the implementation of gasboys (http://gasboys.net), a free, non-profit Internet website dedicated to anaesthetic education. The goal of gasboys is to make it less onerous for anaesthesia trainees to pass their Fellowship examinations and, hopefully, become better anaesthetists. The site contains a growing database of over 7000 anaesthesia related multiple-choice questions with an innovative customised web-interface known as PETi. Any Internet visitor to the site may create a free account and begin answering questions.
Each time a user logs on, PETi uses a simple algorithm to ask them the questions they answered incorrectly and questions they have not previously answered. The algorithm prioritises each question by summing the scores from the user’s previous responses. Questions are further sorted to prioritise questions that have not been answered recently by the same user over questions that the user has recently answered.
Visitors to the gasboys website frequently answer questions at a rate exceeding 300/hour. The interface is optimised for keyboard use. The user simply presses the letter key corresponding to the option they wish to choose. They then press the space-bar to move to the next question.
Using an approach similar in many ways to that adopted by Wikipedia (http://en.wikipedia.org), users may also add, edit and comment on questions. This approach to knowledge aims to be consistent with that taken by educationalists "as dynamic, open ended, multidimensional, and public rather than static, finite, linear, and private".[3]
Each question in the database is associated with several supporting fields. Over 99 percent of gasboys questions contain one or more references to a textbook, journal article or respected website. Most journal references contain active hyperlinks to online copies when available. Questions usually have a brief explanatory text that displays after the user answers the questions to contextualise the question. Where relevant, questions may include diagrams. Finally, another field enables question editors to discuss questions.
Background
The gasboys website arose as a by-product of my own preparation for the Australian and New Zealand College of Anaesthetists (ANZCA) primary examination. Rather than writing study notes as I studied, I wrote myself MCQs and placed them in a database. These questions were based on material I was reading in textbooks, notes, and also on past examination papers recorded in the "Black Bank" (http://anaesthesiaMCQ.com). In order to facilitate my own study, I wrote an interface to the database that enabled me to run through the questions and automatically record the responses. I then developed a simple scoring system to enable the database interface to sort the questions so that questions I had got wrong were appropriately prioritised.
After I passed the ANZCA primary, I created an Internet interface to the question database. The new PETi interface not only enables Internet users to answer questions, it also enables users to edit and expand the question bank.
Technical infrastructure
PETi uses a standard set of the established tools for modern Internet-based applications. The site runs on commercially provided Apache web-server running under Linux. PETi itself is written in the PHP programming language (http://www.php.net), using SQL to communicate to a MySQL relational database (http://mysql.com). The PHP programme scripts generate HTML, with embedded JavaScript, to run in users’ web-browsers. PHP makes use of cookies to register sessions, enabling users to remain logged-on as they navigate around the site.
In order to avoid the delay associated with Internet communication, PETi caches or pre-loads the MCQs. This feature required JavaScript programming as this type of caching is not an inherent part of standard web-browser communication with web-servers. PETi initially supplies 20 questions for the web-browser to load into the client’s memory when it starts. Each question is displayed individually to the client as the user answers the question. When the client web-browser has less than 10 questions remaining in memory it requests another 20 questions from the server. This caching enables users to answer questions and receive feedback without any perceptible delay as each new question is already in the user’s computer before they request a new question.
Unfortunately, because it depends on JavaScript, PETi is not compatible with all web-browsers. JavaScript is poorly implemented in many web-browsers, especially older ones. For example, PETi does work on Internet Explorer 5.5 and up, but does not work on Internet Explorer 5.0, which was the last version Microsoft released for Macs. However, PETi does run on Mozilla/Firefox (http://www.mozilla.org) including the Mac version. Despite the browser limitations, PETi runs well on most computers manufactured after 1997 with 64MB of RAM, running Windows 98. It also runs on Apple Macs. The ability to run on such low-specification computers may explain the popularity of gasboys in developing countries (see Appendix C).
Security
Because of the nature of the Internet, security threats exist for users of gasboys and also for the gasboys site itself. While security breaches on the web-server increase the potential administrative load, the fear of security problems has the potential to deter users from using the site.
Internet security is a complex issue and even professionally run government web-sites suffer from sporadic hacking. A number of potential security threats exist for the gasboys site.
Potentially, hackers could exploit currently unknown vulnerabilities within the operating system, web-server or PHP to hack into the database and damage the tables and database schema or into the server and damage or delete PHP script files. This threat is mitigated by having the site hosted on a professionally run server with current security patches applied.
On one occasion the gasboys website was hacked into and the front page was replaced with political graffiti relating to the conflict between the Palestinians and Israelis. The hosting company responded within 12 hours to the attack and the site was restored without loss of any data.
Hackers could also guess or steal a user’s account name and password to log into their accounts. If this were to happen it would be possible to undo any action committed by a user within a certain time frame. Users are provided with a "personal profile" report which enables them to see the dates and times that their account has been active.
On the face of it, enabling any anonymous Internet user to create an account and immediately begin adding or modifying questions seems to be an invitation to hackers to vandalise the questions. However, the experience to date has been to the contrary, with all user modifications to the question bank being positive. As an insurance against malicious modification of the question bank, PETi records all modifications to the questions so that, if necessary, they could be undone.
Providing the interface to users within standard web-browsers, without requiring any special downloads, helps prevent most users from avoiding using the site because of security concerns. However, some users are extremely cautious on the Internet and feel uncomfortable enabling cookies and JavaScript on their browsers.[4] It is not known what impact these concerns have on the site’s popularity.
Privacy and legal matters
Privacy, and the regulation of privacy, is another complex issue for those seeking to exploit to the full opportunities made available by the Internet. Public anxiety about the potential for cheap and rapid collection and dissemination of personal information has lead to a wave of government regulation.
While the gasboys website is not legally required to comply with the 10 National Privacy Principles (NPPs) of the Commonwealth Privacy Act 1988, the Federal Privacy Commissioner[5] encourages exempt organisations to comply with the Act to maximise consumer confidence. Of the many overlapping privacy guidelines available, the Internet Industry Association’s 2003 draft Privacy Code,[6] consistent with the 10 NPPs, is the most applicable to the gasboys website. Gasboys complies with this draft code and the gasboys privacy policy is available at http://gasboys.net/privacy.
Publication of the gasboys website required resolution of several other legal matters pertaining to copyright and liability. The bottom of every page includes a link to a disclaimer and a Creative Commons (http://creativecommons.org) usage licence, and copyright notices. The ability of all users to add material to the site raises the risk of copyright infringement by users and then gasboys being vicariously liable. Users are specifically warned not to infringe the rights of others every time they add or modify content.
Marketing
Marketing includes not only promotional activity to recruit new users, but also actions taken to make the site more appealing to users once they have visited the site, to retain them as active users.
Marketing is traditionally divided into product, price, promotion and placement considerations. The gasboys "product" is presented to users as a website. As the website was initially developed based on the experience of a single user, it was important to provide facilities for users to provide feedback on their experience. Users are encouraged to access an online forum to make comments, suggestions or to report problems or difficulties. Especially in the early months of the website, user feedback was critical for allowing identification and rectification of several bugs and technical problems. Since then, the limited qualitative feedback received has been positive.
As for "price", gasboys is free. It was felt that charging even the smallest fee would subvert the goal of enlisting users to help expand and improve the question bank. The extremely low cost of hosting a web-site means that gasboys will remain free into the future.
Initially, gasboys was promoted to potential users by describing the site on other websites visited by anaesthetic trainees, such as that operated by Kerry Brandis (http://anaesthesiaMCQ.com). The long-term strategy is to rely on word-of-mouth. Placement of gasboys on the Internet opens it to a global market (see Appendix C).
The activity on the gasboys website (see Appendix B) is partially a reflection of the success of marketing activities, particularly the number of users visiting the site (see graph 1). Rather than simply counting the number of users or total questions, the fraction of users who stay and answer many questions is probably a better indicator of the perceived utility of the site by the users.
Issues with multiple choice questions
Gasboys is dependant on the MCQ format because of the ease of computerised marking. This format is particularly suited to facilitate rote learning through repetitive prioritised testing. The ability of the the PETi interface to allow users to answer over 300 MCQs per hour makes it feasible to provide complete coverage of a subject area. Even with the availability of human markers, MCQs allow better validity and reliability of assessment.[7]
However, the MCQ format is not without limitations and many of the abilities required by practising anaesthetists cannot be taught within the constraints of an MCQ. For example, although it is possible to construct MCQs that assess analytical ability, repeated exposure to the same question enables the answerer to rely on memory alone to answer the question.[7] In fact, some computer-aided learning systems deliberately avoid providing the correct answer to students to avoid rote learning.[2] However, one of the underlying assumptions of gasboys is that anaesthetic training is underpinned by a significant amount of rote learning.
Other issues with use of MCQs were well summarised by Hansen and Dexter:[8]
Good ones are difficult to construct; poorly constructed MCQs can lead to an inaccurate assessment of a student’s true ability. Constructing good items is time consuming, finding plausible distractors is often difficult, measuring the ability to organize and express ideas may be ineffective with MC items, and scoring can be influenced by reading ability.
Hansen and Dexter provide guidelines for the construction of MCQs and these are summarised in Appendix A.
With gasboys, the adverse effect of guessing is readily mitigated by increasing the number of times the candidate answers the question as the probability of consistently correctly guessing falls away exponentially. Increasing the number of distractors and the number of questions answered also decreased the impact of guessing.[7]
Therefore, while the MCQ and gasboys does allow efficient rote learning with extensive coverage, it does not provide a comprehensive teaching or assessment tool.
Conclusion
Assessing the merit of novel educational resources such as gasboys is both important and difficult. Greenhalgh et al[9] attempted to apply standard medical methods of scientific evaluation to medical education. They concluded that:
Educational questions have a more complex taxonomy, a less direct link with particular preferred study designs, and no universally accepted criteria for assessing validity.
While a randomised controlled trial of the impact of gasboys on FANZCA exam performance would be optimal, and might be possible, it would be very difficult. The low failure rate would necessitate a large sample size. For example, a study with an 80 percent chance of detecting a drop in the exam failure rate from 30 percent to 25 percent with 95 percent confidence requires a sample size of 1251.[10] This large sample size makes formal, objective assessment of the impact of gasboys on candidate performance impractical and suggests the need to rely on more subjective measures, as well as less direct objective measures. Production of optimal educational resources ultimately requires primary and secondary research evidence, students, peers, academics, critics and external assessment.[9]
Greenhalgh[3] observed that initial development of computerised learning materials for medical students requires input from senior faculty. This contrasts with approach taken by gasboys, which is essentially a student-led initiative. Although gasboys would be strengthened by collaboration with College Fellows, a student-led approach to material development is more consistent with post-graduate education. The tasks of quality assurance of the material and integration with other resources remain only partially addressed, and indeed remain as issues in other medical education projects.[11]
It remains to be seen whether gasboys will continue to expand and develop. Computer-aided learning initiatives, including gasboys, often depend on individual expert-champions.[3] Allowing others to contribute to the database and the use of a shareable MCQ format is a mechanism to defend the system from the loss of interest or availability of the originator. Furthermore, interaction with students, as provided by gasboys, is an excellent method of improving the question bank.[7]
Acknowledgements
The author wishes to express his sincere thanks to his wife Kitty Carra, Dr Kerry Brandis of the Gold Coast Hospital and Dr Jim Warren of the University of Auckland.
1. Harris JM Jr, Salasche SJ, Harris RB. The internet and the globalisation of medical education BMJ, November 10, 2001; 323(7321):1106.
2. Buchanan T. The efficacy of a world-wide web mediated formative assessment. JCAL. 2000;16(3):193 doi:10.1046/j.1365-2729.2000.00132.x.
3. Greenhalgh T. Computer assisted learning in undergraduate education. BMJ 2001 Jan 6;322(7277):40-4.
4. Federal Privacy Commissioner. Protecting your privacy on the Internet.2005. Available from: http://www.privacy.gov.au/internet/internet_privacy/index.html. Accessed 9 August 2005.
5. Federal Privacy Commissioner. Coverage of and exemptions from the private sector provisions. Office of the Federal Privacy Commissioner Information Sheet 12-2001. ISBN 1-877079-3-2; 2001.
6. Internet Industry Association. IIA draft Privacy Code (version 1.0). 2003. Available from: http://www.iia.net.au/IIA_PrivacyCode(EUdraft).pdf. Accessed 9 August 2005.
7. Tanner DE. Multiple-choice items: pariah, panacea or neither of the above? Am Sec Ed. 2003 Spr;31(2):27-36.
8. Hansen JD, Dexter L. Quality multiple-choice test questions: item-writing guidelines and an analysis of auditing testbanks. J Educ for Bus. 1997 Nov/Dec;73(2):94-97.
9. Greenhalgh T, Toon P, Russell J, Wong G, Plumb L, Macfarlane F. Transferability of principles of evidence based medicine to improve educational quality: systematic review and case study of an online course in primary health care. BMJ 2003 Apr 26; 326 (7395): 933.
10. Campbell MJ, Julious SA, Altman DG. Estimating sample sizes for binary, ordered categorical, and continuous outcomes in two group comparisons. BMJ 1995 Oct 28;311(7013):1145-8. BMJ 1996 Jan 13;312(7023):96.
11. Fleiszer DM, Posel NH, Steacy SP. New directions in medical e-curricula and the use of digital repositories. Acad Med. 2004 Mar 1; 79(3):229-235.
Appendix A: Guidelines for writing MCQs
Source: Hansen and Dexter[8]
1. Present a single, clearly formulated problem in the stem of the item. If more than one problem is given and the student fails the question, it is not possible to identify which problem caused the error.
2. State the stem in simple, clear language. Poorly written or complex questions may cause knowledgeable students to answer incorrectly. Avoid unnecessary statements in the stem and do not continue teaching on an exam.
3. Put as much wording as possible into the stem. It is inefficient to repeat words, and students will have less difficulty with shorter items.
4. When possible, state the stem in positive form. Asking a student to identify an incorrect alternative does not necessarily test whether the student knows the correct answer. Knowing what is true is generally a more important learning outcome than knowing what is not true. Negatively phrased items are often written, however, because they are easier to create: Positively stated items require the author to devise three distractors for a four-alternative question, but a negatively stated item requires that only one plausible alternative be devised-the answer.
5. Emphasize (by using italics and/ or boldface) negative wording whenever it is used in the stem. Not emphasizing negative wording may cause such wording to be overlooked.
6. Be certain that the intended answer is correct or clearly the best. Test quality will be improved and arguments from students will be lessened.
7. Alternatives should be grammatically consistent with the stem and parallel in form. Violations of this guideline may provide clues to the correct answer or aid students in eliminating distractors that do not match.
8. Avoid verbal clues that may eliminate a distractor or lead to the correct answer. There are several forms of verbal clues:
(a) Avoid similarity of wording in the stem and the correct answer. Similar wording can make the correct response more attractive to students who do not know the answer.
(b) The correct answer should not be more detailed or include more textbook language than the distractors.
(c) Avoid absolute terms in the distractors. Test-wise students will eliminate distractors containing words like "all," "only," or "never," because such statements are usually false.
(d) Avoid pairs of responses that are all-inclusive. This structure allows students to eliminate other alternatives because the inclusive pair covers all possibilities. An uninformed student would have a 50% chance of guessing the correct answer.
(e) Avoid responses that have the same meaning. Students will eliminate those alternatives because there can be only one correct answer.
(f) If alternatives consist of pairs of answers, avoid a structure that yields the correct answer-an intersection of repeated terms. For example, say the correct answer is x and y. To discriminate between students who know only part of the answer, an author might supply these alternatives:
- a) x and y
- b) x and z
- c) w and y
- d) l and p
A test-wise student who does not know the answer is attracted to the first alternative because the importance of x and y is signaled by their repetition.
9. Make all distractors plausible to those who do not know the correct answer. Good multiple-choice items depend on effective distractors.
10. Avoid using "all of the above." Students can select it as the correct answer by identifying any two alternatives as correct without knowing that they are all correct. Or, students can eliminate it by observing that any one alternative is wrong.
11. Use "none of the above" with caution. This may only measure the ability to detect incorrect answers. Although this alternative is more defensible in computation-type problems, "none of the above" is often used where the author has difficulty devising another plausible distractor.
12. Follow the normal rules of grammar and punctuation. For example, stems in question form should have alternatives that begin with capital letters. Alternatives in statement completion items should begin with lower-case letters. Periods should not be used with numerical alternatives, to avoid confusion with decimal points.
Appendix B: Activity on the gasboys.net web-site from inception until January 2008
Graph 1: Number of gasboys users by month, categorised by number of questions each user answered in the month
Graph 2: Total number of questions answered per month. The cyclic variation is largely explained by pre-exam activity
Appendix C: Geographic location of gasboys visitors June 2006 to January 2008[1]
[1] Data courtesy of Google Analytics. (https://www.google.com/analytics)
