The Utility of Bad Questions on Tests - and in Life

I received an inquiry on Friday, 12 October 2012 from a student whom I will call, "X".  X was concerned about a couple of poor-quality questions on an exam.*  After taking around 3 hours to figure out what was happening, I wrote a detailed response - about the questions, in question.  

It was a huge time investment - particularly on a vacation day, but I couldn't figure out why it continued to perplex me even after I went through all the trouble to figure it out.  I could not put the issue to rest.  In retrospect, I believe it had something to do with valuable lessons I learned regarding questions, and how they are framed, so here we go...

The following letters offer insight regarding teaching, learning, test-taking - and even life.   

For privacy's sake, I have culled out anything from the correspondences that might identify the student, "X.".

"Dear X,

Background

My requirement for you to have detailed notes is so that we can address whatever deficiencies in understanding that may arise.  Moreover, having your detailed notes helps me to pinpoint any deficiencies in the curriculum when a student makes a claim that s/he is not being adequately prepared for what s/he is being tested on.   Most of all though, is the fact that your notes are immensely beneficial to helping you develop disciplined thinking.

... Basically, I had two choices:

1] Reject your claim and let your grade stand, or

2] Personally review the Direct Instruction and lab Lectures you have alluded to. 

I decided to go back and review the lectures, and all other related material in order to determine whether your claim is in fact substantiated.  I don't mind doing this however, it is very time-intensive, therefore inefficient and not something I can do on a regular basis with the three hundred science students that I work for.  I hope you appreciate this being one of my reasons for requiring detailed notes from you all.

Findings

This is what I found in my investigation regarding your claims about the exam on Measurement on Chemistry - which you received an 84%: 

1]  Question 10:

The difference between an accepted value and an experimental value is called a(n) ____.

a.	error.
b.	percent error.
c.	measured value
d.	precise measurement.

You answered, "percent error."  
The answer was "error."  - that was covered in the lecture entitled, "How Reliable are Measurements."  of your lecture covering "percent error".  This is why you missed it:

There are two formulas you need to know in order to answer the question:

a)  error is the difference between accepted value and experimental value:  The testers have given you the answer in the question.  This is not a chemistry error on your part, but rather, your failure to logic out the answer.  The testers are attempting to get you to rely on prior knowledge from Algebra.   They are checking whether you have an enduring understanding of arithmetic and word problems. 

Mathematics tells us that the word, "difference" means subtraction.  The way the sentence sequence in the question is worded, tells us that experimental value is subtracted from the accepted value.  

The lecture did not cover the "+" or "-" values, but that has to do with whether the experimental value was greater or lesser than the expected value. 
 
b)  for percent error:  % error = (experimental value/accepted value) x 100% - so, you see, you were parroting an answer that you had memorized without actually understanding what you were being tested on.  

You may consider this to be trick question, but this is precisely where students of science make common errors; they think they know an answer without being skeptical about what they are being asked. 

2]  Question 21

"How close a series of measurements are to each other"

You answered, "percent error"  the answer was "Precision" - and, it was also covered in the Lecture, "How Reliable are Measurements."   at the 58 second mark.  It was the archery example; arrows landing in close proximity on a target may be precise and still be lacking in accuracy. 

3]  Question 24:

An experimenter's best estimate of how far an experimental quantity might be from the "true value" is called __________.

a.	uncertainty	c.	science
b.	an inference	d.	a guess

You answered, "D."

The answer was, "uncertainty." - which was not covered directly in your lecture.  But let's go through the possible answers:

uncertainty - actually has to do with the difference between what we know and what we don't know.  Thus, uncertainty deals with accuracy and precision - both of which are related to one another based upon qualitative and/or quantitative differences.  Thus, by deduction, this seems the most plausible answer because it actually deals with numbers. 

Science - is a discipline that involves  study  - in the bigger picture, while it does involve making estimations, it would be wrong to define science as making estimates between experimental quantities and true values.  In other words, this definition is too limited to describe what we do in science.

Inferring - is the process whereby we draw a conclusion about something or have a hunch that something either is going to occur or has occurred.  It too could mean that we are estimating but again, while inferring includes estimating, estimating is not inferring.

a guess - is another word for hypothesis which means drawing a conclusion between what we see and making some sort of prediction about why something is occurring based on our observations.

You may not like this problem because it was not spelled out for you in either lecture.  But, this is what real science is like; it is about being to discern minute differences based on your observations.  It is what separated people like Isaac Newton, Albert Einstein and the late Richard Feynman (the physicist who figured out why the space shuttle Challenger blew up - it had to do with expansion and contraction of an "O" ring) from the rest of us mere mortals...

CONCLUSION

I really want you to put all your notes - prior notes included - into outline form because you will be able to reference all of them for the cumulative exam....you should always be able to review your notes for future classes - either here or at the university once you graduate and move on from here.

I also want you to burn this in your brain; your notes are an extension of your memory.  That said, sloppy notes =sloppy thinking.  If you invest your energy into excellent notes, you will remember finer details, and you will organize them much better that you are more apt to recall them on demand.  Failing that, you can always rely on them for all your assessments, quizzes and exams...

More than that, I think it would be wonderful for you to actually see the qualitative improvement in your note-taking skills, and their direct correlation to your test scores over time - from now until you graduate.*

Please let me know what you want to do, and above all, do not lose heart.  This is tough material you are working with.  The intellectual playground you are now in, is one which all science students throughout history have had to learn the rules in order to play there.  You are no different.  We all have to pay these dues.

Always do your best and never quit.

Mr. M"

*...I occasionally look back upon my notes from high school and it brings both a chuckle and a smile.  I owe so much to Mr. Mike Gruner, my high school science teacher and All-American Hero.  Thank you Mr Gruner.  I can never pay you back for all you did...  :O)

Now here is the follow up letter to Mr X.  

"Dear X,

I have been giving a lot of thought to your e-mail yesterday, and I want to thank you. 

It occurred to me when I awoke this morning, that the reason why I invested so much time - 3 hours - into my response was because I saw it a s a teaching moment for you - and for me.

It helped me to understand the utility of difficult, or even 'bad' questions on tests.  To my way of thinking, poorly worded questions are more reflective of what actually happens in the real world - especially science - where neatly defined answers rarely come. 

I am willing to argue that the, "percent error" question taught both of us more than we ever bargained for.  It did so because it caused us to think deeper about the implications of what an error in science is defined to be, it also helped us to realize that in science - as in life - some answers are not so obvious; more importantly, we should never discount the obvious because sometimes the simplest answers are the most complex.

That is what all great thinkers throughout have found; the simple is truly profound.  I recently put two different posts on the blog;

1]  The Scientific Method - Why it counts (click here)

2]  Occam's Razor (click here)
 
They both speak to this issue.  You ought to visit the blogs, because both posts offer us insight regarding the discipline of Science, and moreover, its utility for dealing with nearly every problem life can throw at us.

It may seem funny to you, but I used the Scientific Method when I was a Freshman in College because I was terribly shy, but loved to dance.  In those days, we needed partners to dance so, I made my observations, came up with a plan - my hypothesis - in order to get the girls to dance with me.  My initial experiment filed so I had to tweak my plan.  It turned out that just walking up to a girl and asking her to dance wasn't really how asking a girl to dance simply didn't work.  I theorized that using humor might help me out so, I tried it and, as the French say, "voila!" it worked.  I planned my work, I worked my plan and I attenuated it to the point where I was soon being asked to dance.

I could go into how I used the Scientific Method to get my parents to let me drive a car, but this is becoming a digression.  Perhaps we can talk about that little venture some other time.

Getting back to the test.

The next question we covered had to do with usage of a common logic tool used in the Scientist's bag of tricks; deductive reasoning.  The question whose answer was, "uncertainty" most certainly was the answer, and we came to it by eliminating each of the other answers.  

I am not really bragging here, but I am pretty proud of myself because I really know how to ask good questions.  That is perhaps the most important thing to know about learning.  We never know answers, but if we can formulate intelligent questions, then we have a better chance of coming up with useful answers.

To that end, I had an epiphany about test-taking - and dealing with life's questions; whenever we are confronted with a question, our natural inclination is to come up with an answer.  Because of that, our approach is flawed.  We should always face a question with a question that way we know what we are in fact being asked.  I am convinced the lion's share of questions on tests, and life for that matter are wrongly answered because we are so solution oriented as a species.  

Moreover, I am absolutely convinced that this is the first point of divergence that truly brilliant Human Beings take.  It is what separates true innovators from the rest.  I would even go so far as to say that it is a trait shared by all of the brilliant minds that have gone before us.  It is a fundamental trait commonly referred to as, "geniuses."

I mentioned Sir Issac Newton - who invented Calculus in order to explain his theory on Gravity and the movement of the planets.  He did this in less than two years - at the age of 23.  Building upon that base of knowledge, Albert Einstein developed the theory of Relativity at age 27 and came up with the famous energy equals mass times the speed of light squared.  He did this because he wondered - at age 16, what it might be like to travel on a light beam.  

I have another post on Femto photography going on at MIT RIGHT NOW (click here) - they are taking photos of photons of light at a trillionth of a second.  In doing so, they are showing the world what Einstein imagined in his head; light can act like a particle, and a wave.  This is called, "the dual nature of light."  Again, you should visit the blog and see for yourself how people with ingenious questions are coming up with remarkable answers.

I also mentioned Richard Feynman - an American Physicist who was a member of the Congressional appointed committee charged with investigating the Space Shuttle Challenger's disastrous explosion.  Through his ingenious, elegant demonstration, he showed how the disaster could have been avoided.  Placing an "O" ring in ice water, he was able to show the world a simple concept of physics - expansion and contraction due to heat transfer.  

Like the other two giants I mentioned here, Richard Feynman had no answer book to turn to.  All he had was his basic knowledge built upon from constantly building upon his learning by making connections with prior knowledge.  They also shared something in common, learning is fun.  Learning is done not because of tests, but rather because it it is a form of play.  That is why I referred to Science as an intellectual playground. 

Mathematics and science for that matter is not at all easy.  But what it lacks in ease of understanding, it more than makes up for in being fun.  If learning chemistry isn't fun, then we are approaching it wrong.  Finally, I want to point out that mathematics and the exercises we must endeavor to master are difficult to master because truly conceptualizing what happens in mathematics really goes on between the ears.  

Einstein lamented, just days before he died, that he didn't know math better.  Yeah right!  Well, Einstein did all of his Science in his head and only later pushed himself to master Mathematics as a means of explaining what he envisioned as he formulated thought experiments in his mind.  Again, this speaks to not having the answer and working to find it. 

I close this correspondence with a reminder; noting ever worth having is easy.  You have demonstrated a talent for Chemistry.  I can confidently predict that one day you will far surpass my ability in this field of study.  I can also predict that when you get to the playground of Physics, you are going to have a blast.  You are a brilliant kid and whether you realize it or not, you are just approaching the time in your life when your brain is going to be working optimally.  

That is why it is important to fill your brain with all of these basics.  Forget about the grade point and learn for the sake of learning.  I assure you the grades will come.  Nothing is easier than when it becomes a game - when it becomes fun.  Take the pressure off yourself and understand that there is nothing more powerful than the creativity of an unencumbered mind.  Humans as a species learn through play, so learn to play and play to learn.

Always do your best, and never quit.

Mr. M"

Occam's Razor - Keeping it Simple

Albert Einstein cautioned us that, "Everything should be made as simple as possible but no simpler." 

His advice echos a commonly practiced principal in science called, "Occam's razor."  

Keep this principal in mind whenever you approach a problem.  This strategy will also prove helpful in the future when you are designing experiments and explaining how, or why something is the way it is.  It is even a helpful tool to use when dealing with whatever problems you encounter in life. 

Here is a pretty good explanation of the concept:

"Ockham's razor - also written, 'occam' is attributed to the fourteenth century Friar, William of Ockham, has become the hallmark of scientific theories ever since.  

Ockham stated that 'Entities should not be multiplied unnecessarily.'  This has been taken to mean that whenever two or more explanations exist for a given phenomenon, the simpler one is generally preferable.

In other words, scientists use Occam's 'razor' to cut away any superfluous elements of a theory preferring to leave only the simplest explanation necessary, for logic dictates that the simpler answer tends to be more preferable than an overly complicated one.  This little rule is useful in any logical situation, even beyond the realm of science."

                                                                                    - From: Physics - Everyday Science at the Speed of Light, 2010

Note-Taking Tips - 2 - Getting on-line notes into a Word Document

This post is very detailed, however once you've gone through it, you will not forget it and the payoff is having awesome notes that you can easily edit and review in the future. 

Create folders for each subject in your Flash Drive - for example;

1]  Click on the 'Save As" button in your 'File' pull-down menu located at the top left corner of your computer screen.
2]  Press the 'New Folder' button and type in, "Science - Fall 2012" into the highlighted area.  A best practice would be to create similar folders for each of your classes.
3]  Open the folder, and save your newly created document into it - make sure you label the document in a way that you can easily identify its contents for easy reference and future use.

You can then copy the provided outline which has already been laid-out for your coursework in the on-line Course Map.  

To copy and paste from it;

1]  Highlight what you want to copy 
2]  Press the buttons, "ctrl" and the key, "c" at the same time, and then 
3]  Open up a new Word Document
4]  paste what you copied into your Word Doc by pressing, "ctrl" and the key, "v" at the same time.

Using the prepared outline for a given subject is helpful because all the guesswork has been taken out of your note-work taking experience; the outline  lets you know exactly what you need to know and even what order you need to know it.  

In education, we call this process, 'scaffolding' - meaning that what you learn builds on what you have already learned in the past.  Courses typically follow this pattern; They start out with basic, general information and then build on it from there.  Stating with simple concepts a course of study then builds upon that knowledge by adding more complicated, and sophisticated information.

Being familiar with the way courses are laid out puts you at a distinct advantage in comparison to your fellow students.  Knowing how to process that information through development of superb note-taking skill will help to assure you are getting the most out of your coursework.  Mastering these techniques, plus a few test-taking strategies, and people will be looking at you in complete awe.  Thanks to your efforts to master these techniques, you are guaranteed to be at the top of that intellectual mountain.  Few people will be able to understand you anymore.  Before you know it, people will be calling you a, 'braniac,' or 'Doctor,' and eventually, 'Boss.'  Go figure...

At this point, you can begin populating your outline with your notes and vocabulary etc.  All you have to do is copy your Vocabulary, Direct Instruction notes,  Lab instruction notes, and homework etc. onto the new Word document you have just created.  Your next step is to fine tune your notes by spell-checking and grammar checking them so they are a beautiful representation of how they are organized in your mind.  Be certain to save your work often so that it doesn't get lost.  Now your masterpiece is available for future access. 

Excellent note-writing is a Habit of Mind that you will soon master.

Copying and correcting for grammatical and/or spelling errors will also help to sharpen your writing skills.  It increases your familiarity with Word's powerful features - like bold facing, underlining, highlighting, italicizing, creation of headers and numbering your pages at the bottom etc.  Working with Word docs  offers you another opportunity to review the study material.  Working so actively on your notes also helps familiarize you - not only with where the information is located in your notes -but how you have it organized too.   These developed skills contribute mastering what you have been studying.

Excellent notes, guarantee your ability to pass every one of your quizzes, assessments and exams with flying colors.

Remember:  Notes are an extension of your memory - the better you write them, the more organized your thoughts will be. 

Always be willing help your peers who are struggling to master this powerful skill.  If you notice someone is having a difficult time, please help them out.

It is critical that we all develop good networking skills, along with the desire to offer people a hand up when they need it.  You never know, someday you may need help as well.  Invest in your,  'Helper's Bank Account' and when a rainy day comes, you will be able to draw from it. 

Remember the axiom, "Helping others helps me." It is a great feeling to know that your thoughtfulness made someone's world a little better.  More than that, people remember acts of kindness.

Note-Taking Tips - 1 - word-processed notes

 

"Any intelligent fool can make things bigger and more complex... It takes a touch of genius --- and a lot of courage to move in the opposite direction." - Albert Einstein 

Note-Taking Tips for Students - Part 1/2

Taking notes when viewing your on-line classes is one of the most effective ways to understand the material being presented in class. Unless you have a photographic memory, you'll need to learn this important skill. Follow these general guidelines and you’ll be a pro in no time.

1. Come to 'class' prepared. Always bring enough paper, a writing instrument of your choice and a flash drive to class.
   
2. Be organized and follow a routine.  If you are also writing out your notes, it's not a bad idea to start a new page for each new class, but your on-line classes have outlines already laid out in a logical order.  All you have to do is copy & paste the outline into a Word document, and then transfer your on-line notes to it.  For written notes, be sure to put the date on the top of the first page. This way you will know exactly where the notes for each topic begin.  For written notes, organize them in their own separate binders/spiral notebooks.
   
3. Don’t try to write down every word your teacher says. You will not be able to, even if you can write very fast. More importantly, in trying to do so, you will miss the overall point your teacher is trying to make.  Whenever possible, write down the information in your own words.  You can always pause the on-line instruction when you are writing your notes.  Using the preexisting course outline format for typing in your notes is a good idea because they have already been prepared perfectly for you to copy what you have already typed into the e-notes and then paste them into Word documents that you are  storing in your flash-drive memory sticks.
   
4. Write down the big ideas. Listen for facts, connections, and main ideas. This may take a while to get used to, because you will need to divide your attention between listening to the teacher and writing your notes. Don’t get frustrated. In time, this will become easier.
   
5. Use abbreviations for commonly occurring names and words. You can develop your own abbreviations, so long as you don’t forget what they stand for. For example, in a lecture on Einstein, you might write his name out the first time, and then abbreviate it as "E" throughout the rest of your notes. Long words such as government can become "govt" and federal can become "fed." Develop your own system and stick to it, once it works.
   
6. Again, for written notes, leave lots of room on the page. When writing, leave ample space between ideas. This is like pausing before you begin a new sentence. Your notes will much easier to read, and you’ll have space to add information later on, if needed. Don’t try to cram everything onto one piece of paper. Copying and pasting your e-notes into Word documents helps you to further organize your thinking because you have to fit the right information you are copying into your outline.  Correcting spelling, and grammar mistakes, helps you sharpen your spelling and writing skills.   Remember to use Word's spell checker (misspelled words are automatically underlined in red.  Green underlining in Word docs indicate mechanical (too many spaces between words) or grammatical errors.  A note of caution: be careful with both features because not all words are stored in the Word doc dictionaries, and not all grammar mistakes are grammar mistakes.  Nevertheless, both features are convenient tools that will  enhance your writing skills.
   
7. Use diagrams and pictures where necessary. Sometimes it is helpful to draw pictures that illustrate the connections between ideas, sequences, or events. Don’t be afraid to draw pictures that will help you understand the material.
   
8. Write down corresponding page numbers from other textbooks or reading materials you are referencing as you study the topic at hand. Teachers often use the textbook to refer to ideas you’re learning in class. Recording the page number of corresponding ideas and homework assignments can come in handy later on.
   
9. Review your notes for accuracy. It’s a good idea to look over your notes sometime after class for accuracy and completeness. Consider doing this just before doing your homework to get yourself back in the mindset of the material. 
10. Compare your notes with fellow students - if that is at all possible. Sometimes you are going to miss something said in the lectures. Consider forming a partnership with another student at the beginning of class on whom you can rely (and vise-versa) for comparing notes. 
11. Get used to typing out your notes because you will be able to do so on a laptop during lectures when you get to university.  Typed notes are easier to read.  You can also learn a lot when you are copy-editing and correcting any errors.  This is a great skill to develop as you listen to your on-line lectures.  Develop it now while you have the time.

* Adapted from McGraw Hill note-taking strategies