[222003330010] |<title>Web Experiment Tutorial: Chapter 10, Recruiting Participants</title>
[222003330020] |Several years ago, I wrote a tutorial for my previous lab on how to create Web-based experiments in Flash.
[222003330030] |This is the final chapter in the original <a>tutorial</a>. 
[222003330040] |<h1>10. Recruiting participants online</h1>
[222003330050] |<b>1.     </b><b>Overview</b> So now you have an experiment implemented on the Web.
[222003330060] |All you need are participants.
[222003330070] |Where do you get them? 
[222003330080] |If you need only very small numbers of subject (50-100), this part is easy.
[222003330090] |If you want larger numbers of subjects, or if you want to run several experiments under the same URL (so as to prevent the same subject from participating in multiple versions of the experiment), this may be the most challenging part of Web-based experiments. 
[222003330100] |There are several methods you can use.
[222003330110] |I recommend using all of them.
[222003330120] |Each will be discussed below in turn, but briefly: you can list the experiments on experiment portal pages, you can recruit from within your own social network, you can buy ads, you can promote the experiments in online forums, you can blog, you can swap links with other researchers, and you can get media attention. 
[222003330130] |Media attention, if you can get it, is far more valuable than all those other methods combined. 
[222003330140] |<b>2.     </b><b>Experiment portal pages</b> There are several web sites that list online experiments.
[222003330150] |By far the one that has provided the most subjects to vacognition is: <a>http://psych.hanover.edu/research/exponnet.html</a> The second-most useful is: <a>http://genpsylab-wexlist.unizh.ch/</a> Others, much less useful, include: <a>http://www.onlinepsychresearch.co.uk/</a><a>http://www.language-experiments.org/</a>
[222003330160] |Another place you can list is: <a>http://www.psychology.org/</a> In the first 3 weeks of May, 2007, vacognition (my previous site) received 251 hits from psych.hanover, 67 from genpsylab-wexlist, 15 from onlinepsychresearch, and only 2 from language-experiments.
[222003330170] |Here are some other lists I have not used, which may or may not be useful: <a>http://www.socialpsychology.org/expts.htm</a><a>http://www.w-lab.de/lab-united/submit.php</a> 
[222003330180] |<b>3.     </b><b>Your social network</b> Your own friends and family are the most likely to be convinced to do your experiments.
[222003330190] |Some of them may pass along the URL to their friends and family.
[222003330200] |Every time I have sent out requests to my F&F, I get about 40-50 participants in various studies. 
[222003330210] |You can also use Web-based social networking.
[222003330220] |For instance, I have an account on Facebook.com.
[222003330230] |My page lists vacognition.
[222003330240] |A friend of mine created a Facebook “group” called “Harvard Studied My Brain.”
[222003330250] |We invited all our friends to join (about 200), and anybody on Facebook could in theory join if they found the group in a search.
[222003330260] |35 people did join, and many more have clicked on the link. 
[222003330270] |To make the group more enticing, I created a “certificate of membership,” which members can download.
[222003330280] |Generally, it is good to think about why anybody would want to participate.
[222003330290] |What can you do to make it more fun? 
[222003330300] |Other social networking sites include Stumbleupon.com, Reddit.com and Digg.com. “vacognition” has accounts for all of those.
[222003330310] |Every time a new webpage mentions your website, it is a good idea to “vote” for that website on Stumbleupon, Reddit and Digg.
[222003330320] |This increases the likelihood people will surf to that page, and then to your page.
[222003330330] |However, these services only have an effect if a fairly large number of people vote for the site, and the traffic may or may not be high-quality.
[222003330340] |At one point, a number of people voted for vacognition on StumbleUpon.
[222003330350] |In the space of an hour, we suddenly got 150 visitors.
[222003330360] |However, most did not participate in any experiments, and the traffic died down within 90 minutes.
[222003330370] |This is likely due to the fact that users of StumbleUpon are randomly sent to the website.
[222003330380] |In contrast, users of Digg or Reddit know what sort of website they are going to and are more likely to actually be interested.
[222003330390] |Visitors we have gotten through Digg have been highly likely to participate in an experiment. 
[222003330400] |You can also add a link to your website as part of your email signature.
[222003330410] |Ask your labmates to do so as well.
[222003330420] |Ask your friends to link to your website from their websites. 
[222003330430] |<b>4.     </b><b>Purchasing ads</b> You can also purchase ads.
[222003330440] |One obvious place to put ads is Google.
[222003330450] |I have never tried this. 
[222003330460] |I did, however, buy adds on Facebook.
[222003330470] |For $5/day, Facebook promised to display my add to at least 10,000 people on the Oberlin network.
[222003330480] |For another $5/day, it was displayed to another 10,000 people in the Harvard network. 
[222003330490] |I bought $20 worth of ads as an experiment.
[222003330500] |Vacognition got about 80 hits.
[222003330510] |That’s 4/$1.
[222003330520] |This is not very impressive, but it may be worth it.
[222003330530] |Also, my ads may not have been very good.
[222003330540] |(Keep in mind that a hit to the website does not mean that the person completed or even started an experiment!) 
[222003330550] |<b>5.     </b><b>Online forums</b> Another way to recruit participants is to mention the website or a particular experiment in an online forum.
[222003330560] |Here, it is particularly important to make the post relevant to the forum discussion.
[222003330570] |Otherwise, you are spamming and may (not unfairly) receive hate mail. 
[222003330580] |There are many psychology or science forums.
[222003330590] |It may be perfectly fair to write a post called “Please help me finish this experiment.”
[222003330600] |Another option is to write about the topic you are studying (“Visual short-term memory is very limited.
[222003330610] |We are trying to find out exactly hoe limited.
[222003330620] |Please do this experiment.”).
[222003330630] |You can also be very oblique about it.
[222003330640] |Post something interesting about your area of research, and just mention your website (“It turns out 1/100 people have prosopagnosia from a young age, not as a result of stroke.
[222003330650] |This is something we’ve found through our online experiments at <a>www.faceblind.org</a>.
[222003330660] |In fact, blah blah blah.”) 
[222003330670] |You can also pick targeted forums.
[222003330680] |If you are studying reading, you can post on dyslexia or reading education forums.
[222003330690] |(“My colleagues and I are trying to better understand reading.
[222003330700] |The results may eventually help us better understand how to teach reading to children.
[222003330710] |We need volunteers for our short, 3-4 minute experiments.
[222003330720] |I thought that participants in this forum might be particularly interested…”) Because I use Flash for my experiments, I have also posted on forums dealing with Flash programming (“You may be interested in this other application of Flash technology…”).
[222003330730] |Also, sometimes I have a question about Flash, and I post the question, with a link. 
[222003330740] |There are also website creation forums where you are encouraged to showcase your website. 
[222003330750] |The best success I have had with this method is when my experiment has been set up as a type of quiz.
[222003330760] |My visual short-term memory experiment gives people a score at the end.
[222003330770] |So I posted the experiment on several forums where people advertise quizzes (“How good is your short-term memory for what you see?”).
[222003330780] |Normally, a forum post generates only a small amount of traffic (0-10 hits), but these posts on quiz forums produced as many as 100 each. 
[222003330790] |Vacognition has accounts on many, many message forums. 
[222003330800] |<b>6.     </b><b>Swapping links</b> Reciprocal advertising is an easy-to-use but very limited strategy.
[222003330810] |Vacognition has a “links” page, where we link to other websites, mostly other Web-based experiments.
[222003330820] |In return, those websites link to ours. 
[222003330830] |This serves two purposes.
[222003330840] |First, visitors to those other sites may click on the link and come to our website.
[222003330850] |This is extremely rare. 
[222003330860] |Second, the more links there are to a website, the better its “page rank” –that is, the higher it appears in the list of search results.
[222003330870] |Swapping links improves your page rank, and thus you are easier to find through Google, etc.
[222003330880] |My data suggest that visitors that come via Google tend to be low-quality visitors –that is, they tend not to participate in experiments.
[222003330890] |However, a few do, and it doesn’t hurt. 
[222003330900] |Usually I arrange these link swaps by emailing the webmasters of websites that I think may be interested.
[222003330910] |Most do not respond, but some do. 
[222003330920] |<b>7.     </b><b>Media attention</b> By far the most effective method is media attention.
[222003330930] |Extremely successful online labs (like <a>faceblind.org</a> or the <a>Moral Sense Test</a>) get a lot of mainstream media attention, and they also get huge numbers of participants. 
[222003330940] |Media attention is hard to orchestrate.
[222003330950] |Ideally, your research will be so interesting that reporters will come to you.
[222003330960] |However, you can contact reporters yourself.
[222003330970] |The university can put out a press release.
[222003330980] |I got a fair amount of media attention after Georgetown wrote a press release about a paper I wrote. 
[222003330990] |In the end, though, you have to have work that is interesting to reporters and the public (see “The most important thing,” below). 
[222003331000] |<b>8.     </b><b>Blogging</b> Bloggers are more approachable members of the media.
[222003331010] |Bloggers of many shades and stripes may be interested in showcasing your experiments.
[222003331020] |And they are much more likely to respond to an email.
[222003331030] |Some blogs produce disappointing traffic.
[222003331040] |I guest-blogged for The New Scientist, whose blog gets a thousand hits a day, but I only got a few dozen hits out of it.
[222003331050] |However, Skepchic blogged (without my contacting her) about one of my experiments, and I got about 300 hits. 
[222003331060] |You can also write your own blogs.
[222003331070] |This will be of minimal help if you don’t attract a following, but even a blog with little following and only one new post every month or so can generate some traffic.
[222003331080] |The links from the blog to your website can also help your page rank (see “Swapping links”). 
[222003331090] |<b>9.     </b><b>Email list</b> We maintain an email list.
[222003331100] |On several parts of the website, visitors are encouraged to join a Google Groups email list, which now has over 100 members.
[222003331110] |The list is emailed when new experiments are posted or results have been posted, although I try to keep this to a minimum.
[222003331120] |If you overuse an email list, people tend not to read the messages and/or withdraw from the list. 
[222003331130] |Setting up a Google Groups email list is simple, and it can be set up so that anyone can join.
[222003331140] |Vacognition’s list can be found here: <a>http://groups.google.com/group/vacognition/web/home</a> <b>10.  </b>
[222003331150] |<b>The most important thing</b> When recruiting participants, you should always keep in mind one question: Why would anybody want to participate in this experiment? 
[222003331160] |Participants are expending resources (time, energy, and sometimes money) in order to participate.
[222003331170] |What is the product that you are selling them? 
[222003331180] |This is particularly important when trying to generate media attention –whether newspapers or bloggers.
[222003331190] |You may get your brother-in-law to blog about your experiment as a favor (mine did), but most bloggers aren’t going to write about something if they don’t find it interesting.
[222003331200] |Make it interesting.
[222003331210] |<a>Testmybrain.org</a> is a great example of a site that is fun, and not surprisingly it gets tons of traffic. 
[222003331220] |However, this issue is important even when using online experiment lists.
[222003331230] |Anyone who visits an online experiment list is already interested in doing online experiments.
[222003331240] |However, these lists post many experiments.
[222003331250] |No visitor is going to do all of them.
[222003331260] |So how do they choose which one(s) to do?
[222003331270] |Presumably, this is partly a function of how interesting the experiment looks.
[222003331280] |Compare: “This experiment investigates the role of proactive interference in estimates of visual short-term memory capacity.” with “How much of what you see can you remember?
[222003331290] |Probably less than you thought.
[222003331300] |Take this 5 minute quiz to see how many visual objects you can remember.
[222003331310] |Typical scores are between 1 and 3 objects.” 
[222003331320] |Which experiment sounds more interesting?
[222003331330] |They are the same experiment. 
[222003331340] |You will want to craft your pitch to your audience.
[222003331350] |If you are posting on a forum for vision scientists, the 2<sup>nd</sup> description above may come across as patronizing.
[222003331360] |However, if you are posting to a forum about online quizzes and games, the 1<sup>st</sup> description will probably get you banned from the forum for spamming. 
[222003331370] |The design of the website itself also matters.
[222003331380] |An ugly, unprofessional-looking website will turn away visitors.
[222003331390] |Many participants are participating because they are interested in science.
[222003331400] |Make sure they learn something.
[222003331410] |Post results.
[222003331420] |Have pages that discuss the research topics.
[222003331430] |Make sure the debriefing is informative.
[222003331440] |Many participants find seeing their own results very motivating, so if possible, try to incorporate that into the experiment. 
[222003331450] |You can also experiment in your advertising.
[222003331460] |Try different pitches.
[222003331470] |See which work the best.
[222003331480] |Modify the website and see if the number of visitors who actually participate in experiments increases or decreases. 
[222003331490] |<b>11. Where does GamesWithWords.org get it's traffic?</b> 
[222003331500] |I currently use Google Analytics to track my web traffic.
[222003331510] |Here is what it shows for the top 10 referrers from Dec 1.
[222003331520] |2009 through April 21 2010: <a><img/></a> As you can see, the biggest chunk of traffic comes from people simply typing in the name of the site.
[222003331530] |Word of mouth seems to do a great deal.
[222003331540] |One thing to consider, however, is also the average time on site and the bounce rate.
[222003331550] |By these measures, the direct traffic is better than those who come via Google. 
[222003331560] |I should note that this traffic to dwarfed by weeks in which I get media attention.
[222003331570] |I can easily get several thousand visits per day when the site is mentioned in a prominent news source (which has not happened in the last few months, unfortunately).
[222003331580] |Notice also that while there are many other sources of traffic beyond the top 10 listed here, all the rest combined only contributed 1,385 visits.
[222003340010] |<title>Web Experiment Tutorial: Chapter 8, Additional MySQL</title>
[222003340020] |Several years ago, I wrote a tutorial for my previous lab on how to create Web-based experiments in Flash.
[222003340030] |I am currently posting that <a>tutorial</a> chapter by chapter. 
[222003340040] |There are a few other useful things you can do in MySQL. 
[222003340050] |<b>1. Selecting certain rows from a table.</b> 
[222003340060] |Maybe you want to know how many people have completed your experiment.
[222003340070] |Your experiment has 6 trials, the last of which is called trial “5”.
[222003340080] |The easiest way to find out, then, is to use the following command: mysql> select * from VSTM where trial = 5;+-------------+-------------+----------------+----------+-------+---------+----------+---------+-------+------------+----------+---------------+------------+| subject_age | subject_sex | subject_vision | initials | trial | correct | stimulus | matches | probe | date       | time     | ip            | subject_id |+-------------+-------------+----------------+----------+-------+---------+----------+---------+-------+------------+----------+---------------+------------+|           2 | male        | no             | jkh      |     5 |       1 |        3 |       1 |     3 | 2007-06-05 | 14:57:33 | NULL          |       NULL ||          26 | male        | yes            | jkh      |     5 |       1 |        3 |       0 |     2 | 2007-06-05 | 16:01:40 | 140.247.95.39 |          4 |+-------------+-------------+----------------+----------+-------+---------+----------+---------+-------+------------+----------+---------------+------------+2 rows in set (0.01 sec) 
[222003340090] |Two rows were found, so two subjects completed trial #5. 
[222003340100] |You can use more complicated where statements: mysql> select * from VSTM where trial = 5 and correct=1; 
[222003340110] |<b>2. Deleting rows from a table.</b> 
[222003340120] |To delete all rows from the table VSTM , type: mysql.> delete from VSTM; To delete only certain rows, try: mysql> delete from VSTM where subject_id = “NULL”; 
[222003340130] |<b>3. Copying a table.</b> 
[222003340140] |I often run multiple version of an experiment under the same name.
[222003340150] |Subjects do not know that the experiment has changed.
[222003340160] |I do this when I don’t want people to participate in the different versions of the same experiment. 
[222003340170] |I want the data from each version to go into separate tables.
[222003340180] |For instance, suppose I’ve finished collecting data from the first version of the VSTM experiment and I want to start a second version. mysql> create table VSTMver1 like VSTM;Query OK, 0 rows affected (0.00 sec) mysql> insert into VSTMver1 select * from VSTM;Query OK, 12 rows affected (0.00 sec)Records: 12  Duplicates: 0  Warnings: 0 mysql> delete from VSTM;Query OK, 12 rows affected (0.00 sec) 
[222003340190] |I now have a table called VSTMver1 with all the data that has been collected so far.
[222003340200] |The table VSTM is now empty.
[222003340210] |(Note that the subject numbers will not start over.
[222003340220] |To do that, you would have to reset VSTM_id_incrementor.
[222003340230] |I’m not sure of any easy way of doing that other than deleting VSTM_id_incrementor and creating it again.
[222003340240] |Just deleting all its rows doesn’t work.) 
[222003340250] |As usual, please leave comments if you have any questions. 
[222003350010] |<title>Web Experiment Tutorial: Chapter 9, Finishing Up</title>
[222003350020] |Several years ago, I wrote a tutorial for my previous lab on how to create Web-based experiments in Flash.
[222003350030] |I am currently posting that <a>tutorial</a> chapter by chapter.
[222003350040] |<h1>9. Finishing Up</h1>
[222003350050] |We still haven’t finished putting the experiment online.
[222003350060] |You will recall that clicking the link in Consent.html opened two files.
[222003350070] |It opened test_in_progress.html.
[222003350080] |It also opens test_popup.php and centers it in the middle of the screen.
[222003350090] |The only thing to be interested in here is the first command:
[222003350100] |$link = “VSTM.swf”. 
[222003350110] |Make sure this is set to the name of your file.
[222003350120] |Notice the extension .swf.
[222003350130] |This is the compiled form of a Flash file.
[222003350140] |We’ll make it momentarily.
[222003350150] |This is what will actually be displayed. 
[222003350160] |<b>2. Making .swf files. </b>
[222003350170] |Each time you “test” a .fla file in Flash, it compiles the file into a .swf file in the same directory.
[222003350180] |However, it is best to go to File->Publish to create the .swf file.
[222003350190] |It will also create an .html file (which simply runs the .swf file) and and .swd file, which does something or other.
[222003350200] |You can use this <a>VSTM.fla</a> file, though hopefully that is identical to the one you made. 
[222003350210] |Take that .swf file and copy it into your VSTM directory on your website.
[222003350220] |Now you can browse to your index.html file and run your experiment directly.
[222003350230] |Everything should work. 
[222003350240] |<b>3. The exit button</b> Add a button to the “finish” frame of your Flash file.
[222003350250] |Change the label to “Exit”.
[222003350260] |Add this code to the button: on (click){     getURL("javascript:updateParent('http://URL/VSTM/done.html'); javaScript:self.close()");} Change “URL” so that it matches the actual path.
[222003350270] |Now, when subjects finish the experiment, they can click on that button.
[222003350280] |It will close the window and change the test_in_progress.html browser window to done.html. 
[222003350290] |<a>VSTM.fla</a> incorporates this. 
[222003350300] |This does not work when you are testing your experiment from your hard drive.
[222003350310] |To see this work, you must be running the experiment through the Web. 
[222003350320] |<b>4. Instructions and debriefing.</b> 
[222003350330] |You will want to add some debriefing information.
[222003350340] |You can do this as part of done.html, or you can do it within VSTM.fla.
[222003350350] |An advantage of doing it within Flash is that you can modify the debriefing depending on the subject’s scores.
[222003350360] |For instance, you could calculate VSTM capacity in this experiment for this subject and display it in the debriefing. 
[222003350370] |You would also want to add instructions.
[222003350380] |This should be fairly straight-forward at this point. 
[222003350390] |As usual, please leave any questions in the comments section.
[222003360010] |<title>Invented Languages</title>
[222003360020] |Those who haven't already seen it might be interested in <a>my article</a> about the role of invented languages in science at <a>ScientificAmerican.com</a>.
[222003370010] |<title>Results: The Best and Worst Puns</title>
[222003370020] |The <a>Puntastic</a> experiment continues to chug along.
[222003370030] |1,376 participants have contributed 59,474 ratings of nearly 2,000 different puns.
[222003370040] |Currently, the most popular pun is "To some, marriage is a word; to others, a sentence." 
[222003370050] |Every participant who has rated that so far has given it the maximum 5 stars. 
[222003370060] |The second most popular is: "The frustrated cannibal threw up his hands." 
[222003370070] |By far the least popular one is: "People adorned with Bogus Deuterium Ingots aroused suspicion.
[222003370080] |Most people said they didn't trust anyone with BDIs." 
[222003370090] |I'm curious whether this is because people really hate this pun, or because they simply didn't get it.
[222003370100] |I actually think it's kind of funny. 
[222003370110] |I'm still collecting data, so if you haven't voted for your favorite puns yet, there is <a>still time</a>.
[222003380010] |<title>Class Notes: Verb Islands</title>
[222003380020] |This is one of several posts based on readings and discussion from a graduate seminar on language acquisition that I have been attending. 
[222003380030] |Modern syntactic theory is complex.
[222003380040] |When I diagrammed sentences in middle school, it looked something like this diagram from <a>A Walk in the WoRds</a> blog: 
[222003380050] |That seemed complicated at the time but is child's play compared with what one finds in many syntax papers.
[222003380060] |For instance, here is a tree from Beatrice Santorini &Anthony Kroch's <a>syntax textbook</a>: 
[222003380070] |Here's another one from Christopher Davis's online class notes at UMass-Amherst: 
[222003380080] |Something that pops out about these trees are the various symbols that don't seem to do anything.
[222003380090] |For instance, saying "smelly dog" involves and adjective and a noun makes sense, but in Davis's tree above, there are extra symbols such as A'.
[222003380100] |The Santorini &Kroch tree makes considerable use of "inflectional phrases," which weren't a part of the sentence diagrams I learned in school. 
[222003380110] |<b>Is grammar really so complex?</b> 
[222003380120] |For a number of years now, <a>Michael Tomasello</a> has been arguing that perhaps linguistic structure is not nearly so complex and does not require nearly so many abstract components (like inflectional phrases), particularly in the case of child speech.
[222003380130] |A lot of the abstraction in linguistic theory is meant to explain how you know what constructions a given word appears in.
[222003380140] |For instance, compare the sentences below: 
[222003380150] |(1) Sarah rolled the orange.
[222003380160] |(2) The orange rolled.
[222003380170] |(3) Sarah ate the orange.
[222003380180] |(4) *The orange ate. 
[222003380190] |The fourth sentence should sound ungrammatical (which is what the asterisk means).
[222003380200] |The question is: how do you know that the verb roll can be used in this way but the verb eat cannot?
[222003380210] |Theories make use of abstract grammatical structure to explain these and other generalizations (the abstractions in the sentence trees at the top of this post are motivated by other considerations, but the idea is the same). 
[222003380220] |Tomasello's point is that in fact young children and even adults typically do not use words in a wide variety of constructions -- therefore, you don't actually need such abstract linguistic theories.
[222003380230] |This is a useful push-back, and the claim has generated a lot of research.
[222003380240] |It is interesting, however, that Tomasello is presenting a theory that explains what people do say, but he is arguing against theories that explain what people can say, which turn out to be quite different things.
[222003380250] |Although nobody is likely to say the sentence in (5), we all know that it is grammatical. 
[222003380260] |(5) We all shall have told the story to the Martians. 
[222003380270] |A complete theory needs to explain that phenomenon, too. 
[222003380280] |<b>More on spoken language</b> 
[222003380290] |Part of Tomasello's argument is that an abstract grammar would predict more variability in the constructions people (particularly children) actually use than are seen in real life.
[222003380300] |Charles Yang has recently argued that this is not the case (<a>see Who's Afraid of George Kingsley Zipf</a>).
[222003380310] |In fact, people are very repetitive.
[222003380320] |Moreover, even if a given word can be used in many constructions, there may be no reason to use all those constructions.
[222003380330] |Sentence (5) was an example of this: the verb tell can be used in the first-person plural future perfect (we all shall have told), but it's hard to imagine many circumstances in which one would need to. 
[222003380340] |Despite the complex math in the paper, Yang's manuscript is more than worth the read.
[222003390010] |<title>How many commercial brands does your kid know?</title>
[222003390020] |A recently published study by Anna McAlister and T. Bettina Cornwell at the University of Michigan reports that smarter kids are more affected by branding and know more brands.
[222003390030] |A number of people are interested in this because the naive prediction might have been that smarter people (including kids) should be less impressionable and less susceptible to marketing, rather than more. 
[222003390040] |The study caught my eye because it is a nice example of a problem that developmental psychologists run into.
[222003390050] |One question a developmental psychologist might be interested in is at what age children acquire a particular ability (like susceptibility to branding).
[222003390060] |This type of research has implications for education, public policy, etc.
[222003390070] |But the age you get depends on the age of the kids you test. 
[222003390080] |It happens to be the case that the children who are most easily recruited into developmental psychology studies tend to be relatively advanced.
[222003390090] |This happens for a number of reasons.
[222003390100] |Developmental labs tend to be in universities, which tend to be surrounded by relatively affluent, well-educated communities.
[222003390110] |Even within a community, not all parents are equally likely to bring their kid into do a study, and those that do seem to often be the sorts of parents that have advanced children.
[222003390120] |Many studies may disproportionately test children of professors and graduate students.
[222003390130] |It's easy to imagine additional reasons. 
[222003390140] |Unfortunately, there is a problem with the direct link to the study.
[222003390150] |It should be the top study on <a>this Google Scholar search</a>.
[222003400010] |<title>Data wants to be free</title>
[222003400020] |<a>It seems</a> that the National Science Foundation will be asking new grant applications to submit a data management plan, apparently including plans for how to make their data available to others. 
[222003400030] |I have mixed feelings about this.
[222003400040] |I certainly approve of high-value data sets being made available.
[222003400050] |I've benefitted a great deal from the wonderful people who put together <a>Penn Tree Bank</a>, <a>VerbNet</a> and similar projects.
[222003400060] |There are now some useful data sets included in libraries for R as well.
[222003400070] |I intend to make the summary data from my pronoun studies available when I publish the associated papers. 
[222003400080] |That said, getting data together in a manner that its interpretable and usable by somebody else is hard.
[222003400090] |However much I document my own data, whenever I have to go back to look at some old data it takes hours if not days to figure out what I'm looking at.
[222003400100] |And I'm the one who created it.
[222003400110] |Fully documenting a data set for someone not associated with the project takes time. 
[222003400120] |Given that NSF will be paying the salaries of the people who spend the time to document the data sets, it's reasonable to ask whether it's cost-effective.
[222003400130] |Just how much of a demand is there for data from other labs?
[222003400140] |I can think of many papers for which I wish I had the original stimuli.
[222003400150] |The number for which I want the original data are much smaller (though there are some for which it would be really useful).
[222003410010] |<title>NSF budget</title>
[222003410020] |It seems the current director of the National Science Foundation <a>thinks it's unlikely</a> NSF will get much of a budget increase this coming year (if any), despite Obama's pledge of an 8% increase.
[222003410030] |Oh well, <a>it was nice while it lasted</a>.
[222003440010] |<title>Why Is Nobody Studying Klingon?</title>
[222003440020] |Doing research <a>for the recent </a><a>Scientific American Mind</a><a> article</a>, I found out that Klingon uses the incredibly rare object-verb-subject (OVS) word order.
[222003440030] |Even though some languages (like Russian) allow relatively free word-order, all languages seem to have a preferred order.
[222003440040] |There are 6 possible.
[222003440050] |The most common are SVO (English), SOV (Japanese), VSO (Classical Arabic).
[222003440060] |The 3 orders that put the object before the subject are relatively rare, with OVS nearly non-existent.
[222003440070] |It does appear occasionally in poetry or other marked uses (The drink drank I), and is claimed to be the dominant word order in at least two extremely rare languages: <a>Guarijio</a> and <a>Hixkaryana</a>.
[222003440080] |Given the degree of debate over how to correctly characterize syntax in well-studied languages like English, I'm always maintain some skepticism about rare, poorly-studied languages (and the sad truth is that all languages are poorly-studied when compared to English). 
[222003440090] |In any case, if one wanted to study the acquisition of Guarijio or Hixkaryana, one would need a decent travel budget and some infrastructure.
[222003440100] |Klingon is spoken closer to home.
[222003440110] |Yet I couldn't find any papers in Google Scholar looking at the acquisition of Klingon, even from a sociological perspective.
[222003440120] |This seems under-studied.
[222003450010] |<title>Lie detection</title>
[222003450020] |A few pioneering lawyers have been <a>attempting to use fMRI-based lie detection tests in court</a>.
[222003450030] |I don't have any broad numbers, but it seems most neuroimagers I talk to are deeply skeptical of such tests, at least at the current stage of technology (and whether such technology can ever catch pathological liars is yet another question). 
[222003450040] |At a recent talk at Harvard, <a>Michael Gazzaniga</a> related the following argument from a colleague on the law end of things: whether fMRI-based lie detection is "good enough" is not a scientific question but a legal one.
[222003450050] |After all, the law allows all kinds of scientifically-suspect "evidence" into the courtroom as is (eye-<a>witness testimony</a>, f<a>ingerprinting</a>, etc.).
[222003450060] |Present all data (along with information about how reliable it is) to the jury and let the jury sort it out. 
[222003450070] |That's one conclusion that could be drawn.
[222003450080] |Another is that perhaps it's time to step back and come up with a broad policy for how evidence is introduced into the legal system.
[222003460010] |<title>The military is making telepathic helmets. Sign me up.</title>
[222003460020] |It appears researchers at UC-Irvine, University of Maryland and Carnegie Mellon have a DOD grant to investigate the development of "<a>though helmets</a>": 
[222003460030] |The devices would harness a person´s brain waves and transmit them as radio waves, where they would be translated into words in the headphones of other soldiers.
[222003460040] |I need one of these.
[222003460050] |<b>Don't blink -- I'm reading your thoughts</b>
[222003460060] |The proposed technology (which they don't expect to be ready for a decade or two) relies on EEG technology -- that is, measuring brain waves.
[222003460070] |As it happens, this is a method I use to run experiments.
[222003460080] |It has its limitations.
[222003460090] |First off, it only really works if people are sitting still and not moving their eyes.
[222003460100] |EEG measures electrical activity.
[222003460110] |Ideally, it measure the brain's electrical activity, but <a>muscles also produce electrical activity</a> and the effects are hundreds of times larger than brain effects.
[222003460120] |People are working on snazzy new algorithms to factor out the thunderclaps of eye blinks (the huge mountains in <a>this picture</a> are individual blinks).
[222003460130] |<b><b>One-of-a-kind</b><b> </b>Another problem is individual variation.
[222003460140] |While blinks are very easy to see in the raw waveforms, thoughts are hard.
[222003460150] |For instance, one of the best known EEG effects in language is the N400 -- a broadly-distributed negative deflection around 400 milliseconds after the participant sees a word.
[222003460160] |Unfortunately, the N400 is so small relative to all the noise in the signal that it's hard to see on a single trial.
[222003460170] |In a typical experiment, each participant sees 30-40 words (or more), and we average across those trials.
[222003460180] |Even then, each individual person's N400 looks very different, so we usually have to average across at least a dozen different people to get a good signal. 
[222003460190] |Thoughts The bulk of EEG research employs a violation paradigm.
[222003460200] |We measure the brain's activity when something unexpected happens.
[222003460210] |For instance, a psycholinguist like myself might compare the two following sentences: (1) Dog bit the man.(2) The man bit the dog. 
[222003460220] |The second sentence is surprising relative to the first, and you can typically see a reasonably large effect in the brain waves (modulo the caveats above).
[222003460230] |Part of the reason we use violation paradigms is they produce large effects.
[222003460240] |Trying to compare two perfectly normal sentences is much, much harder. 
[222003460250] |Other typical effects people can find are differences between function words (e.g., prepositions) and content words (e.g., nouns) -- though, again, this is hard to see on a trial-by-trial basis. 
[222003460260] |Get me one There are many, many other obstacles in the way of a mind-reading EEG helmet.
[222003460270] |That isn't to say that I don't think such a helmet will be built eventually, or that the government is wasting it's money.
[222003460280] |Technology constantly improves, and setting an on-the-face absurd goal can be excellent motivation. 
[222003460290] |But I wouldn't start saving up to buy one of your own just yet.
[222003460300] |Though I'd love one.
[222003460310] |A helmet that worked that well would make my research go much, much faster.</b>
[222003470010] |<title>Your age</title>
[222003470020] |Who participates in Web-based experiments?
[222003470030] |I recently analyzed preliminary results from about 4,500 participants in <a>Keeping Things In Mind</a>, an experiment I'm running in collaboration with a colleague and friend at <a>TestMyBrain.org</a>. 
[222003470040] |One of the things we're interested in is the age of people who participate.
[222003470050] |Here is the breakdown: 
[222003470060] |Not surprisingly, the bulk are college age (particularly freshmen).
[222003470070] |There are still a sizable number in their 30s, 40s and early 50s, but by the 60s it drops off considerably. 
[222003470080] |And then there are the few jokers who claim to be 3 or 100. 
[222003470090] |This is pretty similar to the breakdown I usually see at <a>GamesWithWords.org</a>, except that I usually have fewer tweens and more people in their 60s.
[222003470100] |But the mode is usually 18. 
[222003470110] |What this means for the experiment is that people in their 50s on up are woefully underrepresented.
[222003470120] |We're <a>continuing to run the experiment</a> in the hopes that more will participate.
[222003480010] |<title>Clarity</title>
[222003480020] |Geoffrey Pullum at Language Log, in the course of <a>a discussion of the incredibly flexible verb see</a>, writes 
[222003480030] |I wonder how and why human languages seem to be so completely content with the wild and multifarious <a>polysemy</a> and ambiguity that afflicts them.
[222003480040] |The people who think clarity involves lack of ambiguity, so we have to strive to eliminate all multiple meanings and should never let a word develop a new sense... they simply don't get it about how language works, do they?
[222003490010] |<title>Revetro responds</title>
[222003490020] |Earlier this month I blogged about a study supposedly produced by Revetro on <a>texting during sex</a>.
[222003490030] |The main point of the post was that researchers have to be careful about ensuring data quality (e.g., are the participants actually paying attention to the questions?).
[222003490040] |I also remarked that I had been unable to find the original article. 
[222003490050] |Jennifer Jacobson at Revetro very kindly emailed me several days ago in order to point me to <a>the original study</a>.
[222003490060] |The survey question under discussion can be found in the "We interrupt this dinner" section. 
[222003490070] |So at least the study exists.
[222003490080] |I'm hoping to find out more about the methods they used.
[222003500010] |<title>You are what you say</title>
[222003500020] |I recently received an email forward about <a>AnalyzeWords.com</a>.
[222003500030] |According to its promoters 
[222003500040] |AnalyzeWords help reveal your personality by looking at how you use words.
[222003500050] |It is based on good scientific research connecting word use to who people are.
[222003500060] |The way the site works is that you enter in someone's Twitter handle and the site analyzes their tweets. 
[222003500070] |The forward included the following comment from someone from whom, indirectly, I got the email, saying "So far it says everyone I've looked at (people, journals, etc) is depressed, except for an account someone set up to chronicle his battle with cancer, which it classified as 'very upbeat'."
[222003500080] |I tried a handle or two myself and got similar results. 
[222003500090] |One possible conclusion is that everyone -- or, at least, everyone who uses Twitter -- is depressed.
[222003500100] |Or the theory behind the website doesn't actually work.
[222003500110] |I found a possible hint in favor of the latter hypothesis on AnalyzeWords' "<a>The Science Behind AnalyzeWords</a>" page: 
[222003500120] |Across dozens of studies, junk words [closed-class words like prepositions and pronouns] have proven to be powerful markers of peoples [sic] psychological states.
[222003500130] |When individuals use the word I, for example, they are briefly paying attention to themselves.
[222003500140] |People experiencing high levels of physical or mental pain automatically orient towards themselves and begin using I-words at higher rates.
[222003500150] |I-use, then, can reflect signs of depression, stress or insecurity.
[222003500160] |Perhaps.
[222003500170] |Or perhaps they're using Twitter to talk about themselves and their latest experiences.
[222003510010] |<title>The Job Search</title>
[222003510020] |The Prodigal Academic has a <a>fascinating post</a> on how candidates for job academic job searches are chosen.
[222003510030] |I've never been through a job search (on either end), so I have no real comment.
[222003510040] |The closest I came was filing job applications for two searches in the History &Religion department at Oberlin.
[222003510050] |What impressed me then, as now, is how many there were.
[222003530010] |<title>Lie detection: Part 2</title>
[222003530020] |I <a>wrote recently</a> about whether fMRI should be used for lie detection in court.
[222003530030] |US Magistrate Judge Tu Pham says "<a>no</a>".
[222003530040] |Science reports:
[222003530050] |But while Judge Pham agreed that the technique had been subject to testing and peer review, it flunked on the other two points suggested by the Supreme Court to weigh cases like this one: the test of proven accuracy and general acceptance by scientists.
[222003530060] |What I find interesting about this argument, as noted in my previous post, is that it's not clear that commonly-accepted "evidence" passes those tests: <a>fingerprinting</a> and <a>eyewitness testimony</a> are two good examples.
[222003540010] |<title>Dothraki -- a response</title>
[222003540020] |The Language Creation Society has officially responded to <a>my open letter</a> requesting that they embed some useful experiments in Dothraki, a language they are creating for a new HBO show.
[222003540030] |You can read <a>the response</a> at Scientific American. 
[222003540040] |This formal response follows a series of informal emails between myself and both David Peterson (the author of the response) and Sai Emrys (the LCS president).
[222003540050] |It was a fun conversation, and while they're not taking me up on my suggestion -- at least not for this language -- I did learn a great deal from them, some of which makes it into their letter, which I recommend reading.
[222003550010] |<title>Overheard: The Prodigal Academic</title>
[222003550020] |I recently started reading The Prodigal Academic, a blog by a professor recently returned to academia after 7 years away.
[222003550030] |Lately she's written a number of useful posts about academia as a career.
[222003550040] |See these posts on spousal <a>hiring</a>, <a>search committee</a> dynamics, <a>interviewing</a> for tenure-track jobs, and <a>women in science</a>.
[222003560010] |<title>Cognitive Science, April 2010</title>
[222003560020] |This week I was tasked by the lab to check the last years' worth (or so) of issues of <a>Cognitive Science</a> and see what papers might be of interest to folks in the lab (other people are covering other journals).
[222003560030] |There of course many good papers not on the list below; I focused largely on the psycholinguistics articles.
[222003560040] |There are a lot of articles, so I'm going to be breaking up issues into separate posts. 
[222003560050] |Fair warning: my discussion of these articles is brief and so somewhat technical. 
[222003560060] |<b>April 2010</b>
[222003560070] |<a>Szymanik &Zajenkowski.
[222003560080] |Comprehension of simple quantifiers: empirical evaluation of a computational model.</a> 
[222003560090] |Different quantifiers seem to require different amounts of computation.
[222003560100] |Formal logic suggests that checking the truth of Some of the cars are blue simply requires checking whether at least one car is blue (or failing to find any).
[222003560110] |Most of the cars are blue probably requires something like finding out how many cars is 1/2 of the cars and checking to see if at least more than  that are.
[222003560120] |That's harder. 
[222003560130] |S&Z had people evaluate the truth value of sentences like those in the examples.
[222003560140] |People were slower for the "harder" quantifiers.
[222003560150] |This suggests people are actually running through something like the formal math theorists use to describe quantifiers. 
[222003560160] |The only odd thing about the results is a ton of research (e.g., <a>Bott &Noveck</a>) has suggested that evaluating sentences with some can be very slow, presumably because it involves a scalar implicature, whereas in the study some was one of the fastest quantifiers.
[222003560170] |This either suggests that for some reason people weren't computing implicatures in their study or that the other quantifiers were really slow (or that Polish, the language they used, is just different). 
[222003560180] |<a>Matthews &Bannard, Children's production of unfamiliar word sequences is predicted by positional variability and latent classes in a large sample of child-directed speech.</a> 
[222003560190] |Two- and three-year olds were asked to repeat back four-word sequences.
[222003560200] |Several things were varied, such as how predictable the final word was based on the first three in the sequence sequence (e.g., jelly probably commonly appears after peanut butter and ... ) and whether the words that do commonly appear as the fourth word in such a sequence are semantically related (e.g., pretty much everything following I drive a ... is going to be some kind of vehicle). 
[222003560210] |Importantly, in the actual sequences presented to the children, the final word was one that hardly ever appears in that sequence (e.g., I hate green boxes).
[222003560220] |Kids were better at repeating the sequences when (1) entropy on the 4th word was high (e.g., many different words commonly follow the first three in the sequence, as in I drive a rather than peanut butter and), and when most words that typically appear in that 4th position are semantically related (I drive a truck/car/bus/Toyota/Ford).  
[222003560230] |The authors (tentatively) suggest that such results are predicted by theories on which young children's grammars are very item-specific, involving many narrow sentence templates (I drive a + Vehicle), rather than theories on which young children's grammars involve broad abstract categories (e.g., Noun + Verb). 
[222003560240] |However, my labmate and collaborator Timothy O'Donnell has been working on a computational model that involves abstract grammatical categories but nonetheless stores high-frequency constructions (which has been allowed but not specifically explained on many grammatical theories such as Pinker's Words &Rules theory that are the traditional alternatives to item-based theories).
[222003560250] |One consequence of his model is that if a particular construction appears very frequently with little variation (peanut butter and jelly; 653,000 <a>hits on</a> Google), the model finds slight alternatives to that construction (peanut butter and toast; 120,000 <a>hits on</a> Google) extremely unlikely. 
[222003560260] |<a>Casasanto, Fotakopoulou &Boroditsky.
[222003560270] |Space and time in the child's mind: Evidence for a cross-dimensional asymmetry.</a> 
[222003560280] |4-5 year-olds and 9-10 year-olds watched movies of two animals traveling along parallel paths for different distances or durations and judged the which one went longer temporally or spatially.
[222003560290] |As has been previously shown in adults, the children's judgments of temporal length were affected by spatial length (e.g., if animal A went farther than B but in a shorter amount of time, children sometimes erroneously said A took the most time) more than judgments of spatial length were affected by temporal length (e.g., if animal A went farther than B but in less time, children were not as likely to be confused about when animal went the farthest). 
[222003560300] |One obvious confound, which the authors consider, is that the stimuli stayed on the screen until the children responded, which meant that information about physical distance was available at response time, but children had to remember the duration information.
[222003560310] |The authors point to a previous study with adults that controlled for this confound and got the same results, but they have not yet run that version with children (since I haven't read the study they refer to and the method isn't described, I can't really comment). 
[222003560320] |These results are taken as evidence of a theory on which our notion of time is dependent on our representations of space, but not vice versa. 
[222003560330] |<a>Fay, Garrod, Roberts &Swoboda.
[222003560340] |The interactive evolution of human communication systems.</a> 
[222003560350] |People played what amounted to repeated games of <a>Pictionary</a>.
[222003560360] |The games were played over and over with the same words, and the question was how the pictorial representations changed over repeated games.
[222003560370] |Participants were assigned either in pairs or communities of 8.
[222003560380] |The pairs played against each other only.
[222003560390] |In the communities, games were still played in pairs, but each person played first against one member of the community, then against another, and so on until they had played with everyone. 
[222003560400] |The people in the pairs condition rapidly created abstract visual symbols for the different target words, as has happened in previous research.
[222003560410] |What was interesting was that in the communities condition, participants also created similarly abstract symbols that were rapidly shared throughout the community, such that participants who had never played against one another could communicate with abstract symbols that they had each learned from others. 
[222003560420] |The study is meant to be a model of vocabulary development and dispersal, and it certainly made for a good read (I've been a fan of Garrod's work in other contexts as well).
[222003560430] |I don't know much about theories of language evolution, so it's difficult for me to say what the theoretical impact of this work is.
[222003560440] |One obvious question is whether it matters that people in a single community know they are in a single community.
[222003560450] |That is, did they continue to use the abstract symbols they'd learned because they reasonably thought the other person might know it, or was it instinct after having used that symbol many times?
[222003570010] |<title>Cognitive Science, January 2010</title>
[222003570020] |In my <a>continuing series</a> on the past year in Cognitive Science: January, 2010. 
[222003570030] |Once again, the discussion of some of these papers will be technical. 
[222003570040] |<b>January</b>
[222003570050] |<a>Lee &Sarnecka.
[222003570060] |A model of knower-level behavior in number concept development.</a> 
[222003570070] |Children <a>learn the full meanings of number words slowly</a>, one word at a time.
[222003570080] |The authors present a Bayesian model of number word acquisition -- or, more specifically, of performance on the famous <a>Give-A-Number task</a>.
[222003570090] |The model assumes that each child has a certain baseline preference to give certain numbers of items more than others.
[222003570100] |It also assumes that the child knows certain number words and not others.
[222003570110] |If the child, say, knows one and two, the child will give that number of items when asked and not when asked about a different number word (e.g., three), even if the child doesn't know what that other number word means. 
[222003570120] |The model was then fed data on the actual performance of a set of actual children and estimates what words the child knows and what the child's baseline preferences are.
[222003570130] |The model learned that children prefer to either give a handful of items or all the available items, which accords well with what has been seen over the years.
[222003570140] |It also seemed to do a reasonable job of doing several other things. 
[222003570150] |None of this was necessarily surprising, in the sense that the model modeled well-known data correctly.
[222003570160] |That said, psychological theories are often complex.
[222003570170] |Theorists (often) state them in loose terms and then make claims about what predictions the theory makes in terms of behavior in different tasks.
[222003570180] |Without specifying the theory in a formal model, though, it's not always clear that those are in fact the predictions the theory makes.
[222003570190] |This paper represents, among things, an attempt to take a well-known theory and show that it does in fact account for the observed data.
[222003570200] |To the extent it gets things wrong, the model presents a starting point for further refinement. 
[222003570210] |There has been a movement in some quarters to make and test more explicit models.
[222003570220] |This is undoubtedly a good thing.
[222003570230] |The question is whether there are many behaviors that we understand sufficiently well to produce reasonable models ... that aren't so simplistic that the formal model itself doesn't really tell us anything we don't know.
[222003570240] |That seems to be a point one could argue.
[222003570250] |One thing I like about this particular model is that the authors attempt to capture fine-grained aspects of individual subjects' performances, which is something we ultimately want to be able to do. 
[222003570260] |<a>Estigarribia.
[222003570270] |Facilitation by variation: Right-to-left learning of English yes/no questions</a> 
[222003570280] |The syntax of questions have played a key role in the development of modern linguistics.
[222003570290] |In particular, a great deal of ink has been spilled about auxiliary inversion.
[222003570300] |Compare That <b>is</b> a soccer ball with <b>Is </b>that a soccer ball.
[222003570310] |Well-known theories of English posit that the auxiliary is is generated in normal declarative position (that is...) and must be moved to the front of the sentence to form a question (is that...). 
[222003570320] |Estigarribia argues that many theories have assumed parents model auxiliary-inverted questions for their children.
[222003570330] |A (smallish) corpus analysis reveals that in fact ~20% of parental yes/no questions with auxiliaries are non-auxiliary-initial (that is a soccer ball?).
[222003570340] |Of all yes/no questions, canonical auxiliary-first questions make up less than half, with sentence fragments being quite common (soccer ball?). 
[222003570350] |Again looking at the corpus of 6 young children, Estigarribia finds that the children begin by producing the simplest, fragment questions (a soccer ball?).
[222003570360] |Next, they begin producing what Estigarribia calls subject-predicate questions (that a soccer ball?).
[222003570370] |Full-on auxiliary-inverted questions appear relatively late (is that a soccer ball).
[222003570380] |Estigarribia finds this consistent with a learning mechanism in which children learn the ends of sentences better than the beginnings of sentences, similar to the MOSAIC model. 
[222003570390] |One limitation is that children have difficulty producing long sentences, and the data are consistent with children producing shorter sentences first and eventually progressively-longer sentences.
[222003570400] |Estigarribia shows that he finds the same order of acquisition even in children who have somewhat longer MLUs at the beginning of the study (that is, produce longer sentences), but one can still worry.
[222003570410] |The fact that children selectively produce the ends of the sentences rather than the beginning could be due to the fact that the end of a question (a soccer ball?) is a lot more informative than the beginning (is that a?). 
[222003570420] |It might be somewhat more impressive if children produce non-inverted questions (that is a soccer ball?) before inverted questions, but Estigarribia does not analyze those types of sentences.
[222003570430] |What I find most compelling about this study is in fact the adult data.
[222003570440] |As Estigarribia points out, we don't want to think of language acquisition as a process in which children ultimately eliminate non-canonical questions (that is, those without inverted auxiliaries), since in fact adults produce many such sentences. 
[222003570450] |<a>Nakatani &Gibson.
[222003570460] |An on-line study of Japanese nesting complexity.</a> 
[222003570470] |Mary met the senator who attacked the reporter who ignored the president is easier to understand that The reporter who the senator who Mary met attacked ignored the president, even though the latter sentence is grammatical (of sorts) and means the same thing.
[222003570480] |Why this is the case has been a focus of study in psycholinguistics for many years. 
[222003570490] |The authors lay out a couple hypotheses.
[222003570500] |On one, the second sentence is harder to interpret because the relevant nouns are far from the verbs, making integrating ignored and the reporter harder to integrate.
[222003570510] |On other hypotheses, all the nested relative clauses (who...) generate expectations about what verbs are coming up.
[222003570520] |The more expectations, the more has to be kept in memory, and the harder the sentence is. 
[222003570530] |These hypotheses (and a similar surprisal hypothesis) are tested using the self-paced reading methodology in Japanese, a language with a few nice properties like relatively free word order, which makes controlling the stimuli slightly easier than it is in English.
[222003570540] |The results ultimately support the expectancy hypotheses over the retrieval hypotheses. 
[222003570550] |One of the interesting things about this paper is that one well-known retrieval hypothesis is actually Gibson's.
[222003570560] |So is one of the expectancy hypotheses, which he developed after he (apparently) decided the original theory was probably wrong.
[222003570570] |The willingness to abandon a cherished theoretical position in the face of new evidence is a trait more prized than seen in academia, and it's something to be admired -- and something very typical of Gibson. 
[222003570580] |<a>Mirman, Strauss, Dixon &Magnuson.
[222003570590] |Effect of representational distance between meanings on recognition of ambiguous spoken words.</a> 
[222003570600] |The authors looked at word recognition using two different paradigms (lexical decision and eye-tracking).
[222003570610] |All the words could be nouns.
[222003570620] |Some had only strong noun meaning (acorn, lobster).
[222003570630] |Some were homophones with two common noun meanings (chest -- chest of drawers or person's chest) and some were homophones with a common noun and a common verb meaning (bark -- the dog barked or the tree's bark). 
[222003570640] |Participants were fasted to interpret the unambiguous words (acorn, lobster), next fastest at recognizing the noun-verb words (bark) and slowest at the noun-noun words (chest).
[222003570650] |They take this in the context of previous research that has shown that words with two closely related meanings are faster to interpret that words with two very different meanings.
[222003570660] |In this study, the semantic relatedness of the two meanings for the noun-verb homophones were no closer than that of the noun-noun homophones.
[222003570670] |So the authors suggest that syntactic distance matters as well -- two meanings of the same syntactic type (e.g., noun) interfere with one another more than two meanings of different types (e.g., noun-verb). 
[222003570680] |An alternative explanation of these data is one of priming.
[222003570690] |2/3 of the stimuli in this study were unambiguously nouns.
[222003570700] |This may have primed the noun meanings of the noun-verb homophones and helped automatically suppress the verb meaning.
[222003570710] |Thus, participants processed the noun-verb homophones more like unambiguous, non-homophonic words.
[222003570720] |The way to test this, of course, would be to run a similar study with unambiguous verbs, verb-verb homophones, and the same noun-verb homophones.
[222003580010] |<title>Video games, rotted brains, and book reviews</title>
[222003580020] |Jonah Lehrer <a>has an extended discussion</a> of his review of The Shallows, a new book claiming that the Internet is bad for our brains.
[222003580030] |Lehrer is skeptical, pointing out that worries about new technology are as old as time (Socrates thought books would make people stupid, too).
[222003580040] |I am skeptical as well, but I'm also skeptical of (parts of) Lehrer's arguments.
[222003580050] |The crux of the argument is as follows:
[222003580060] |I think it's far too soon to be drawing firm conclusions about the negative effects of the web.
[222003580070] |Furthermore, as I note in the review, the majority of experiments that have looked directly at the effects of the internet, video games and online social networking have actually found significant cognitive benefits.
[222003580080] |That, so far as it goes, is reasonable.
[222003580090] |My objection is to some of the evidence given:
[222003580100] |A 2009 study by neuroscientists at the University of California, Los Angeles, found that performing Google searches led to increased activity in the dorsolateral prefrontal cortex, at least when compared with reading a "book-like text."
[222003580110] |Interestingly, this brain area underlies the precise talents, like selective attention and deliberate analysis, that Carr says have vanished in the age of the Internet.
[222003580120] |Google, in other words, isn't making us stupid -- it's exercising the very mental muscles that make us smarter.
[222003580130] |This cuts several ways.
[222003580140] |Extra activation of a region in an fMRI experiment is interpreted different ways by different researchers.
[222003580150] |It could be evidence of extra specialization ... or evidence that the brain network in question is damaged and so needs to work extra hard.
[222003580160] |Lehrer is at least partially aware of this problem:
[222003580170] |Now these studies are all imperfect and provisional.
[222003580180] |(For one thing, it's not easy to play with Google while lying still in a brain scanner.)
[222003580190] |This is the line I have a particular issue with.
[222003580200] |If the question is whether extra Internet use makes people stupid, why on Earth would anyone need to use a $600/hr MRI machine to answer that question?
[222003580210] |We have loads of cheap psychometric tests of cognition.
[222003580220] |All methodologies have their place, and a behavior question is most easily answered with behavioral methods.
[222003580230] |MRI is far more limited. 
[222003580240] |Lehrer's discussion of the 2009 study above underscores this point: the interpretation of the brain images rests on our understanding of what behaviors the dorsolateral prefrontal cortex has shown up with in other studies.
[222003580250] |The logic is: A correlates with B correlates with C, thus A correlates with C.
[222003580260] |This is, as any logician will tell you, an unsound conclusion.
[222003580270] |When you add that using MRI can cost ten thousand dollars for a single experiment, it's a very expensive middleman! 
[222003580280] |Which isn't to say that MRI is useless or such studies are a waste of time.
[222003580290] |MRI is particularly helpful in understanding how the brain gives rise to various types of behavior, and it's sometimes helpful for analyzing behavior that we can't directly see.
[222003580300] |Neither applies here.
[222003580310] |If the Internet makes us dumb in a way only detectable with super-modern equipment, I think we can breath easy and ignore the problem.
[222003580320] |What we care about is whether people in fact are more easily distracted, have worse memory, etc.
[222003580330] |That doesn't require any special technology -- even Socrates could run that experiment. 
[222003580340] |**** Lehrer does discuss a number of good behavioral experiments.
[222003580350] |Despite my peevishness over the "Google in the scanner" line, the review is more than worth reading.
[222003590010] |<title>Cognitive Science, March 2010</title>
[222003590020] |In my <a>continuing series</a> on the past year in Cognitive Science: March, 2010. 
[222003590030] |Once again, the discussion of some of these papers will be technical. 
[222003590040] |<b>March</b>
[222003590050] |<a>Baroni, Murphy, Barbu, Poesio.
[222003590060] |Strudel: A corpus-based semantic model based on properties and types.</a> 
[222003590070] |You are who your friends are.
[222003590080] |A number of computational linguists have been interested in just how much you can learn about a word based on the other words it tends appear with.
[222003590090] |Interestingly, if you take a word (e.g., dog) and look at the words it tends to co-occur with (e.g., cat), those other words often describe properties or synonyms of the target word.
[222003590100] |A number of researchers have suggested that this might be part of how we learn the meanings of words. 
[222003590110] |Baroni et al. are sympathetic to that literature, but they point out that such models are only learned that dog and cat are somehow related.
[222003590120] |So they don't actually tell you what the word dog means.
[222003590130] |Moreover, Dog is also related to leash, but not in the same way it's related to cat, which is something those models ignore.
[222003590140] |Their paper covers a new model, Strudel, which attempts to close some of the gap. 
[222003590150] |The model also keeps track of what words co-occur with a target word.
[222003590160] |It additionally tracks how those words are related (e.g., dogs and cats is considered to be different from dogs chase cats).
[222003590170] |The more different types of constructions that connect the target word and a given "friend", the more important that friend is thought to be. 
[222003590180] |This model ends up doing a better job than some older models at finding semantic associates of target words.
[222003590190] |It also can cluster different words (e.g., apple, banana, dog, cat) into categories (fruit, animal) with some success.
[222003590200] |Moreover, with some additional statistical tricks, they were able to clump the various "friends" into different groups based on the type of constructions they appear in. Properties, for instance, often appear in constructions involving X has Y. Conceptually-similar words appear in other types of constructions (e.g., X is like Y). 
[222003590210] |This presents some clear advantages over previous attempts, but it has some of the same limitations as well.
[222003590220] |The model discovers different types of features of a target word (properties, conceptually-similar words, etc.), but the label "property" has to be assigned by the researchers.
[222003590230] |The model doesn't know that has four legs is a property of dog and that like to bark is not -- it only knows that the two facts are of different sorts. 
[222003590240] |<a>Perruchet &Tillman.
[222003590250] |Exploiting multiple sources of information in learning an artificial language: human data and modeling. </a> 
[222003590260] |Over the last 15 years, a number of researchers have looked at statistically-based word segmentation.
[222003590270] |After listening to a few minutes of speech in an unknown language, people can guess which sequences of phonemes are more likely to be words in that language. 
[222003590280] |It turns out that some sequences of phonemes just sound more like words, independent of any learning.
[222003590290] |The authors check to see whether that matters.
[222003590300] |Participants were assigned to learn one of two languages: a language in which half of the words a priori sounded like words, and a language in which half the words a priori sounded particularly not like words.
[222003590310] |Not only did participants do better in the first condition on the words that sound like words, they did better on the "normal" words, too -- even though those were the same as the "normal" words in the second condition.
[222003590320] |The authors argue that this is consistent with the idea that already knowing some words helps you identify other words. 
[222003590330] |They also find that the fact that some words a priori sound more like they are words is easy to implement in their previously-proposed PARSER model, which then produces data somewhat like the human data from the experiment. 
[222003590340] |<a>Gildea &Temperley.
[222003590350] |Do grammars minimize dependency length?</a> 
[222003590360] |Words in a sentence are dependent on other words.
[222003590370] |In secondary school, we usually used the term "modify" rather than "depend on."
[222003590380] |So in The angry butcher yelled at the troublesome child, "the angry butcher" and "at the troublesome child" both modify/depend on yelled.
[222003590390] |Similarly, "the angry" modifies/depends on butcher.
[222003590400] |Etc. 
[222003590410] |This paper explores the hypothesis that people try to keep words close to the words they depend on.
[222003590420] |They worked through the <a>Wall Street Journal corpus</a> and calculated both what the actual dependency lengths were in each sentence (for each word in the sentence, count all the words that are between a given word and the word it depends on, and sum) and also what the shortest possible dependency length would be.
[222003590430] |They found that actual dependency lengths were actually much  closer to the optimum in both the WSJ corpus and the <a>Brown corpus</a> than would be expected by chance.
[222003590440] |However, when they looked at two corpora in German, while dependency lengths were shorter than would be expected by random, the effect was noticeably smaller.
[222003590450] |The authors speculate this is because German has relatively free word order, because German has some verb-final constructions, or some other reason or any combination of those reasons. 
[222003590460] |<a>Mueller, Bahlmann &Friederici.
[222003590470] |Learnability of embedded syntactic structures depends on prosodic cues. </a> 
[222003590480] |Center-embedded structures are hard to process and also difficult to teach people in artificial grammar learning studies that don't provide feedback.
[222003590490] |The authors exposed participants to A1A2B1B2 structures with or without prosodic cues.
[222003590500] |Participants largely failed to learn the grammar without prosodic cues.
[222003590510] |However, if a falling contour divided each 4-syllable phrase (A1A2B1B2) from each other, participants learned much more.
[222003590520] |They did even better if a pause was added in addition to the falling contour between 4-syllable phrases.
[222003590530] |Adding an additional pause between the As and Bs (in order to accentuate the difference between As and Bs) did not provide any additional benefit.
[222003600010] |<title>Ask a stupid question, get a stupid answer</title>
[222003600020] |<a>This morning</a>, Slate is running a bizarre feature on transportation.
[222003600030] |Cities and transportation are in crisis, we're told, and we need new ideas to solve problems of traffic, efficiency, greenhouse gas emissions, etc.
[222003600040] |We need new ideas to solve the problems of our cities "and so we need new visions for the city."
[222003600050] |Alternatively, we could just build what so many cities around the world already have.
[222003600060] |The article mentions free Wi-Fi in buses (<a>don't we already have that?</a>).
[222003600070] |A few months ago, I had lunch with a professor from a university in Switzerland.
[222003600080] |He had recently moved to that university from another university a good 1-2 hours drive away.
[222003600090] |With his kids in school and a wife with a job, he didn't want to move them.
[222003600100] |Luckily, there was a high speed train with working Wi-Fi (take that, <a>Bolt Bus</a>!) that only took about an hour each way, so he was commuting in, working on the train both directions. 
[222003600110] |In Hong Kong, you can check your luggage in at a station in the city center up to 24 hours before your flight.
[222003600120] |You only have to hop on the express train to the airport just before your flight, so you can enjoy downtown without your luggage on the day you head out of the city.
[222003600130] |Also downtown in Hong Kong, incidentally, they've built a pedestrian street one floor above the vehicular street, so that pedestrians can walk to and from offices and shops without getting in the way of the traffic -- safer and more convenient. 
[222003600140] |Anyone who has spent much time traveling abroad knows that the US transportation system is a good half century (or more) behind the more developed parts of the world.
[222003600150] |Even where our transportation is technologically on par (e.g., of places I've lived, Spain and Russia), it often works better, runs faster and has more geographic coverage.
[222003600160] |There are many ideas out there -- most of them not new -- that work very well in other countries.
[222003600170] |So any real discussion would not be about finding clever new ideas, but figuring out how to implement them in the US. ---- This brings up a different question: what happened to Slate?
[222003600180] |Some years ago, it was the first place I turned for news, but the quality has steadily declined.
[222003600190] |Some of it is just attrition (e.g., the sublime and irreplaceable movie critic <a>David Edelstein</a> was "replaced" by Dana Stevens).
[222003600200] |The science coverage has been turned over largely to <a>William Saletan</a> who -- bless his heart -- tried very hard, but simply doesn't know enough about science to understand what he's writing about, leading to articles that are either shallow or just wrong (see <a>here</a> and <a>here</a>).
[222003600210] |Not that shallow science writing is <a>a problem specific to Slate</a>. 
[222003600220] |Slate's travel writing used to be incredible, written by interesting folks with deep, deep knowledge of the places they were visiting.
[222003600230] |So several years ago, when I pitched a piece to Slate about the Trans-Siberian railway, I assumed I never got a response because despite a couple years in Russia, I wasn't up to their level of expertise.
[222003600240] |Recently, though, Slate's ad critic (generally one of my favorite writers) posted an article about <a>his trip on the Trans-Siberian</a>, written with detailed horror of life in Russia (which he can only observe from a distance, since he's afraid to ride in platskart, which he describes as "P.O.W." camp, but which is more accurately called "a party which begins in Moscow and ends 7 days later in Vladivostok").
[222003600250] |Though, in Stevenson's defense, the article wasn't nearly so bad nor so clueless as<a> Daniel Gross's description of his visit to Japan</a>, during which he discovered (wow!) that the Japanese really like things written in English. 
[222003600260] |Seriously, Slate -- I expect better.
[222003610010] |<title>Need data by morning</title>
[222003610020] |In preparing a talk for a workshop I've organized tomorrow, I realized there was one simple experiment that would tie several pieces together neatly.
[222003610030] |Unfortunately, I hadn't run it.
[222003610040] |But I figured, hey, I can get data quick on Amazon Mechanical Turk. 
[222003610050] |Turk let me down.
[222003610060] |I don't know why, but today there aren't a lot of fish biting.
[222003610070] |So I turn to my usual, pre-Turk subject pool: you.
[222003610080] |<a>The experiment</a> takes 1-2 minutes -- it's really just 3 questions.
[222003610090] |As an added inducement to get people to run the experiment now, I'll be posting the results later this week or early next week.
[222003620010] |<title>Overnight data on lying and bragging</title>
[222003620020] |Many thanks to all those who responded to <a>my call for data</a> last week.
[222003620030] |By midnight, I had enough data to be confident of the results, and the results were beautiful.
[222003620040] |I would have posted about them here on Friday, but in the lead-up to this presentation, I did so much typing I burned out my wrists and have been taking a much-needed computer break. 
[222003620050] |The study looked at the interpretation of the word some.
[222003620060] |Under some conditions, people interpret some as meaning some but not all, but other times, it means simply not none.
[222003620070] |For instance compared John did some of his homework with If you eat some of your green beans, you can have dessert. 
[222003620080] |Changing some to some-but-not-all doesn't change the meaning of the first sentence, but (for most people) changes the interpretation of the second. 
[222003620090] |This phenomenon, called "scalar implicature" is one of the hottest topics in <a>pragmatics</a> -- a subdivision of linguistic study.
[222003620100] |The reasons for this are complex -- partly it's because Ira Noveck and his colleagues turned out a series of fascinating studies capturing a lot of people's attention.
[222003620110] |Partly it's because scalar implicature is a relatively easily-studied test case for several prominent theories.
[222003620120] |Partly it's other reasons. 
[222003620130] |<b>Shades of meaning</b> 
[222003620140] |On most theories, there are a few reasons some might be interpreted as some-but-not-all or not.
[222003620150] |The usual intuition is that part of why we assume John did some of his homework means some-but-not-all is because if it were true that John did all of his homework, the speaker would have just said so ... unless, of course, the speaker doesn't know if John did all his homework or if the speaker does know but have a good reason to obfuscate. 
[222003620160] |At least, that's what many theorists assume, but proving it has been hard.
[222003620170] |Last year, <a>Bonnefon, Feeney &Villejoubert</a> published a nice study showing that people are less likely to interpret some as some-but-not-all in so-called "face-threatening" contexts -- that is, when the speaker is being polite.
[222003620180] |For instance, suppose you are a poet and you send 10 poems to a friend to read.
[222003620190] |Then you ask the friend what she thinks, and she says, "Some of the poems need work."
[222003620200] |In this case, many people suspect that the friend actually means all of the poems need work, but is being polite. 
[222003620210] |<b>The study</b> 
[222003620220] |In this quick study, I wanted to replicate and build on Bonnefon et al's work.
[222003620230] |The experiment was simple.
[222003620240] |People read short statements and then answered a question about each one.
[222003620250] |The first two statement/question pairs were catch trials -- trials with simple questions and obvious answers.
[222003620260] |The small number of participants who got those wrong were excluded (presumably, they misunderstood the instructions or simply weren't paying attention). 
[222003620270] |The critical trial was the final one.
[222003620280] |Here's an example:
[222003620290] |Sally: 'John daxed some of the blickets.'
[222003620300] |'Daxing' is a neutral activity, neither good nor bad.
[222003620310] |Based on what Sally said, how likely is it that John daxed ALL the blickets?
[222003620320] |As you can see, the sentence contained unknown words ('daxing', 'blickets'), and participants were presented with a partial definition of one of them (that daxing is a neutral activity).
[222003620330] |The reason to do this was that it allowed us to manipulate the context carefully. 
[222003620340] |Each participant was in one of six conditions.
[222003620350] |Either Sally said "John daxed some...," as in the example above, or she said "I daxed some..."
[222003620360] |Also, "daxing" was described as either a neutral activity, as in the example above, or a negative activity (something to be ashamed of), or a positive activity (something to be proud of). 
[222003620370] |<b>Results</b> 
[222003620380] |As shown in the graph, whether daxing was described as positive, negative or neutral affected whether participants thought all the blickets were daxed (e,g, that some meant at least some rather than some-but-not-all) when Sally was talking about her own actions ("I daxed some of the blickets"). 
[222003620390] |This makes sense, if 'daxing' is something to be proud of, then if Sally daxed all of the blickets, she'd say so.
[222003620400] |Since she didn't, people assume she daxed only some of them (far right blue bar in graph).
[222003620410] |Whereas if daxing is something to be ashamed of, then even if she daxed all of them, she might prefer to say "I daxed some of the blickets" as a way of obfuscating -- it's technically true, but misleading. 
[222003620420] |Interestingly, this effect didn't show up if Sally was talking about John daxing blickets.
[222003620430] |Presumably this is because people think the motivation to brag or lie is less strong when talking about a third person. 
[222003620440] |Also  interestingly, people weren't overall more likely to interpret some as meaning some-but-not-all when the sentence was in the first-person ("I daxed..."), which I had predicted to be the case.
[222003620450] |As described above, many theories assume that some should only be interpreted as some-but-not-all if we are sure the speaker knows whether or not all holds.
[222003620460] |We should be more sure when the speaker is talking about her own actions than someone else's.
[222003620470] |But I didn't find any such affect.
[222003620480] |This could be because the theory is wrong, because the effect of using first-person vs. third-person is very weak, or because participants were at <a>floor</a> already (most people in all 6 conditions thought it was very unlikely that all the blickets were daxed, which can make it hard detect an effect -- though it didn't prevent us from finding the effect of the meaning of 'daxing'). 
[222003620490] |<b>Afterword</b> 
[222003620500] |I presented these data at a <a>workshop on scalar implicature</a> that I organized last Thursday.
[222003620510] |It was just one experiment of several dozen included in that talk, but it was the one that seemed to have generated the most interest.
[222003620520] |Thanks once again to all those who participated. 
[222003620530] |-------------- Bonnefon, J., Feeney, A., &Villejoubert, G. (2009).
[222003620540] |When some is actually all: Scalar inferences in face-threatening contexts Cognition, 112 (2), 249-258 DOI: <a>10.1016/j.cognition.2009.05.005</a>
[222003630010] |<title>Caveat emptor: Is academia a pyramid scheme?</title>
[222003630020] |That's the question on the blogs this week (see <a>here</a> and <a>here</a>).
[222003630030] |The question arises because each professor will have some number of students during their career (10-20 is common among the faculty I know), whereas the number of professorships increases very slowly.
[222003630040] |So the number of PhDs being produced far exceeds the number of academic positions. 
[222003630050] |As pointed out <a>elsewhere</a>, this neglects the fact that many PhD students have no intention of going into academia.
[222003630060] |Even so, it's clear the system is set up to produce more graduates who want academic jobs than there are jobs available.
[222003630070] |Prodigal Academic <a>wonders</a> if that's any different from any profession -- generally, there are more people who want the best jobs than there are best jobs to go around.
[222003630080] |Unlike PA, who doesn't think there's a problem, Citation Needed <a>thinks</a> most people entering graduate school aren't aware of how unlikely it is that they will get a tenure-track job, partly because it isn't in the schools' interest to mention this. 
[222003630090] |<b>It depends</b> 
[222003630100] |I largely agree with these fine posts, but I think they overgeneralize.
[222003630110] |Not all PhD programs are the same.
[222003630120] |Different fields vary wildly in terms of number of students produced, the likelihood of getting an industry job, etc., and also in terms of the caliber of the program.
[222003630130] |For instance, nearly every graduate of the psych program at Harvard goes on to get a tenure-track job.
[222003630140] |A sizable percentage get tenure-track jobs at the top institutions (Harvard, Yale, UChicago, etc.). 
[222003630150] |On the other hand, at even highly-respected but lower-ranked schools, getting a tenure-track job seems to be the exception.
[222003630160] |Here I have less personal experience, but a friend from Harvard who is a post-doc at a well-known state school was surprised to discover basically none of the students in that program expected to get an academic job.
[222003630170] |I've heard similar stories from a few other places. 
[222003630180] |<b>A common problem</b> 
[222003630190] |This isn't unique to academia.
[222003630200] |Many people believe lawyers earn a lot of money.
[222003630210] |Much fuss is made in the New York Times about how starting salary at a major law first is around $170,000/year (or was, prior to the Great Recession).
[222003630220] |While basically anyone who graduates from the top three law schools who wants such a job can get one (some go into lower-paying public-interest or public-service work), at most law schools, few if any graduates land such jobs and most lawyers<a> never earn anywhere near that money</a>.
[222003630230] |As a first approximation, nobody who graduates from law school lands a big firm job, just as, as a first approximation, nobody with a PhD gets a tenure track job at a top research institution. 
[222003630240] |From my vantage point, the problem is that media (newspapers, movies, etc.) fixate on the prosperous tip of the iceberg.
[222003630250] |Newspapers do this because their target audience (rather, the target audience of many of the advertisers in newspapers) are people who themselves graduated from Harvard or Yale and for whom getting a tenure-track job or being partner at a major law firm is a reasonably common achievement.
[222003630260] |Movies and television shows do this for the same reason everyone is beautiful and rich on the screen -- nobody ever said Hollywood was realistic. 
[222003630270] |This is fine as it goes, but can get people into trouble when they don't realize (a) that the media is presenting the outliers, not the norm, and/or (b) just where their own school/program fits into the grand scheme of things.
[222003630280] |As Citation Needed points out, it's not necessarily in the interest of less successful schools to warn incoming students that their chances of a job are poor.
[222003630290] |And, particularly in the realm of undergraduate education, there are certainly there are schools who cynically accept students knowing that their degree is so worthless that the students will almost certainly <a>default on their loans</a>. 
[222003630300] |<b>What to do</b> 
[222003630310] |Obviously the real onus is on the student (caveat emptor) to make sure they know what their chances of getting the job they want are prior to matriculating -- and this is true for every degree, not just PhDs.
[222003630320] |For most schools -- undergraduate and particularly graduate -- you can get data on how graduates fare in the marketplace.
[222003630330] |This can help determine not only which school to go to but whether it's worth going to school at all (it may not be).
[222003630340] |But to the extent it is in society's interest that people aren't wasting time and money (often as not, taxpayer money), it is worth considering how, as a society, we can make sure that not only is the information available, but people know that it's available and where to get it.
[222003640010] |<title>Do professors teach?</title>
[222003640020] |Luis Von Ahn has an excellent discussion on his blog about the <a>teaching/research balance</a> at major research universities.
[222003640030] |The comments are worth a read as well, especially <a>this one</a> and Von Ahn's <a>response</a>.
[222003650010] |<title>Friends of science in government</title>
[222003650020] |The Democratic congress continues its support of American basic research.
[222003650030] |The House subcommittee recommended a <a>7.2% increase for NSF</a> in the coming year, despite general belt-tightening <a>hysteria</a>.
[222003650040] |It's not <a>as much as is needed</a>, but it's still a nice change from 2001-2008.
[222003650050] |Hopefully it'll survive the rest of the legislative process.
[222003660010] |<title>Do Language Universals Exist?</title>
[222003660020] |Is there an underlying structure common to all languages?
[222003660030] |There are at least two arguments in favor of that position.
[222003660040] |One is an in principle argument, and one is based on observed data. 
[222003660050] |Since <a>Chomsky</a>, many researchers have noted that language would be impossible to learn if one approached it without preconceptions.
[222003660060] |It's like solving for 4 variables with only 3 equations -- for those of you who have forgotten your math, that can't be done.
[222003660070] |<a>Quine pointed out the problem for semantics</a>, but the problem extends to syntax. 
[222003660080] |The data-driven argument is based on the observation that diverse languages share many properties.
[222003660090] |All languages, it is claimed, have nouns and verbs.
[222003660100] |All languages have consonants and vowels.
[222003660110] |All languages put agents (the do-ers; Jane in Jane broke the window) in subject position and patients (the do-ees; the window in Jane broke the window) in object position.
[222003660120] |And so on.
[222003660130] |(<a>Here's an extensive list</a>.) 
[222003660140] |Though many researchers subscribe to this universal grammar hypothesis, it has always been controversial.
[222003660150] |Last year, Evans and Levinson published an extensive refutation of the hypothesis in Behavioral and Brain Sciences.
[222003660160] |They don't tackle the in principle argument (it's actually tough to argue against, since it turns out to be logically necessary), but they do take issue with the data-based argument. 
[222003660170] |<b>Rare languages</b> 
[222003660180] |Evans and Levinson point out that at best 10% of the world's 7,000 or so languages have been studied in any great detail, and that the bulk of all work on language has focused on English.
[222003660190] |They claim that researchers only believe in linguistic universals because they've only looked at a relatively small number of often closely-related languages, and they bring up counter-examples to proposed universals found in obscure languages. 
[222003660200] |This argument cuts both ways.
[222003660210] |The correct characterization of a language is very, very hard.
[222003660220] |Much of the work I have been doing lately has been an attempt to correctly characterize the semantics of about 300 related verbs in English.
[222003660230] |Hundreds of papers have been written about these verbs over the last half-century.
[222003660240] |Many of them have turned out to be wrong --  not because the researchers were bad, but because the problem is hard. 
[222003660250] |That's 300 verbs in the most-studied language on the planet, and we still have work to do.
[222003660260] |Evans and Levinson are basing their arguments on broad-scale phenomena in extremely rare, poorly-studied languages. 
[222003660270] |<b>A friend of a friend told me...</b> 
[222003660280] |The rare languages that Evans and Levinson make use of are not -- as they readily acknowledge -- well-understood.
[222003660290] |In arguing against <a>recursion</a> as a linguistic universal, they bring up Piraha, a language spoken in a handful of villages deep in the Amazon.
[222003660300] |Without discussing recursion in detail, the basic claim is that there are sentences that are ungrammatical in Piraha, and these sentences are ungrammatical because they require recursion. 
[222003660310] |To my knowledge, there is one Spanish-Piraha bilingual speaker, in addition to two English-speaking missionaries who, as adults, learned Piraha.
[222003660320] |The claim that Piraha doesn't have recursion is based on the work of one of those missionaries.
[222003660330] |So the data that sentences with recursion are ungrammatical in Piraha is based on a limited number of observations.
[222003660340] |It's not that I don't trust that particular researcher -- it's that I don't trust any single study (including my own), because it's easy to make mistakes. 
[222003660350] |Looking back at English, I study emotion verbs in which the subject of the verb experiences an emotion (e.g., fear, like, love).
[222003660360] |A crucial pillar of one well-known theory from the 1990s was that such verbs can't be prefixed with "un".
[222003660370] |That is, English doesn't have the words unfeared or unliked.
[222003660380] |While I agree that these words sound odd, a quick Google search shows that <a>unfeared</a> and <a>unliked</a> are actually pretty common.
[222003660390] |Even more problematic for the theory, <a>unloved</a> is a perfectly good English word.
[222003660400] |In fact, many of these verbs do allow "un" prefixation.
[222003660410] |The author, despite being an experienced researcher and a native speaker of English, was just wrong. 
[222003660420] |Even assuming that you are correct in claiming that a certain word or sentence doesn't appear in a given language, you could be wrong about why.
[222003660430] |Some years ago, <a>Michael Tomasello (and others) noticed </a>that certain constructions are more rare in child speech than one might naively expect.
[222003660440] |He assumed this was because the children didn't know those constructions were grammatical.
[222003660450] |For instance, in inflected languages such as Spanish or Italian, young children rarely use any verbs in all possible forms.
[222003660460] |A number of people (e.g., <a>Charles Yang</a>) have pointed out that this assumes that the children would want to say all those words.
[222003660470] |Take a look at this chart of all the forms of the Spanish verbs <a>hablar, comer</a><a> and </a><a>vivir</a>.
[222003660480] |The child might be excused for never using the form habriamos hablado ("we would have spoken") -- that doesn't mean she doesn't know what it is. 
[222003660490] |In short, even in well-studied languages spoken by many linguists, there can be a lot of confusion.
[222003660500] |This should give us pause when looking at evidence from a rare language, spoken by few and studied by fewer. 
[222003660510] |<b>Miracles are unlikely, and rare</b> 
[222003660520] |Some centuries ago, David Hume got annoyed at people claiming God must exist, otherwise how can you explain the miracles recorded in the Bible?
[222003660530] |Hume pointed out that by definition, a miracle is something that is essentially impossible.
[222003660540] |As a general rule, seas don't part, water doesn't turn into wine, and nobody turns into pillars of salt.
[222003660550] |Then consider that any evidence you have that a miracle did in fact happen could be wrong.
[222003660560] |If a friend tells you they saw someone turn into a pillar of salt, they could be lying.
[222003660570] |If you saw it yourself, you could be hallucinating.
[222003660580] |Hume concludes that however strong your evidence that a miracle happened is, that could never be as strong as the extreme unlikelihood of a miracle actually happening -- and, in any case, the chance that the Bible is wrong is way higher than the chance that Moses in fact did part the Sea of Reeds. 
[222003660590] |(For those of you who are worried, this isn't necessarily an argument against the existence of God, just an argument against gullibility.) 
[222003660600] |Back to the question of universals.
[222003660610] |Let's say you have a candidate linguistic universal, such as recursion, that has shown up in a large number of unrelated and well-studied languages.
[222003660620] |These facts have been verified by many, many researchers, and you yourself speak several of the languages in question.
[222003660630] |So the evidence that this is in fact a linguistic universal is very strong. 
[222003660640] |Then you come across a paper that claims said linguistic universal doesn't apply in some language X. Either the paper is right, and you have to toss out the linguistic universal, or it's wrong, and you don't. Evans and Levinson err on the side of tossing out the linguistic universal.
[222003660650] |Given the strength of evidence in favor of some of these universals, and the fact that the counter-examples involve relatively poorly-understood languages, I think one might rather err on the other side.
[222003660660] |As they say, extraordinary claims require extraordinary evidence. 
[222003660670] |<b>The solution</b> 
[222003660680] |Obviously, the solution is not to say something about "extraordinary claims" and wander on.
[222003660690] |Evans and Levinson's paper includes a plea to researchers to look beyond the usual suspects and start doing more research on distant languages.
[222003660700] |I couldn't agree more, particularly as many of the world's language are dying and the opportunity to study them is quickly disappearing. 
[222003660710] |------- Evans, N. and Levinson, S. (2009).
[222003660720] |The myth of language universals: Language diversity and its importance for cognitive science Behavioral and Brain Sciences, 32 (05) DOI: <a>10.1017/S0140525X0999094X</a>
[222003670010] |<title>Universal meaning</title>
[222003670020] |<a>My earlier discussion</a> of Evans and Levinson's critique of universal grammar was vague on details.
[222003670030] |Today I wanted to look at one specific argument. 
[222003670040] |<b>Funny words</b> 
[222003670050] |Evans and Levinson briefly touch on universal semantics (variously called<a> "the language of thought" or "mentalese"</a>).
[222003670060] |The basic idea is that language is a way of encoding our underlying thoughts.
[222003670070] |The basic structure of those thoughts is the same from person to person, regardless of what language they speak.
[222003670080] |Quoting Pinker, "knowing a language, then, is knowing how to translate mentalese into strings of words and vice versa.
[222003670090] |People without a language would still have mentalese, and babies and many nonhuman animals presumably have simpler dialects." 
[222003670100] |Evans and Levinson argue that this must be wrong, since other languages have words for things that English has no word for, and similarly English has words that don't appear in other languages.
[222003670110] |This is evidence against a simplistic theory on which all languages have the same underlying vocabulary and differ only on pronunciation, but that's not the true language of thought hypothesis.
[222003670120] |Many of the authors cited by Evans and Levinson -- particularly Pinker and Gleitman -- have been very clear about the fact that languages pick and choose in terms what they happen to encode into individual words. 
[222003670130] |<b>The Big Problems of Semantics</b> 
[222003670140] |This oversight was doubly disappointing because the authors didn't discuss the big issues in language meaning.
[222003670150] |One classic problem, which I've discussed before <a>on this blog</a>, is the gavagai problem.
[222003670160] |Suppose you are visiting another country where you don't speak a word.
[222003670170] |Your host takes you on a hike, and as you are walking, a rabbit bounds across the field in front of your.
[222003670180] |Your host shouts "gavagai!"
[222003670190] |What should you think gavagai means? 
[222003670200] |There are literally an infinite number of possibilities, most of which you probably won't consider.
[222003670210] |Gavagai could mean "white thing moving," or "potential dinner," or "rabbit" on Tuesdays but "North Star" any other day of the week.
[222003670220] |Most likely, you would guess it means "rabbit" or "running rabbit" or maybe "Look!"
[222003670230] |This is a problem to solve, though -- given the infinite number of possible meanings, how do people narrow down on the right one? 
[222003670240] |Just saying, "I'll ask my host to define the word" won't work, since you don't know any words yet.
[222003670250] |This is the problem children have, since before explicit definition of words can help them learn anything, they must already have learned a good number of words. 
[222003670260] |One solution to this problem is to assume that humans are built to expect words of certain sorts and not others.
[222003670270] |We don't have learn that gavagai doesn't change it's meaning based on the day of the week because we assume that it doesn't. 
[222003670280] |<b>More problems</b> 
[222003670290] |That's one problem in semantics that is potentially solved by universal grammar, but not the only.
[222003670300] |Another famous one is the linking problem.
[222003670310] |Suppose you hear the sentence "the horse pilked the bear".
[222003670320] |You don't know what pilk means, but you probably think the sentence describes the horse doing something to the bear.
[222003670330] |If instead you find out it describes a situation in which the bear knocked the horse flat on its back,  you'd probably be surprised. 
[222003670340] |That's for a good reason.
[222003670350] |In English, transitive verbs describe the subject doing something to the object.
[222003670360] |That's not just true of English, it's true of almost every language.
[222003670370] |However, there are some languages where this might not be true.
[222003670380] |Part of the confusion is that defining "subject" and "object" is not always straightforward from language to language.
[222003670390] |Also, languages allow things like passivization -- for instance, you can say John broke the window or The window was broken by John.
[222003670400] |When you run into a possible exception to the subject-is-the-doer rule, you want to make sure you aren't just looking at a passive verb. 
[222003670410] |<a>Once again</a>, this is an example where we have very good evidence of a generalization across all languages, but there are a few possible exceptions.
[222003670420] |Whether those exceptions are true exceptions or just misunderstood phenomena is an important open question. 
[222003670430] |------- Evans, N. and Levinson, S. (2009).
[222003670440] |The myth of language universals: Language diversity and its importance for cognitive science Behavioral and Brain Sciences, 32 (05) DOI: <a>10.1017/S0140525X0999094X</a> 
[222003670450] |<a>photo credit</a>
[222003680010] |<title>Confusing verbs</title>
[222003680020] |The <a>first post on universal grammar</a> generated several good questions.
[222003680030] |Here's an extended response to <a>one of them</a>:
[222003680040] |You said that a 1990's theory was dead wrong because sometimes emotion verbs CAN be prefixed with -un.
[222003680050] |Then you give examples of adjectives, not verbs, that have been prefixed: unfeared, unliked, unloved.
[222003680060] |I know these words are also sometimes used as verbs, but in the prefixed versions they are clearly adjectives. 
[222003680070] |The theory I'm discussing wanted to distinguish between emotion verbs which have experiencers as subjects (fear, like) and those who have experiencers as objects (frighten, confuse).
[222003680080] |The claim was that the latter set of verbs are "weird" in an important way, one effect of which is that they can't have past participles. 
[222003680090] |This brings up the obvious problem that "frighten" and "confuse" appear to have past participles: "frightened" and "confused".
[222003680100] |The author then argued that these are not actually past participles -- they're adjectives.
[222003680110] |The crucial test is that you can add "un" to an adjective but not a participle (or so it's claimed).
[222003680120] |Thus, it was relevant that you can say "unfrightened" and "unconfused", suggesting that these are adjectives, but you can't say "unfeared" or "unliked", suggesting that these are participles, not adjectives. 
[222003680130] |The problem mentioned in the previous post was that there are also subject-experiencer verbs that have participles which can take the "un" prefix, such as "unloved".
[222003680140] |There are also object-experiencer verbs that have participles which can't be "un" prefixed, like "un-angered" (at least, it sounds bad to me; try also "ungrudged", "unapplauded", or "unmourned").
[222003680150] |So the "un" prefixation test doesn't reliably distinguish between the classes of verbs.
[222003680160] |This becomes apparent once you look through a large number of both types of verbs (here are complete lists of <a>subject-experiencer</a> and <a>object-experiencer</a> verbs in English). 
[222003680170] |There is a bigger problem, which is that the theory assumes a lack of homophones.
[222003680180] |That is, there could be two words pronounced like "frightened" -- one is a past participle and one is an adjective.
[222003680190] |The one that can be unprefixed is the adjective.
[222003680200] |So the fact that "unfrightened" exists as a word doesn't rule out the possibility that "frighten" has a past participle. 
[222003680210] |To be completely fair to the theory, the claim that object-experiencer verbs are "weird" (more specifically, that they require syntactic movement) could be still be right (though I don't think it is).
[222003680220] |The point here was that the specific test proposed ("un" prefixation) turned out to provide different results.
[222003680230] |It actually took some time for people to realize this, and you still see the theory cited.
[222003680240] |The point is that getting the right analysis of a language is very difficult, and typically many mistakes are made along the way.
[222003690010] |<title>Sell off Harvard Medical School!</title>
[222003690020] |Writing in the Chronicle of Higher Education, <a>Andrew Hacker and Claudia Dreifus</a> contend
[222003690030] |Colleges are taking on too many roles and doing none of them well.
[222003690040] |They are staffed by casts of thousands and dedicated to everything from esoteric research to vocational training—and have lost track of their basic mission to challenge the minds of young people...
[222003690050] |Spin off medical schools, research centers, and institutes...
[222003690060] |For people who want to do research, plenty of other places exist—the Brookings Institution, the Rand Corporation, the Howard Hughes Medical Institute—all of which do excellent work without university ties.
[222003690070] |Never mind that <a>Howard Hughes is intimately tied to the present university system</a>, let's say we're in favor: sell off Harvard Medical School, Harvard Law School, Harvard Kennedy School, etc., until all that's left is the College.
[222003690080] |That'd make it what? -- <a>Wellesley</a> + men?
[222003690090] |(This question is meant to be snarky, but not anti-Wellesley, for which I have the utmost respect, as will be clear in the rest of the post).
[222003690100] |<b>It's the money, stupid.</b>
[222003690110] |The blogosphere has been <a>rising</a> to the <a>defense</a> of the research university, with posters and commenters focusing on the (<a>alleged</a>) claim that universities use research dollars to fund the loss-leading undergraduate programs.
[222003690120] |Here's <a>Mike the Mad Biologist</a>:
[222003690130] |[O]n a federal grant, usually somewhere between 30-40% of the total grant award doesn't go to the researcher for research costs (salaries, supplies, etc.), but to the institution.
[222003690140] |Now, some of that money is spent on actual administrative costs, but the rest goes to the university*.
[222003690150] |So if the university spins off $50 million, or $100, or, <a>in the case of the University of Iowa</a>, $169,175,021 of NIH funding alone (never mind other government sources), that's tens of millions of dollars that have to be recovered.
[222003690160] |Since I've called for more of a <a>research</a> institute <a>model</a>, I'm not opposed to spinning off research institutes.
[222003690170] |But I have no idea how universities that receive a lot of research dollars will make up the revenue shortfall.
[222003690180] |There's an easy way of answering the question: write to any of the numerous, high-calliber exclusively-undergraduate institutions that makes the American education system so interesting: Wellesley, Swarthmore, Amherst, Grinnell, Oberlin, etc.
[222003690190] |For the last 150-200 years, such schools have focused on teaching, and teaching caliber is weighted heavily in tenure decisions.
[222003690200] |I had phenomenal professors.
[222003690210] |To name a few, <a>Arlene Forman</a> could have taught a turnip to speak Russian, and <a>Jim Walsh</a> delivered spellbinding lectures despite unpromising subject material (e.g., linear algebra).
[222003690220] |People who had never even attended Ron DiCenzo's classes nonetheless raved about the vicarious experience.  
[222003690230] |<b>Research University vs. Liberal Arts College</b> 
[222003690240] |I loved the small liberal arts college experience and wouldn't have traded it for anything.
[222003690250] |But I have friends who feel the same way about the large research university: the inspirational presence of movers and shakers in the research world, they feel, is irreplaceable.
[222003690260] |I'm skeptical, but the great thing about the American education system is that it provides both options, something that many (all?) other countries lack.
[222003690270] |The only distressing thing is that so many students -- along with the Chronicle of Higher Education and essentially every blogger I read and all their commenters -- seem completely unaware that an alternative to the research university exists. 
[222003690280] |America has research-only institutes.
[222003690290] |It has undergraduate-only schools.
[222003690300] |And it has that fabulous hybrid institution: the research university.
[222003690310] |Arguing that we need to start founding undergraduate-only schools is like saying America really needs subways.
[222003690320] |Maybe we need more subways (I think we do!), but claiming they don't exist is just ignant, and it's an insult to the ones that exist and the people who made them possible.
[222003700010] |<title>Garbage in, Garbage out</title>
[222003700020] |While watching television, have you ever had a fatal heart attack? 
[222003700030] |If you answered "yes" to this question, you would have been marked as a "bad participant" in Experimental Turk's <a>recent study</a>.
[222003700040] |The charitable assumption would be that you weren't paying attention.
[222003700050] |Importantly for those interested in using Amazon Mechanical Turk for research, participants recruited through AMT were no more likely to answer "yes" than participants tested in a traditional lab-based setting (neither group was likely to say "yes"). 
[222003700060] |It's a nice post, though I think that Experimental Turk's analysis is over-optimistic, for reasons that I'll explain below.
[222003700070] |More interesting, though, is that Experimental Turk apparently does not always include such catch trials in their experiments.
[222003700080] |In fact, they find the idea so novel that they actually cited a 2009 paper from the Journal of Experimental Social Psychology that "introduces" the technique -- which means the editors and reviewers at this journal were similarly impressed with the idea. 
[222003700090] |That's surprising. 
[222003700100] |<b>Always include catch trials</b> Including catch trials is often taught as basic experimental method, and for good reason.
[222003700110] |As Experimental Turk points out, you never know if your participants are paying attention.
[222003700120] |Inevitably, some aren't -- participants are usually paid or given course credit for participation, so they aren't always very motivated.
[222003700130] |Identifying and excluding the apathetic participants can clean up your results.
[222003700140] |But that's not the most important reason to include catch trials. 
[222003700150] |Even the best participant may not understand the instructions.
[222003700160] |I have certainly run experiments in which the majority of the participants interpreted the instructions differently from how I intended.
[222003700170] |A good catch trial is designed such that the correct answer can only be arrived at if you understand the instructions.
[222003700180] |It is also a good way of making sure you're analyzing your data correctly -- you'd be surprised how often a stray negative sign worms its way into analysis scripts. 
[222003700190] |Sometimes participants also forget instructions.
[222003700200] |In a recent study, I wasn't finding a difference between the control and experimental groups.
[222003700210] |I discovered in debriefing that most of the participants in the experimental group had forgotten the key instruction that made the experimental group the experimental group.
[222003700220] |No wonder there wasn't a difference!
[222003700230] |And good thing I asked. 
[222003700240] |The catch trial -- the question with the obvious answer -- is just one tool in a whole kit of tricks used to validate one's results.
[222003700250] |There are other options, too.
[222003700260] |In reading studies, researchers often ask comprehension questions -- not because the answers themselves matter (the real interest is in what the participants do while reading), but simply to prove that the participants in fact did read and understand the material. 
[222003700270] |Similar is the embedded experiment -- a mini experiment embedded into your larger experiment, the only purpose of which is to replicate a well-established result.
[222003700280] |For instance, in a recent experiment I included a vocabulary test (which you can also find in <a>this experiment</a> I'm running with Laura Germine at TestMyBrain.org).
[222003700290] |I also asked the participants for their SAT scores (these were undergraduates), not because I cared about their scores per se, but I was able to show that their Verbal SAT scores correlated nicely with performance on the vocabulary test (Math SAT scores less so), helping to validate the our vocab test. 
[222003700300] |<b>Beyond Surveys</b> Although I described catch trials mostly in terms of survey-format studies, the same techniques can be embedded into nearly any experiment.
[222003700310] |I've used them for reading-time, eye-tracking and ERP experiments as well.
[222003700320] |The practice isn't even specific to psychology/cognitive science.
[222003700330] |During my brief sojourn in a wet lab in high school, my job was to help genotype genetic knock-out mice to make sure that the genes in question really were missing from the relevant mice and not from the control mice.
[222003700340] |It probably wouldn't have occurred to the PIs in that lab to just assume the knock-out manipulation worked.
[222003700350] |Fail that, and none of the rest of the experiment is interpretable. 
[222003700360] |A version of the catch trial is even seen in debugging software, where the programmer inserts code that isn't relevant to the function of the program per se, but the output of which helps determine whether the code is doing what it's supposed to. 
[222003700370] |It is true that some experiments resist checks of this sort.
[222003700380] |I have certainly run experiments where  by design I couldn't easily confirm that the participants understood the experiment, were paying attention, etc.
[222003700390] |But that is better avoided if possible -- which is why when I don't see such checks in an experimental write-up, I assume either (a) the checks were performed but deemed too unimportant/obvious to mention, or (b) <b>An Odd Omission</b> If catch trials are a basic aspect of good experimental design, how is it that Experimental Turk and the Journal of Experimental Social Psychology didn't know about it?
[222003700400] |I'm not sure.
[222003700410] |Part of it may be due to how experimental design is taught.
[222003700420] |It's not something you look up in an almanac, and though there are classes on technique (at least in psychology departments), they aren't necessarily that helpful since there are hundreds of types of experiments out there, each of which has its own quirks, and a class can only cover a few. 
[222003700430] |At least in my experience, experimental design is learned through a combination of the apprenticeship method (working with professors -- or, more often, more experienced graduate students) and figuring it out for yourself.
[222003700440] |The authors at Experimental Turk, it turns out, come from fields relatively new to experimental design (business, management, and political science), so it's possible they had less access to such institutional knowledge. 
[222003700450] |As far as the Journal of Experimental Social Psychology...
[222003700460] |I'm not a social psychologist, and I hesitate to generalize about the field.
[222003700470] |A lot of social psychology uses questionnaires as instruments.
[222003700480] |They go to a great deal of difficulty to validate the questionnaires -- show that they are predictive of results on other tests or questionnaires, show that the questionnaires have good test-retest reliability, etc.
[222003700490] |Many of the techniques they use are ones I would like to learn better.
[222003700500] |But I actually haven't ever run across one (again, in my limited experience) that actually includes catch trials.
[222003700510] |Which in itself is interesting. 
[222003700520] |<b>A clever idea </b><b> </b>I should add that while Experimental Turk cites said journal article for suggesting using questions with obvious answers, that's not actually what the paper suggests.
[222003700530] |Rather, it suggests using instructions telling participants to ignore certain questions.
[222003700540] |For instance: 
[222003700550] |<u>Sports Participation</u>
[222003700560] |Most modern theories of decision making recognize the fact that decisions do not take place in a vacuum.
[222003700570] |Individual preferences and knowledge, along with situational variables can greatly impact the decision process.
[222003700580] |In order to facilitate our research on decision making we are interested in knowing certain factors about you, the decision maker.
[222003700590] |Specifically, we are interested in whether you actually take the time to read the directions; if not, then some of our manipulations that rely on changes in the instructions will be ineffective.
[222003700600] |So, in order to demonstrate that you have read the instructions, please ignore the sports item below, as well as the continue button.
[222003700610] |Instead, simply click on the title at the top of this screen (i.e., "sports participation") to proceed to the next screen.
[222003700620] |Thank you very much.
[222003700630] |That's a clever idea.
[222003700640] |One of my elementary school teachers actually wrote a whole test with instructions like that to teach the class a lesson about reading instructions carefully (and it worked -- I still do!).
[222003700650] |So it's a good idea I've never seen used in an experimental setting before, but that doesn't mean it hasn't been used.
[222003700660] |In any case, the discussion in the paper doesn't mention catch trials or other methods of validating data, so it's hard to know whether they did a thorough literature search. 
[222003700670] |<b>More training</b> A bad movie can still make entertaining watching.
[222003700680] |A bad experiment is irredeemable.
[222003700690] |If the participants didn't understand the instructions, nothing can be gleaned from the data.
[222003700700] |And there are so many ways to run bad experiments -- I know, because I've employed many of them myself.
[222003700710] |There are a lot of datasets out there in psychology that have proven, shall we say, resistant to replication.
[222003700720] |Some of this has to be due to the fact that experimental design is not as good as it could and should be. 
[222003700730] |<b>Addendum</b> As I mentioned higher up, I think Experimental Turk is overly optimistic about the quality of data from AMT.
[222003700740] |I've run a couple dozen experiments on AMT now, and the percentage of participants that fail the catch trials varies a great deal, from as few as 0% to as many as 20-30%.
[222003700750] |I haven't made a systematic study of it, but there seem to be a number of contributing factors, some of which are general to all experimental venues (length of the experiment, how interesting it is, how complicated in the instructions are) and some of which are specific to AMT (the more related HITs, the more attractive a target the experiment is to spammers). 
[222003700760] |All the more reason to always include catch trials. 
[222003700770] |----------- Oppenheimer, D. M., Meyvis, T., &Davidenko, N. (2009).
[222003700780] |Instructional manipulation checks: Detecting satisficing to increase statistical power Journal of Experimental Social Psychology, 45, 867-872