A.C. Thomas, Scientist tag:www.acthomas.ca,2009-10-17:/comment//1 2012-04-09T02:56:09Z Movable Type Pro 4.32-en Prediction, The Big Discovery and Heartbreak tag:www.acthomas.ca,2012:/comment//1.46 2012-04-08T19:23:19Z 2012-04-09T02:56:09Z It's a year old, but I only just heard the story of what happened to a baseball researcher I first read about 10 years ago:http://espn.go.com/blog/sweetspot/post/_/id/6835/voros-mccracken-changed-the-gamehttp://www.thepostgame.com/features/201101/sabermetrician-exile"Voros" McCracken had a particular insight about pitcher ability in the late 1990s, namely that a... Andrew C. Thomas
http://espn.go.com/blog/sweetspot/post/_/id/6835/voros-mccracken-changed-the-game
http://www.thepostgame.com/features/201101/sabermetrician-exile

"Voros" McCracken had a particular insight about pitcher ability in the late 1990s, namely that a subset of the data -- "defense-independent" pitching statistics-- was not only an excellent predictor of the runs allowed by a pitcher, but is also highly persistent from year to year. Others discovered the same principle around the same time, but it was the publication of this work that got the attention: the strong claim that the differences between pitchers, on batted balls in play, were so small as to be ignored. While it wasn't quite correct -- there is some predictive power in the remaining information -- it was enough to change people's ideas about how the game works.

And now, McCracken is out of baseball, applying analytical methods to undisclosed professional soccer clubs, having made a meager living while working for the Boston Red Sox in the early 2000s. As one of the figures I read about before going to grad school in statistics, it definitely got my attention when I heard this news.

There are two things I take away from this whole story:

1) After 12 years of exposure to real data and methods, I feel like it should have been *screamingly obvious* to find the best set of predictors of success, and an assignment I could give to an undergrad with existing databases. But would it be obvious enough to construct a narrative around it, and convince the public? I can think of a good explanation after the fact, that there's a lot of general uncertainty once the bat hits the ball, like the angle of attack, but I've had years to think about it.

2) Any research I do on sports is for the sake of teaching, or just as a hobby, and the value it brings me to share something with the world is a nice bonus. I'm not under pressure to find a brilliant discovery to keep my job -- at least, not when it comes to sports.

Gelman has said that the most noteworthy discoveries these days aren't the small effects that come out with more data, but the big ones that everyone else just missed. This definitely qualified as one of those. Whatever the next big discovery, I'm sure we'll all think it was obvious years later, even if it wasn't.
]]> Statistics and Scrabble, Together At Last tag:www.acthomas.ca,2011:/comment//1.45 2011-07-14T14:48:43Z 2011-07-14T15:15:35Z Sitting on my to-do list for a while now has been an exploration of Scrabble from an experimental design point of view; how to better design a tournament to make the variance as small as possible while still preserving the... Andrew C. Thomas currently popular version in Europe that has no defensive element to it.

I'm proud (relieved?) to say that I've finally finished the first draft of this work for two-player head-to-head games, with a duplication method that ensures that if the game were repeated, each player would receive tiles from the reserve in the same sequence: think of the tiles being laid out in order (but unseen to the players), so that one player draws from the front and the other draws from the back. Like Lady and the Tramp with spaghetti:

tramp.jpg

I modified the Scrabble simulator Quackle to accept a predetermined tile order, then simulated over one million matches between Quackle's "Speedy Player"s using each of 10,600 tile orders 100 times. One goal of this was to figure out how much of the variance in score comes from the tile order and how much comes from the board, given that a tile order would be expected. It turns out to be about half-bag, half-board, so that if this scheme could be used in tournaments, it would visibly cut down the number of matches needed to figure out the best player (though it would need a Goldbergian apparatus to implement in live games.)

Some other findings from the simulations:

  • The blank is worth about 30 points to a good player, each S about 10.
  • The Q is a burden to whichever player receives it, effectively serving as a 5 point penalty for having to deal with it due to its effect in reducing bingo opportunities, needing either a U or a blank for a chance at a bingo and a 50-point bonus.
  • The J is essentially neutral pointwise.
  • The X and the Z are each worth about 3-5 extra points to the player who receives them. Their difficulty in playing in bingoes is mitigated by their usefulness in other short words.
I have yet to make any other conclusions about how I think the game should be modified, mainly because it's premature without testing these ideas out on human players. Any volunteers?
]]> Geek talk: Running R processes remotely through ESS tag:www.acthomas.ca,2011:/comment//1.44 2011-04-17T04:47:14Z 2011-04-17T05:09:22Z I've been wondering for a while how to use the convenience of Emacs on a local machine, while running R processors on a server remotely. According to http://www.xemacs.org/Documentation/packages/html/ess_3.html, this can be done with the ess-remote environment.1) Use M-x shell to... Andrew C. Thomas http://www.xemacs.org/Documentation/packages/html/ess_3.html, this can be done with the ess-remote environment.

1) Use M-x shell to open a shell environment.

2) Use ssh to connect to the server of interest. If desired, use "screen" to make the process resilient to disconnection.

3) Run R.

4) Use M-x ess-remote to enable the shell as an ESS process.

The advantages of this process are the added convenience of an environment that allows instant execution of lines of R code. Files still must be loaded on the remote server, rather than referenced on the local machine, but it's still an improvement.

]]> Big chances to change redistricting this year... tag:www.acthomas.ca,2011:/comment//1.43 2011-02-06T20:20:43Z 2011-02-06T20:45:02Z ...assuming we can take advantage of them. State legislatures are preparing to conduct their decennial redistricting processes, now that the data from the last census has been processed. In states where the legislature does it directly, the result is known... Andrew C. Thomas
State legislatures are preparing to conduct their decennial redistricting processes, now that the data from the last census has been processed. In states where the legislature does it directly, the result is known to be far less than fair for its constituents, in that either one party tries to take control of an "unfair" number of seats (remember the Oklahoma exodus!) or the incumbents on both sides work to protect their own re-election prospects (some people have all the luck).

There is hope that a bipartisan commission, whose members cannot run for office in their newly drawn districts, would be able to break the easily recognizable incumbency advantage and, from there, create a map that would be fairer to all parties. But how likely is this? In fact, there seems to have been very little difference in the maps produced by commissions and by legislatures in terms of the absolute performance of electoral systems, or in the change in their performance after redistricting has taken place.

Read all about it in my "editorial" paper that proposes questions about the redistricting process that commissions should be asking to guide their work this year. Not easy to publish a paper of questions unless you're near or past retirement age, let alone when those questions come from null results!

]]> Catcher Spotting Data Now Available tag:www.acthomas.ca,2011:/comment//1.42 2011-01-03T18:22:20Z 2011-01-03T18:24:58Z Thanks to all those who took part in my trial of Catcher Spotting utilities. As promised, I've posted the data for all those who want to work with it.The paper is posted here and is in submission for the JQAS... Andrew C. Thomas the data for all those who want to work with it.

The paper is posted here and is in submission for the JQAS special issue on the NCSSORS conference.
]]> Joe And Ted's Excellent Accomplishments: Streaks and the Evolving Sport of Baseball tag:www.acthomas.ca,2010:/comment//1.40 2010-10-18T15:19:51Z 2010-10-18T15:30:11Z Summarizing my latest sports piece, and first peer-reviewed publication on baseball. Get it for free here.Even after nearly a century and a half of major league play, baseball still has no shortage of great questions and puzzles, one of them... Andrew C. Thomas sports piece, and first peer-reviewed publication on baseball. Get it for free here.

Even after nearly a century and a half of major league play, baseball still has no shortage of great questions and puzzles, one of them being the phenomenon of the streak. Just how remarkable was Joe DiMaggio's 56-game hitting streak, and Ted Williams's less celebrated 84 consecutive games reaching base? If we could rerun the past 139 seasons of baseball, how likely would we be to see a streak of that length (or longer) again? And how can we trust that the answer we get back is in any way a reliable one, without the use of a time machine?

]]> streak was somehow magical, other lesser streaks would be expected. After all, while it might not be all that likely that Ty Cobb, Paul Molitor and Jimmy Rollins would run up hitting streaks of 40, 39 and 38 games respectively as their career bests, it isn't all that unusual to say that in these hypothetical do-overs, someone would have done as well in each person's place, and that streaks of roughly these lengths would end up as numbers 6, 7 and 8 on the all-time list.

To test these outcomes, I started with a publicly available database on yearly player outcomes, and assumed that each player would have the same number of games and plate appearances in each year. I then produced several plate appearances for each player in each hypothetical game by, essentially, the Strat-O-Matic or "Wheel of Fortune" method - spin a (computer-based) wheel marked with "hit", "walk" and "out", where each wedge on the wheel has as much space as the probability of each event - and noted whether a hit (or a walk) was recorded in the game. (This, of course, assumes that there is no "hot hand" effect that would cause players on a streak to keep performing above their expected ability.) This is done repeatedly to get a large number of histories to compare to the real McCoy.

For hitting streaks, it turns out that this method does a great job for creating those lesser streaks for 1950 and onwards, but overpredicts how long these high-ranked streaks would be from 1900 until 1950, and is even more wildly high for the corresponding streaks in the 19th century. An easy fix for this is to allow the hit probability to vary from day to day in these early eras - on some days the batter has a higher average than normal, on some days a lower average - and by choosing the right variabilities, those lesser streaks in simulation for each era line up with their "real" counterparts. This produces a highly educated guess about how rarely we would see a 56-game streak or more: a little less than 5% of the time since 1901, a streak of the same length as DiMaggio's would be observed. In my eyes, that's certainly a  remarkable record, whatever factors led to it.

But even this easy tweak opens up a lot of questions. If this extra fluctuation is enough to make the model look like the real world, what real-world factors would produce it? One possibility is that games pre-1940 were called due to darkness more often; another is that while on the road, players would encounter many different types of ballparks; most likely of all, the quality of opposing pitching was far more variable than today, and that a few good (or lucky) pitchers were far better at stopping streaks than others.

If this explanation is true, then we have a new issue to contend with: on the aggregate, the past 60 years of streaks needed no extra variability in day-to-day hitting to produce a streak list that was comparable to reality. Does that mean that the modern use of relief pitching has created a game where a player's opposing pitching is virtually indistinguishable from day to day? Or, more extremely, could it mean that on the whole, any two given major league pitchers are effectively indistinguishable in stopping hits from occurring? That's one philosophy of baseball research, first suggested by Robert "Voros" McCracken a little over a decade ago and still a hot topic for debate, but it also has a taste of Stephen Jay Gould, who pointed out that the variability between hitters has been decreasing over time. While Gould suggested that this would mean the .400 hitter was a thing of the past, the implication here is that this lower variability would encourage longer hitting streaks: without pitchers that are dramatically different from their counterparts, a player on a streak would be less likely to be shut down by an opposing ace today than 100 years ago.

This method isn't nearly as successful for figuring out on-base streaks; in fact, the same machinery that was just used for hits alone grossly overestimates how long history's on-base streaks would last, no matter how much reasonable difference in ability we estimate between pitchers. While this certainly implies that the opposing pitchers still vary widely in their ability to prevent bases-on-balls (the flip-side to McCracken's argument), it gives us no shortage of alternative explanations to consider as the season begins, and just as much continuing mystery about this era of the game that we are privileged to witness.]]> A Catcher Spotting Tool: "Hot Or Not?" For Baseball Pitches tag:www.acthomas.ca,2010:/comment//1.39 2010-09-27T16:04:57Z 2010-09-27T16:25:33Z Catcher Spotting is a project I've been working on casually for about 4 years, starting when I got curious about the last uncaptured bit of data from a baseball game: the set-up of the catcher, implying the intended target of... Andrew C. Thomas missing his spots", it's evidence of a loss of control, but attempts to measure the impact of this loss are utterly stymied by the lack of quality data out there.

That's where the Catcher Spotting project comes in: I am very interested in figuring out exactly how this data can be collected on a wide scale. While commentators can be in agreement about whether a pitcher has missed his spot, codifying this hasn't been established yet. And I'm far from convinced that technology can do it alone through video analysis, especially in cases of "intentional deception" like when a runner is on second. Crowdsourcing seems to be the obvious solution -- how well can you distribute the task to many different coders?

To that end, I've built an applet that tries to answer that question: users simply click their mouse to indicate where the catcher has set up, and where the ball actually goes. By collecting data from (hopefully) many users, we should know how many different human coders would be needed to get a reliable sense of a pitcher's intent, as signalled through a catcher. The idea is the same as sites like Hot Or Not, except that I'm explicitly concerned with how the same rater will judge different pitches, so that we can know how trustworthy a single rater could be.

Please give the applet a try!
]]> Invite Us To Your Parties, Scientists, Because Applied Statisticians Bring the Good Stuff tag:www.acthomas.ca,2010:/comment//1.38 2010-05-05T16:31:57Z 2010-05-05T16:45:42Z From an official response to the CRU email incident, largely clearing the actions of climate scientists:The Report points out where things might have been done better. One is to engage more with professional statisticians in the analysis of data. Another,... Andrew C. Thomas an official response to the CRU email incident, largely clearing the actions of climate scientists:

The Report points out where things might have been done better. One is to engage more with professional statisticians in the analysis of data. Another, related, point is that more efficacious statistical techniques might have been employed in some instances (although it was pointed out that different methods may not have produced different results). Specialists in many areas of research acquire and develop the statistical skills pertinent to their own particular data analysis requirements. However, we do see the sense in engaging more fully with the wider statistics community to ensure that the most effective and up-to-date statistical techniques are adopted and will now consider further how best to achieve this.
I'm not sure anything that was said in the report will placate the denier crowd, but this part here gives me hope. There are plenty of people out there who are well-experienced in cross-field statistical analysis that would love to have sit-downs with real data and provide a sounding board. Indeed, it's one of the founding purposes of my department.
]]> Because Cosma can say it better, and longer, than I can tag:www.acthomas.ca,2010:/comment//1.37 2010-04-29T00:12:32Z 2010-04-29T00:17:33Z Our paper is posted: how homophily and contagion in social networks are confounded, given the impact of latent variables on past and present outcomes, as well as network ties, in social network analysis.... Andrew C. Thomas Our paper is posted: how homophily and contagion in social networks are confounded, given the impact of latent variables on past and present outcomes, as well as network ties, in social network analysis.
]]> A Worthy Repost of Someone Else's Material tag:www.acthomas.ca,2010:/comment//1.36 2010-04-22T20:27:54Z 2010-04-22T20:45:53Z How To Publish A Comment has come up twice in the last week: once at a conference, once at a blog post. Boy, I can't wait for this to happen to me.... Andrew C. Thomas How To Publish A Comment has come up twice in the last week: once at a conference, once at a blog post. Boy, I can't wait for this to happen to me.
]]> The Statistics of the Putter's Game tag:www.acthomas.ca,2010:/comment//1.35 2010-03-15T22:18:24Z 2010-03-15T22:46:43Z When I first saw this piece on improved putting statistics in the Wall Street Journal, I put on my typical skeptic's face when it comes to science in the news, and especially when it comes to the overplay of data... Andrew C. Thomas this piece on improved putting statistics in the Wall Street Journal, I put on my typical skeptic's face when it comes to science in the news, and especially when it comes to the overplay of data analysis and "new statistics" that creeps up in this kind of reporting.

On reading the actual paper by operations researchers Douglas Fearing, Jason Acimovic and Stephen Graves, I was pleasantly surprised at the care and attention they've put into the problem. Having a rich data set is essential, and they've got one in the PGA putting database; I would cringe if the authors were required to gather their own data and make overly broad conclusions on that basis.

The writers have deftly avoided the kinds of oversimplifications that make students of sports analysis cringe. Their model is simply stated -- figure out the factors that lead to making putts, and for those that weren't made, model how bad the misses are -- but the tricks to getting the computation right are subtle. Most importantly, they validate their models against data and resist the temptation to overfit, and they do well to produce a relevant quantity for each player (shots gained through putting, compared to a baseline) that can be predicted and therefore validated on a regular basis.

I have the usual gripes about graphics and other statements; I want to see error bars on graphs, for example, and I really want to know about the predictive error -- that is, how well the putts-gained-per-round statistic will predict future putting performance (within one tournament, one month, one year, etc.) -- but all in all, I'm glad I went to read the paper myself and be reassured that the PGA is advertising a worthwhile product.
]]> Following the Shalizi Model for Blog Maintenance tag:www.acthomas.ca,2010:/comment//1.34 2010-02-08T16:43:15Z 2010-02-08T16:52:33Z My attempt to put up a web presence is negated by the fact that I don't make many trivial updates or statements; as a result, I'm less concerned with the immediate payoff of these sorts of writings and more about... Andrew C. Thomas Cosma Shalizi pointed out to me when I started this site that there are two steady states for successful blogs:

  • those that are fast to update, have lots of constant yet ephemeral traffic, and have their spam problems mitigated by a quality comment model, and
  • those that are infrequently updated, have occasional yet consistent traffic, and have their spam problems eliminated by removing the ability to comment.
I've become aware that if there's anything I want to say, I want to think carefully about it first, and make it last in the end, mirroring Cosma's model. So goodbye comments and trackbacks; if you want to respond to anything I write, you know how to find me.

]]> Sitting this one out tag:www.acthomas.ca,2010:/comment//1.33 2010-01-19T20:03:19Z 2010-01-19T20:07:11Z As a former resident of Massachusetts, it's been interesting for me to watch the Facebook reaction to the Coakley-Brown race purely from the psychology angle. I don't trust the polls or the probabilities in this case because there's so little... Andrew C. Thomas ]]> Off the grid tag:www.acthomas.ca,2009:/comment//1.32 2009-12-19T22:11:35Z 2009-12-19T22:20:36Z For the first time since I've started using email on a regular basis, I'll be without it for the next week as I spend quality time with the family. For that matter, it'll be my first time in a while... Andrew C. Thomas ]]> Tentative syllabus for 36-724: Applied Bayesian Statistical Computing tag:www.acthomas.ca,2009:/comment//1.31 2009-12-19T21:48:31Z 2009-12-19T22:11:25Z As previously offered, this course was a full semester 12 unit course following three semester courses in mathematical statistics, regression modelling and computation. Now, as there is room only for six weeks and no precursor course in computing, I'm still... Andrew C. Thomas
Carnegie Mellon University, Spring 2010: 36-724: Applied Bayesian Statistical Computing
Instructor: Andrew C. Thomas (acthomas at stat.cmu.edu)
Class Time/Place: MWF 11:30-12:20, CFA 211

Required text:
Andrew Gelman and Jennifer Hill (2007) "Data Analysis using Regression and Multilevel/Hierarchical Models". Cambridge University Press. Buy the softcover version.

Prerequisites: 36-705 ``Intermediate Statistics'', 36-707 ``Intermediate Regression''. If you have not taken these classes specifically, examine the syllabuses for these courses and make an appointment to see me within the first week of class.

The goal of this course is to give a meaningful introduction and exploration of Bayesian statistical methods through computational techniques in seven weeks. We will focus on the principles of Bayesian hierarchical modelling methods that can be programmed efficiently and remain scientifically valid, and methods for debugging without pulling too much hair out. We will not be explicitly covering discriminative machine-learning topics, but we will cover the same debugging concepts that will make things easier when coding them up.

Programming language: R will be the supported language for the course, with the possible use of WinBUGS.

Tentative outline of the course:

Week 1: Introductions. "Central Dogma of Generative Modelling", One-level models, prior specifications and conjugacy; introduction to sampling and simulation in R.
Week 2: A reintroduction to Markov Chain theory, beginning with discrete models and moving to one-dimensional continuous models.
Week 3: Generalized linear models. Grid sampling, the Metropolis-Hastings algorithm, Gibbs sampling.
Week 4: Gaussian multilevel models. Partial and full pooling of variance components; autocorrelation and cross-correlation in chains; diagnostics for convergance.
Week 5: Generalized multilevel models; posterior predictive checking.
Week 6: Varying-slope models in the multilevel context.
Week 7: Special topics to be determined; Bayesian graphical models, causal inference.

If you have any suggestions for topics that ought to be considered, please let me know.
]]>