Overview
The aim of detailed stats is to provide more data than standard stats, which only collects summary data (player statlines and final team scores) for each game. Once each question is tagged and each complete scoresheet is entered, it becomes possible to compile aggregate data by question, category, author, team, or player. This form of detailed stats, often called conversion stats, has been produced for a few tournaments over the last few years. Besides bringing gratification to players, editors have been able to use them to make many improvements to packets between mirrors. Recently, a new form of detailed stats (which has no established name yet) goes even further than conversion stats by tracking exactly where players buzzed on each tossup question.
ACF Regionals 2018 was the most widely played tournament yet to record buzz points, and the first to have detailed stats with so many teams playing on the exact same questions. There were 16854 buzzes (93% of which have a buzz location) and 13015 bonuses recorded in 737 games (using up to 69 rooms across 12 sites). About 12 packets’ worth of questions (240 tossups and 206 bonuses) were heard more than 30 times. Some questions, however, have rather smaller sample sizes, so please carefully consider that the data may not be entirely representative of the question set. For example, a team that might have performed well on a bonus did not get the opportunity to answer it.
Slight changes were made to the question set after the main group of mirrors, some of which were informed by the detailed stats and the private discussion forum.
Spreadsheets
There is a separate detailed stats spreadsheet for each site, as well as an “All sites combined” spreadsheet¹. Compared to previous iterations, there are no major changes to the spreadsheet structure. The Categories sheet lists aggregate tossup and bonus conversion by subcategory. Tossups and Bonuses list conversion for every question. The performance of each team by subcategory is in Teams–Categories. The performance of each player is in Player, which is broken down by category and subcategory in the remaining tabs.
The spreadsheets are mainly intended for casual observation and only contain rudimentary reports and breakdowns.
Raw data
The raw tossup and bonus data have been published (in TSV format and denormalized for easier ad-hoc analysis) in a GitHub repository under an open license as part of the launch of a new open quizbowl data initiative. If you need more data than this for your research, please email me.
Solicitation of research
Last week, I posted a warm-up thread as a prelude to releasing the detailed stats. I wanted to prepare some concluding remarks on how I felt about the outcome of that exercise, but lengthy posts like these take quite a while to write (though you are of course welcome to start that discussion yourself). Here I will briefly describe how the “deal” worked out and explain my motive, because it was a frequently asked question and intentionally somewhat veiled (which I have already elaborated on to some people in private).
So I implore you to explore the data. Harness its potential – these spreadsheets barely scratch the surface of possibilities of the underlying dataset. I don’t particularly care how you interpret the data. Just use it!I wrote: Originally, the people who emailed me were asked to post concrete predictions in the thread in exchange for early access to the draft spreadsheets, then test their hypotheses against the data, and finally post their findings. I thought this was a fair request, but only three people ever ended up emailing me. (I really just needed your email to give you access!) So instead of asking for predictions, I then gave access to anyone who managed to contact me and made any sort of productive contribution to the thread.
My true motive was to promote progress in quizbowl discourse. The incentive was meant to encourage people to explore the purportedly broad possibilities of this long-desired kind of dataset – to do some real science! – and not just scroll passively through a straightforward spreadsheet. I hoped to use early stats access as bait to trick people into investing some effort digging into data, placing value in their own discoveries, and exchanging ideas in a centralized public forum.
For a while I’ve been demoralized by the stagnation and balkanization of quizbowl discourse in general. It seems as if many of the most knowledgeable people rarely participate in mature, intelligent discussion here (not necessarily theorizing). I tend to enjoy quizbowl more when I hear and learn from more of those voices. I don’t really care for daft banter or whatever the bizarre topic du jour is. I do believe it is important to expand quizbowl to a wider audience (can there be a group reading of “The Big Vision” each year?) and that people might stick around longer if the discourse were less off-putting and more analytical, inclusive, and introspective. For an ostensibly intellectually curious activity, I expected there to be more engagement from the quizbowl community, especially during this pivotal opportunity.
Despite my pessimism, I still wanted to try to bring people together and provoke some positive change. It’s not that I think boosting engagement with detailed stats will just advance the discourse, but also the activity as a whole. Quizbowl isn’t a lifelong activity; most alumni soon drift apart from it. I can’t be relied on to have this role forever, so if the community wants to be sustainable, it must continue to pick up the technological slack.
And I’m not speaking only to the technically competent: anyone can learn Excel or similar tools. I’m happy that the incentive has motivated a moderate amount of people to participate – it may not be the most enthusiastic or enlightening discourse – but I still hope to include as much of the community as possible. I can’t be satisfied until I hear what everyone has to say, even the cagey non-posters like myself. Getting involved in this exercise shouldn’t be risky or embarrassing (as long as you didn’t defeat the whole purpose by looking at the stats before posting predictions!).
Please post your results here, along with any methodology, code, spreadsheet work, or whatever other resources are needed to make your research reproducible. (I recommend using a design that makes it easy to plug in an updated version of the data.) Make sure to fully label your plots (annotations can be very useful) so that readers can easily understand the context. If you have been accumulating tables, graphs, and analyses, or only circulating them in chat rooms or silly social media groups, I invite you to please share them publicly in this thread. I want everyone (including future quizbowl historians) to be able to see what others have investigated, build upon each other’s creativity, and make something new.
Errata
I am certain that the scoresheets have numerous mistakes, most of which involve incomplete and misattributed data. A few rooms did not record any buzz points at all, while others just forgot to paste them in at the end of a match (and could not be reached after the tournament). A small number of moderators logged incorrect tossup answers in their scoresheets as an optional enhancement to help the authors better understand how questions played, but there is a good chance that some wild guesses are assigned to the wrong team.
Please do not report corrections or problems accessing the spreadsheets in this thread. I will accept errata via email only. This is because I expect to receive a large volume of reports and I may need to give you access to the scoresheet. Your report must contain a full description of the errata (including the site, room, round, packet, question number, team, player, etc.) so that I can easily find it, or link directly to the relevant scoresheet. Referring only to the answerline (“I didn’t get credit for my buzz on Cervantes!”) will not be sufficient. If you have some kind of proof or reliable source (such as a notebook, recording, or witness) to corroborate the report, you may provide that as well. Please try to determine whether a mistake encompasses several questions (thus indicating a misalignment nearby) or only a single question.
Here is a list of errata so far. Some changes have already been made, and there may be more over time.
Thanks
I would like to thank all tournament directors, moderators, and volunteers for staffing and for putting up with an unusual scorekeeping system. My goal is to improve the experience of every player and every moderator – not to make quizbowl, a fun activity, into a burden.
Thanks as well for your patience. I spent a lot of time developing the system for creating these detailed stats and much appreciate your support. Here are the spreadsheets.
Links to spreadsheets
It may take a long time to load these large web pages. They are static “published” views of complicated spreadsheets that may update occasionally, but are not interactive like before.
All sites combined¹ · Connecticut · Georgia Tech · Kansas State · Minnesota · Oxford Brookes · Penn State · Rice · Toronto · UCF · UCSD · UIUC · Virginia
Tips
If you would like to support me creating new quizbowl technology, you can donate to my Patreon. Please don’t feel like you have to give me anything. Thank you to Aseem, Jakob, Rob, Will, and all my patrons for your generous support!
Future
I already wrote a bit about the future of the current scorekeeping system, which I would now like to discontinue. I’m not that proud of these miserable spreadsheets (I detest working with them, honestly), and they obviously can’t become the standard method for quizbowl scorekeeping.
The reason I chose to implement a scorekeeping system in this esoteric, improvised manner was for relative flexibility: it enabled the collection of valuable data without interfering much with established procedures (normal packets but with clickable words, normal scoresheets but with extra columns). If moderators could just modify a scoresheet as usual, I could worry less about business logic or unrepresentable edge cases and making a fully reliable and robust app. Collaboration, infrastructure, real-time updates, browser support/accessibility, and familiarity are pretty compelling existing advantages offered by Google Sheets (not mentioning the many disadvantages) that would have to be rebuilt in whatever successor technology comes along.
Of course, I am not content with the current presentation of the detailed stats. I did plan to make a nice web application for viewing all the varieties of stats you could wish for, but that major project is on hold in part due to a backlog of responsibilities, injury, and travel. Some elements of the existing software will eventually be open source, so follow me on GitHub if interested.
_____
¹ Unlike the others, the “All sites combined” spreadsheet only includes selected sheets due to the constraints of a single Google Sheets document (maximum number of cells, maximum number of simultaneously calculating formulas, maximum number of sheets to import external data from). Be aware that this spreadsheet does aggregate data from the Oxford Brookes site, which played a slightly different version of the set, and therefore cannot be strictly compared properly.