Does quizbowl need a public Ginseng-analogue?

[Excelsior (smack)]

I don't know what other people's experience with writing tournaments is, but in my four years of BHSAT, I have found it to be the Worst Thing. Perhaps the only thing less pleasant than writing chemistry questions for high-schoolers is _assembling_ those questions into an actual packet set.

This being the case, I am strongly considering developing some sort of web application to make the act of creating tournaments less unpleasant. I have some fragmentary code underway already, but before I get too deep into this, I want to know if anybody would even use something like this.

The basic idea here would be to develop something that is like Ginseng, but tailored for groups of people writing one-off tournaments - i.e. most college and high school independent tournaments, which do not need structured support for question banks, privilege/role management, and similar features that HSAPQ and NAQT definitely do need. It would probably make most sense to start with support for housewrites, and then introduce baked-in support for packet submission later. This would then be made freely available to all comers.

I believe that an automated tournament-preparation system like this will have a number of add-on benefits if it sees widespread adoption, such as the ability to impose a standardized structured data interchange format for quizbowl packets (probably XML-ish), which would make quizbowl software that layers on top of packets (e.g. the various question databases; things like Abacus might benefit too) easier to develop. More broadly (and perhaps grandiosely), I have a long-term vision of a more powerful technology ecosystem for quizbowl, and a public tournament-creation tool is an integral part of that.

Thoughts? (Also, is there any work already being done in this direction outside QEMS/Ginseng?)

[jonah]

I started working on this about five times, but I don't think any of my work survives. I'd be happy to talk about the general ideas I had for data architecture / schemata, or about Ginseng (in a non-trade-secret-revealing way, of course).

I will mention that I have used MediaWiki installations to collaborate on tournament projects frequently, and found it to be the least painless method (though still fairly painful).

[bdavery]

If all questions are written/edited before you assemble packets, it's much faster to put questions in by subject instead of by game. For a 10-round set, cut and paste the 10 chemistry questions in, one per round, from round 1 to round 10. Then the 10 bio, or 10 US hist, or whatever. Doing it that way, you can assemble 10 rounds of tossups pretty easily in 90 minutes or less. After all the tossups, do all the bonuses.

Doing it 1 game at a time instead , 20 tossup categories at a time and then 20 bonus categories, does indeed take forever.

[Excelsior (smack)]

If all questions are written/edited before you assemble packets, it's much faster to put questions in by subject instead of by game. For a 10-round set, cut and paste the 10 chemistry questions in, one per round, from round 1 to round 10. Then the 10 bio, or 10 US hist, or whatever. Doing it that way, you can assemble 10 rounds of tossups pretty easily in 90 minutes or less. After all the tossups, do all the bonuses.

True, but this is just one of many issues that arise from manually assembling packets. Others include:

• Questions that get split across pages
• Inconsistent formatting because Word is awful
• Repeats possibly being created by incorrect copy-pasting
• Loss of formatting because of conversion between Google Docs / Word / what have you
• Randomization / pseudo-randomization done by hand is error-prone and takes longer than it needs to
• Inability to factor out common document elements like a header ("XYZ Tournament 2014, written by...") or a stop-at-20-questions notice, causing inconsistency

In addition to those issues, other problems unrelated to the actual assembly process include:

• Lack of version history, forcing you to resort to talking to people to figure out who changed that clue and why (Google Docs doesn't really help here because its revision history is a bear to trawl through; Mediawiki would help, but I had no idea people used wikis for this at all until Jonah just mentioned this)
• Inability to automatically validate questions for style considerations (e.g. "for 10 points" vs. "FTP"; square brackets vs. parentheses for alternate answerlines)
• At least in projects I've been involved in, the actual questions are decoupled from the answer spreadsheet, sometimes causing inconsistency
• Extracting the particular iteration of a question that a given player played is tedious, and requires trawling through large piles of saved docfiles
• Password-controlling documents is tedious
• Making bulk changes to all the packets in a set (e.g. converting straight quotes to smart quotes) is tedious
• It is often difficult to get Word documents to convert cleanly to PDF for maximum portability (and even if you do get the conversion right, they often look ugly)
• Combining packets for packet-sub tournaments requires lots of tedious copying and pasting and careful management of questions to ensure none of them end up in the wrong packet

A system like this could also bring about benefits that - while not _problems_ per se in the current paradigm of tournament-writing - would nonetheless be useful. For example:

• Integration with some question-database to help you find those late-middle and giveaway clues for a given tossup
• Quick-links to authoritative reference sources to be consulted for somewhat earlier clues (e.g. for math topics, Wolfram MathWorld)
• A topic-suggestion engine of some sort, trained on existing packets (particularly useful for high school tournaments, I would imagine)
• Semi-automated feng shui checks - so that one can, for example, tag literature questions by their tradition and genre, and then impose constraints like "each packet must contain at least one drama question and at most one African literature question"
• An auto-formatter that generates forum-friendly BBcode for posting questions

I'd be happy to talk about the general ideas I had for data architecture / schemata, or about Ginseng (in a non-trade-secret-revealing way, of course).

I'd love to hear anything you have to say about this. In terms of data architecture - I haven't thought _too_ deeply about this yet, but it seems like there should be two basic types of data underlying a system like this, namely 1.) questions, and 2.) packet templates [by which I mean a thing that says "in packet 1, tossup 1 is on George Washington; tossup 2 is on Billy Shakes; ...; in packet 2, tossup 1 is on ..."]. I've been thinking about having questions be version-controlled XML (which would be exposed to the user as plaintext - the XML is generated on the backend to denote e.g. "this part of the question is the question, this part is the answer, this segment is in power, etc.") with metadata attached for things like comments, tags, etc. Packet templates can be whatever, really, with the important feature being that they can either be manually imposed or automatically generated based on constraints.

Everything else (an answer document, the actual packets, etc.) should be constructible from these two items, plus some auxiliary items like styling directives (probably in LaTeX with some sort of friendly interface layer).

[rajk]

I'd be willing to help with this project and maybe provide hosting.

Another advantage of having XML templates is that other software (Quinterest, Protobowl, etc.) can easily parse the packets for questions and answers - I believe Kevin of Protobowl is working on a public parsed packet repository right now.

[AKKOLADE]

Any such project would have the inherent flaw of leaving access available to the questions to whoever is hosting the site.

[Excelsior (smack)]

Any such project would have the inherent flaw of leaving access available to the questions to whoever is hosting the site.

Well, in principle, it is possible to encrypt everything client-side so that the server operator never sees any of your data (cf. for example Mega.co.nz), though this still requires that you either 1.) have read and understood the entirety of the client-side implementation, or 2.) trust that the server operator isn't harvesting your encryption keys. This would, of course, be trickier to implement, and I foresee this sort of thing as being incredibly low-priority.

In any case, I don't see this as being a huge risk factor. Packet sets are routinely distributed to tens of people around the country prior to their clear dates for reasons of mirroring, and we do not, in practice, see leaks occurring. Giving one extra person (the server operator) access does not seem like it should cause much worry. Supposing that I do manage to get this up and running, though, I would in practice forbid its use for tournaments that I would be eligible to play. (But I'm not going to be in college for a little while, so this would only really apply to open tournaments, which are a small fraction of all the tournaments that get written.)

[whaatt]

The basic idea here would be to develop something that is like Ginseng, but tailored for groups of people writing one-off tournaments - i.e. most college and high school independent tournaments, which do not need structured support for question banks, privilege/role management, and similar features that HSAPQ and NAQT definitely do need. It would probably make most sense to start with support for housewrites, and then introduce baked-in support for packet submission later. This would then be made freely available to all comers.

This sounds awesome! Raleigh Charter's yearly housewrite is collaboratively written using Garuda, an open-source project I embarked on coding about one-and-a-half years ago; to date, it's been used (decently successfully, I guess) for four quiz bowl tournaments and two science bowl tournaments. I've learned a lot in the intervening period about what does/doesn't work, and I was actually planning to commence work on an entirely revamped second iteration of my application. But if there's interest in a solution built with community input, I'd very much be willing to help with that instead. So I guess I can maybe speak to what I've observed from the perspective of a developer/administrator/editor.

• Security: Right now I am the sole individual with access to the MySQL database where all questions and user information are stored; admittedly this isn't much of a problem since most usage is strictly in-house. Multiple people have asked me if they might be allowed to install it on their own servers for their personal and group needs, and fortunately I happened to write Garuda with this sort of configuration in mind. I think that a MediaWiki or WordPress-type installable app might be the way to go for future work.

• Role Fragmentation: I split user roles into four rather poorly-named categories: Directors, Administrators, Managers, and Editors. Directors and Administrators are privileged users with the ability to change categories, edit permissions, generate output packets, and edit everything. Managers and editors deal solely with the writing and editing of questions. Editors are users able to submit questions, edit their own questions, and leave comments on other people's questions. Managers, however, have edit access to one or more specific subjects. I think that these roles could be substantially simplified, particularly with some extensive implementation of revision control à la GitHub.

• Packet Generation: So the way Garuda works is that administrators promote questions when they're ready to be used in the context of a finalized packet. Then they go to the packet generation menu and select their preferred distribution of subjects for the packets they generate. Garuda allows you to select the number of packets, number of questions per packet, and whether you want to overwrite existing packet assignments or append to existing packets. The server then randomly seeds the packets according to the selected distribution. The caveat is that users need to actually write questions to roughly match this distribution for a good packet; someone above mentioned the idea of packet templating—that is, the ability to create a skeleton of a packet and have contributors fill it in. I wish I had considered that.

• Input and Output: Garuda lets you download a packet as a cleanly formatted HTML file with a title and everything. PDF generation from HTML is getting better all the time, but there were still quirks the one time I tried to implement a Download As PDF feature. LaTeX would be cool, but so would QBML or some other semantically representative format. Anyway, my work did not include the implementation of a packet upload feature, though someone did let me know that ACF-style packets are generally very parse-able. What if everyone transitioned to structured data?

• User Feedback: The learning curve wasn't too steep from our old, email and MS Word-based way of doing things; that being said, I should have included explanatory text more copiously. One thing that my coach and other users kept asking for was more user accountability statistics—who completed their question responsibilities, how much work each person put in, who edited and approved questions, and so on—but I suppose that's a problem specific to certain groups and not necessarily others. But pretty charts and graphs of progress would certainly make things more inviting or motivating in some sense.

Sorry about the long ramble! The possibility of this kind of project excites me, and I hope this work really gets somewhere. BTW, if you're interested in trying out Garuda, check the link above. If you really want to see the mess that constitutes my code, this page might be of some use.

[Excelsior (smack)]

Garuda

Oh, this is awesome. I haven't looked over your code in detail (though I probably will do so at some point), but here are some brief thoughts about the points you raised:

Security - I don't think that making per-site installations easy is something that is worth spending too much time on. I think it is inevitable that an application like this is going to have its fair share of bugs, and it'll be a _lot_ easier to deal with them if you have only one centralized location at which the application exists.

Roles - For the small one-off projects that Garuda and my hypothetical application will presumably deal with, I think that the role system needs to only be two-tiered (administrators and everyone else). In practice, in all the writing things I've been involved with (which take place over Google Docs, which lacks anything resembling privilege control), this problem has been effectively dealt with by just telling people not to edit other questions if they weren't supposed to. Assuming you trust the people on your writing team and/or know where they live so you can go yell at them in person (which I imagine will be the case for small projects), this should avoid the need for granular control over writing and editing privileges. I do agree that you need an "administrator" role who should have additional access to things like adding people to a project and so forth. Like you suggested, implementing some kind of revision control (possibly just an interface to git/Mercurial or SVN) would be very helpful here.

I/O - the primary reason I'm thinking about piping this directly through LaTeX is that PDF generation via LaTeX is more sophisticated than any other (free) tool. HTML -> PDF isn't there, and may never be, because of fundamental differences between the two formats. (To address the concerns of LaTeX luddites - this would not require you to know how to read and edit LaTeX, since drop-in templates would be provided.) That said, there is probably also value in generating HTML output - PDFs are basically read-only, while HTML is manipulable. I'm not sure what applications this would have (since packet-consuming software like databases would probably benefit more from a purely-structured, non-user-readable representation), but I'm sure they exist, e.g. Abacus-like reading applications, which can probably consume HTML more easily than they can consume .docx/.pdf (which probably requires a Scribd-like external service).

Feedback - I think there is generally high demand for user-accountability data, because pretty much every tournament has its share of flaky writers (like me) - that will be a high priority for me. Other than that, you're very right about the need for explanatory text (and also intuitive interfaces - not saying that yours isn't; just in general). The more I read about user interface/experience design, the more I come to realize that intelligent people can still be pretty bad at navigating unfamiliar technology.

[grapesmoker]

Thanks to Raj Kesavan for alerting me to this thread. I had been working on a Ginseng-like project for collaborative question editing, and I've gotten maybe halfway to something usable before I had to shelve it to focus on my editing work. You can find what exists of it here: https://github.com/grapesmoker/QuEST. It's a Django project (probably needs to be updated for latest Django versions) that uses a MySQL backend for storage; if you speak Python, everything that happens here should be pretty easily comprehensible to you. Feel free to take it for a spin or use it as a base for your own projects. I'm hoping that once Nationals passes and I'm no longer editing, I can shift my focus to this project and bring it to the appropriate level of maturity.

[grapesmoker]

Any such project would have the inherent flaw of leaving access available to the questions to whoever is hosting the site.

Whoever hosts would have to be someone who doesn't really play tournaments anymore. I'm not sure if I want to volunteer for that position, but it's a post I could fill.

I think we're really at a place right now where there's a lot of technical talent floating around and trying to do something good for quizbowl. I suggest that come summer, we have a little tech discussion in this subforum to try and coordinate our efforts and see if we can come up with a common vision so that we can avoid duplicating efforts.

[Excelsior (smack)]

Thanks to Raj Kesavan for alerting me to this thread. I had been working on a Ginseng-like project for collaborative question editing, and I've gotten maybe halfway to something usable before I had to shelve it to focus on my editing work. You can find what exists of it here: https://github.com/grapesmoker/QuEST. It's a Django project (probably needs to be updated for latest Django versions) that uses a MySQL backend for storage; if you speak Python, everything that happens here should be pretty easily comprehensible to you. Feel free to take it for a spin or use it as a base for your own projects. I'm hoping that once Nationals passes and I'm no longer editing, I can shift my focus to this project and bring it to the appropriate level of maturity.

Ah-ha! I _thought_ you were working on something like this! This sounds great, and I no longer see any reason for me to start my own project.

I suggest that come summer, we have a little tech discussion in this subforum to try and coordinate our efforts and see if we can come up with a common vision so that we can avoid duplicating efforts.

We should _definitely_ do this. Quizbowl is far too heavily littered with the corpses of failed software. Perhaps just to get people thinking, I see three major and immediate needs for quizbowl software - tournament creation (i.e. this), scorekeeping (first, an SQBS replacement, and then, perhaps, a fully paperless system), and question practicing/viewing (e.g. Quinterest, etc. - you're working on a database too, aren't you, Jerry?).

(My Django chops aren't great [though I've worked with jinja2 a bit], but I'm quite fluent in Python, so I would love to contribute some code to QuEST once a coherent vision for it has been hammered out.)

[AKKOLADE]

Any such project would have the inherent flaw of leaving access available to the questions to whoever is hosting the site.

Whoever hosts would have to be someone who doesn't really play tournaments anymore.

Yeah, that's definitely what I was trying to imply.

[grapesmoker]

i.e. this), scorekeeping (first, an SQBS replacement, and then, perhaps, a fully paperless system), and question practicing/viewing (e.g. Quinterest, etc. - you're working on a database too, aren't you, Jerry?).

Yeah, I actually have three separate projects that are intended to be part of a complete suite of tournament management, submission, and scorekeeping. Right now, they are essentially independent, but the goal is that they should be seamlessly integrated. You can find all of that stuff on my Github (username grapesmoker); the scorekeeping projects is called qscore, and the database is called qbdb, just like my old one was.

[Victor Prieto]

I don't know anything about computer science, but wouldn't it make more sense for this to be a downloadable, offline program, like SQBS? It wouldn't really a collaborative tool as much as a subject organizer and packet compiler. If this is indeed a program intended for housewritten tournaments written by under 10 people, presumably writers can just email properly formatted questions to one person, who would be in charge of adding them to the program. This also solves the problem of question security and having a permanent host.

[grapesmoker]

I don't know anything about computer science, but wouldn't it make more sense for this to be a downloadable, offline program, like SQBS? It wouldn't really a collaborative tool as much as a subject organizer and packet compiler. If this is indeed a program intended for housewritten tournaments written by under 10 people, presumably writers can just email properly formatted questions to one person, who would be in charge of adding them to the program. This also solves the problem of question security and having a permanent host.

The problem with this is that you need packets to be in a machine-readable format. If you have a web interface that people are entering stuff into, then you can guarantee that their entries will be represented in a certain way on the back-end and also guarantee intercompatibility with any other program that can read plain text. Unfortunately, you cannot guarantee this with something like Word or Google Docs in any way. So you need a structured interface that prevents people from doing things like introducing various formatting irregularities or what have you.

[Cody]

Any tournament writing software has to be online (or sync from a central database, but that would be a tedious/bad solution to an already solved problem). Without an online system, you have more points of failure and you lack the ability to do things like search for repeats (among many other necessary things).

Plus an online program is easier for the application writer and the user.

[whaatt]

you lack the ability to do things like search for repeats

This is huge. Right now Garuda does not do this, and it has created minor problems for Raleigh Charter's tournaments after the packets have been generated. I envision having something like what you see on StackOverflow's question submission form when you start typing various keywords.

I don't know anything about computer science, but wouldn't it make more sense for this to be a downloadable, offline program, like SQBS? It wouldn't really a collaborative tool as much as a subject organizer and packet compiler. If this is indeed a program intended for housewritten tournaments written by under 10 people, presumably writers can just email properly formatted questions to one person, who would be in charge of adding them to the program. This also solves the problem of question security and having a permanent host.

The problem with this is that you need packets to be in a machine-readable format. If you have a web interface that people are entering stuff into, then you can guarantee that their entries will be represented in a certain way on the back-end and also guarantee intercompatibility with any other program that can read plain text. Unfortunately, you cannot guarantee this with something like Word or Google Docs in any way. So you need a structured interface that prevents people from doing things like introducing various formatting irregularities or what have you.

Plus for many HS housewrites where the average question-writing ability isn't necessarily super high, the ability for experienced team members to edit and provide feedback is crucial. What would be cool is if this were implemented as a Google Docs-style highlight/annotate function.

I see three major and immediate needs for quizbowl software

If there were a full-featured, highly-integrated suite of software, it would revolutionize quiz bowl. Perhaps it could even exist under the auspices of HSQuizBowl, but that's just me thinking out loud. The main hurdle would be to work out an organized, user-responsive system in which dozens of QB developers could contribute collectively; I suppose that's what GitHub does well. Fragmentation currently reigns in the world of QB apps, and I think Jerry's projects—or something entirely new, if that's what the consensus says—might be a good place to start for something more seamless.

first, an SQBS replacement, and then, perhaps, a fully paperless system

RCHS uses in-house tournament manager software to completely run our tournaments from registration to results. Teams come in the morning, have their pictures taken and uploaded at a registration station; game pairings are then projected on screens throughout the school before every match. The software allows for graphical point-and-click scoring for various game formats, and a central waiting area displays live team and individual stats from every room of the tournament. At the end of the day, it can be configured to automatically FTP an SQBS-style report to a location in the cloud. I'd love to see something similar replicated and integrated with a hypothetical suite of software, particularly if it were also browser-accessible.

[rajk]

I'd love to see something similar replicated and integrated with a hypothetical suite of software, particularly if it were also browser-accessible.

This is getting somewhat off-topic, but I do have software like this almost completed and integrated with Abacus.
Features remote statistics upload and anytime editing of any player/team/tournament name, adding players, teams, etc. There's no need for moderators to write down individual player's names, they just click the team name and everything is loaded from an online database.
Here's some screenshots: http://i.imgur.com/048VECR.png, http://i.imgur.com/lgHUisS.png, http://i.imgur.com/5WFPPUz.png
If anyone wants to test these features, let me know.

In regards to the security issue, I agree with Jerry - any system that stores quizbowl packets before they're cleared should not be run by anyone in the current circuit.

[Corry]

I'd love to see something similar replicated and integrated with a hypothetical suite of software, particularly if it were also browser-accessible.

This is getting somewhat off-topic, but I do have software like this almost completed and integrated with Abacus.
Features remote statistics upload and anytime editing of any player/team/tournament name, adding players, teams, etc. There's no need for moderators to write down individual player's names, they just click the team name and everything is loaded from an online database.
Here's some screenshots: http://i.imgur.com/048VECR.png, http://i.imgur.com/lgHUisS.png, http://i.imgur.com/5WFPPUz.png
If anyone wants to test these features, let me know.

In regards to the security issue, I agree with Jerry - any system that stores quizbowl packets before they're cleared should not be run by anyone in the current circuit.

Yeah, and it's super-cool. Everybody should try this out now.

[VinaiRachakonda]

Can someone explain to me how to use this? I am having trouble understanding the webstie

[rajk]

Can someone explain to me how to use this? I am having trouble understanding the webstie

There's documentation at the top-right. Message me at [email protected] for more information so we don't hijack this thread.

On topic, does anyone want to set up a Google Hangouts meeting so we can get this started? Perhaps it could be best done after ACF Nationals, though.

[grapesmoker]

Can someone explain to me how to use this? I am having trouble understanding the webstie

There's documentation at the top-right. Message me at [email protected] for more information so we don't hijack this thread.

On topic, does anyone want to set up a Google Hangouts meeting so we can get this started? Perhaps it could be best done after ACF Nationals, though.

Yeah, after Nationals is over, I'll start a more general thread about tech projects in quizbowl. We can use that thread to organize some ideas and then transition to a Hangout or whatever else seems appropriate.