Problem Preparation Guide

What is This Document?
Contest Basics
Problem Basics
The Problem Statement
Problem
Input
Output
Limits
Sample
(Optional) Explanation
The Input/Output Generator
The Input Veri er
Sample Solutions
Sample Incorrect Solutions

Creating the Problem

Di culty
Floating-Point Numbers
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Floating Point Numbers
Big Integers
Enforcing Tight Limits
Multiple Correct Answers
Problem Preparation Roles
Clari cations
What If I Make a Mistake?

Google Code Jam-Speci c Rules

Code Jam Basics
Philosophical Musings
Input and Output File Sizes
Running Time
Small vs. Large
Judging

Authors
Comments?

What is This Document?

What is it?

This document describes the process used to prepare problems for Google Code Jam, with two aims in mind. One is to shed some light on
the process for anyone who is interested; we use the guidelines here internally. The other is to help contest organizers who are interested in
running an algorithmic programming contest of this sort. Every year, new coaches of ACM ICPC teams embark on the process of preparing
their first contests. We speak from years of experience when we say guidance can make that much easier.

What isn't it?

A guarantee. There's no promise, explicit or implied, that we will follow these guidelines in preparing future Code Jam problems. This
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 A guarantee. There s no promise, explicit or implied, that we will follow these guidelines in preparing future Code Jam problems. This
document also isn't a book of absolute rules; feel free to ignore any part of it, but bear in mind that a lot of the more specific pieces of advice
come from hard-won experience.

Contest Basics
This guide should be helpful in contests where contestants are asked to solve algorithmic problems in a limited period of time. The
contestant's job should be to devise an optimal solution to a problem; the problem writer's job will be easiest if there is only one correct
solution. Examples of such contests include Google Code Jam, ACM ICPC, TopCoder Algorithm Competitions and IPSC.

Problem Basics
Problems should be algorithmic in nature. Problems with standard algorithms like Dijkstra, minimum spanning tree, greatest common divisor,
etc. are fine, but the problem should not just be to implement a standard algorithm. The best problems are interesting, and not generic: they
require creativity to solve, rather than just knowledge. "Write a program to solve this puzzle" is better than "compute eigenvalues of a matrix,"
even if the puzzle is solved by computing eigenvalues.

Problems should generally not require knowledge from the outside world (e.g. the rules of baseball) to be solved. Code Jam in particular is a
global competition, and it's easy to assume a level of common knowledge that simply isn't there.

Every problem should have the following components:

Problem statement.
Input/Output generator.
Input verifier (really).
Sample solutions.
(Optional) Incorrect sample solutions (that you know are incorrect).

We'll talk about each of those in turn.

The Problem Statement

This is the specification of the problem that users will see. Preparing this is an art. Here are the pieces you'll need:
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 This is the specification of the problem that users will see. Preparing this is an art. Here are the pieces you ll need:

Problem
A description of the problem, probably starting with some sort of motivating back-story. Describe precisely what problem the user is going to
try to solve. If a diagram would help, make one, even if it's just ascii art.

Motivate it

The best problem statements start by motivating the problem in some non-mathematical way. You aren't performing operations on points in
the real plane: you're upgrading cell phone towers for your employer. You aren't maximizing the number of bends in a line of length R: you're a
freaking ninja swinging on a rope. From there you can get into specifics, and you've created a vocabulary ("cell phone towers" instead of
"points") to make the aspects of the problem more memorable. Sometimes a problem is very mathematical, but a little flavour goes a long
way. You can go too far with flavour, though. In Fair Warning our intention was to warn contestants that big integers are fair game, but the
flavour overwhelmed the problem in a very bad way.

Be clear

The most important thing is to be clear and unambiguous. Notice that "unambiguous" on its own isn't good enough. In Fair Warning one of our
limits was "ti ≠ tj for some i, j." Mathematically that's unambiguous: there is some value of i, and some value of j, such that ti ≠ tj... and that's
what we should have said instead. Not everyone is a mathematician. Not everyone is a native English speaker. In the last hour of that contest
we got five clarifications from people who had seen the testdata and complained that it was invalid, because in some cases ti = tj.

The same round's Snapper Chain was much easier for people who knew that a snap is an action that creates a sound; the people who didn't
know that were completely confused.

Be unambiguous

What is a triangle, anyway? Is ((0,0), (0,0), (0,0)) a triangle? Do the three points have to be distinct? If we're counting triangles, do (a, b, c) and
(b, a, c) both count? If the vertices all have to be inside a circle, is it OK if they're exactly on the edge?

This is why we have three people look at a problem statement. Of course you know what you meant when you wrote it. Hunt for ambiguity
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 This is why we have three people look at a problem statement. Of course you know what you meant when you wrote it. Hunt for ambiguity
with a magnifying glass, and ask your co-preparers to do the same.

Why this is important

In a job interview or a conversation, it's OK for someone to understand a problem approximately. If I ask someone in an interview to output a
series of strings, it might not matter if she sorts them or not, because I know she could have done either -- so it can be OK to be ambiguous. If
I describe the problem poorly, she can ask me to give examples, or clarify in another way -- so it can be OK to be unclear. It can even be helpful
to test a candidate's ability to change a vague specification into a precise one.

A programming competition isn't a job interview and it isn't a conversation. If the problem is ambiguous, then a person who interprets it one
way is going to get it right and a person who interprets it another way is going to get it wrong (and probably not know why). If the problem is
unclear, then the contestant is going to spend valuable time in the contest trying to understand it and asking for clarification. In contests the
difference between "b" and "B" is usually "wrong answer" rather than "good enough". This is the wrong context for testing someone's ability to
understand and to disambiguate. While unclear and ambiguous puzzles are fun sometimes, it is unfair to reward lucky guesses when it comes
to interpreting problem statements.

Input
What exact format will the input take? In Code Jam we start almost every statement with something like "The first line of the input gives the
number of test cases, T. T test cases follow." Variable names should be bolded for easy identification. We do this to set a familiar style for all
problems, which makes it easier for contestants to scan the problem statement quickly and focus in on the particulars of the problem.

Output
What exact format should the user's output take? How precise you need to be will partly depend on your judge software. Code Jam largely
ignores whitespace, and lets you format floating-point numbers however you choose, so we don't need to specify those. Text, on the other
hand, has to be exactly right, so if you want the user to output "NO SOLUTION" you should be as precise as possible: say 'Output "Case #x: NO
SOLUTION"', not 'output no solution.' In Code Jam we start output from each test case with "Case #%d: ". More on that later.

If you're running an ACM or ACM-style contest, particularly on PC^2, formatting is sometimes considered important (an extra space in the
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 If you re running an ACM or ACM style contest, particularly on PC 2, formatting is sometimes considered important (an extra space in the
wrong place is a "format error") so you should be extremely precise with your specification.

Limits
What will you promise the user about the range of each input variable? Don't skip anything. If you have mentioned a variable, there should be a
limit on it in this section. This includes the number of test cases. Contestants have a limited time for their solutions to run, and a solution that
can solve 5 test cases in 5 seconds could look very different from a solution that can solve 100 test cases in 5 seconds.

In Code Jam we have a section for limits that apply to both the Small and Large datasets, then one for the Small dataset, then one for the
Large.

It is important to make sure that the input files match the limits stated in the problem statement. Not only must the limits be obeyed (which is
what the input verifier is meant to guarantee), but they must be tight, too.

In Code Jam, we generally enforce the honesty of the limits in our input verifier. If the number of test cases is ≤ 20, we require that there are 20
test cases (since the number of test cases can't possibly take on more than one value, we choose the worst case). If 1 ≤ n ≤ 1000, we make
sure n is 1 at some point and n is 1000 at some point in the input. We'll talk more about tightness of limits in the input/output generator
section.

Sample
The Sample section contains a valid input, and the output a correct program should produce when given that input.

Sample I/O isn't just here so your contestants can test their code: it's here so your contestants can test their understanding of the problem. If
there's a special case in your problem's specification, put it in the sample input. Sometimes it's hard to describe a problem; if someone who
doesn't read your contest's language very well might have difficulty understanding what to do in a certain case, put it in the sample input.

Of course, the sample input shouldn't be a totally exhaustive test, and sometimes there are whole categories of cases (in which the correct
behaviour is clearly specified) that the user might not think of. You'll have to decide whether your problem will be made better by including
them or excluding them.
If your problem description contains an example, it's a good idea to include it in the sample input.
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 If your problem description contains an example, it s a good idea to include it in the sample input.

(Optional) Explanation
If the sample output is not obvious, or describing it might help contestants' understanding of the problem, it is a good idea to include an
explanation of why the output is what it is. (e.g. "In the first case, you can buy one 1-gram box and two 2-gram boxes.") Don't feel obliged to
describe every case if your main interest is describing just one of them.

The Input/Output Generator

If you're going to test the contestants' code as exhaustively as possible, you'll want to generate your testdata in at least a semiautomated
fashion. In Code Jam we generate multiple input files, since contestants may make multiple attempts on the Small.

A good input/output generator is one of the most important aspects of a problem. It's a single program that can generate all input files (even,
as in the case of an ACM contest, if there's just one file) and the corresponding answers (output files).

Be honest

Always look for worst cases allowed by the specification -- those that are most likely to cause solutions to fail -- and include them in the input.
Inputs are not the place to be nice: if you want to limit the input space, the limitation must be specified in the Limits section of the problem
statement.

This is important because for the contestants, the limits specify how good their algorithm needs to be. They don't know what you're thinking,
so it would be misleading for your limits to say n ≤ 10000, but for n only to reach 1000 in your data. This would penalize good contestants
(those who spend extra time to write an efficient solution) and reward the less diligent ones (those who try their luck with a brute-force
solution).

Put succinctly, it should be possible to determine from the problem, including the limits, whether an algorithm will work or not; otherwise your
contestants are simply playing "guess what the contest organizer is thinking."

Language independence

Where possible, if you allow multiple programming languages, make it so that all languages are fast enough to solve the problem.
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Where possible, if you allow multiple programming languages, make it so that all languages are fast enough to solve the problem.
Occasionally that's impossible; in Code Jam we generally calibrate by making sure a somewhat optimized Python program can solve a
problem. Sometimes, though, there's just no way to have Python succeed with the best algorithm and C++ fail with a worse one. In those
cases we'll typically decide that sometimes in life, you just need to use a faster programming language. We try to avoid making that decision if
we can.

Why do I need to generate both inputs and outputs simultaneously?

For many problems, generating inputs from outputs is much easier than going the other way. Use this to your advantage. Having a generator
that creates inputs out of outputs is better than having another person write a sample solution (although you should have both). Writing the
generator will force you to think about the problem from another angle. If your solution agrees with your generator, this can increase your
confidence that the problem is correct.

If at all possible, avoid duplicating code in your generator and your sample solution; instead, find a genuinely different way of solving the
problem. It is a great idea to use a brute force solution to generate many (or all) small cases and then use an output-to-input approach to
generate a few large test cases. For example, if the problem is to find a minimum spanning tree, you can start with a light tree and add
random heavy edges to get a test case with a known answer. Then use your sample solution to make sure that there are no bugs in your
generator. Finally, when the two agree, ask another person for an alternate solution.

What's with the "Case #%d:" stuff?

In Google Code Jam, the output for every test case always starts with "Case #X" where X is the 1-indexed index of the case. It is a good idea
to require the output to contain case numbers because they make it easier to find errors. When two sample solutions disagree, being able to
see the case number in the diff makes it easier to pull out the right test case. This is especially useful during a contest if there is an error in
the problem, and time is precious.

It also gives submitters a measure of security: every single output file should start with "Case #1: ". We tell the user if the file they've uploaded
doesn't have that, which lets us ignore the submission and give the user a chance to upload the right file.

The Input Verifier

This is a script that checks whether a given input file is correct, according to the problem specifications. Part of its purpose is to prevent a
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 This is a script that checks whether a given input file is correct, according to the problem specifications. Part of its purpose is to prevent a
very common mistake, where the problem writer changes the input limits in the generator but forgets to change the problem statement, or
vice versa. Here is a very simple example veri er.

Do not skip this. It is incredibly easy to make an assertion in the problem statement and then accidentally fail to enforce it in the data. Be
thorough. If you say there will be no more than 100 test cases, assert it in the verifier. If you say the solution to every case will be no more than
500, include a solution in your verifier and make sure the answer is not more than 500. If you say there will be a unique shortest path, assert
that. This is particularly useful because the person writing the verifier is likely to spot if the problem is underspecified, since she has to be able
to check if it exactly meets its specifications.

You should also use the verifier to make sure the limits are tight. If 1 ≤ n ≤ 1000, make sure n is 1 at some point and n is 1000 at some point in
the input. The sole exception to this is the number of test cases, which should almost certainly be the maximum possible number; in Code
Jam we only say T ≤ 100 rather than T = 100 because we want the sample input to be valid.

In Code Jam, we get very serious about the verifier. If the first thing in the input is an integer in a line on its own, we make sure nothing
precedes the integer, make sure it fits in 32 bits, make sure it's followed by a newline... essentially we check every byte of the input in order to
make sure we're extremely strict with following the format. Different languages have different ways of parsing the input: C++'s operator<<
will happily skip whitespace in the input, but some contestants may be using getline() or Java's BufferedReader, and could be foiled by
extraneous whitespace. We go to this level of care because there are a lot of people involved and there's a lot of money at stake; for smaller
contests we don't necessarily recommend this level of care.

Sample Solutions
This is a program that can read an input file and generate a correct answer. Here is an example solution.

For a small contest you should have at least two solutions created totally independently; for an international contest with prizes, we do 3 and
make sure that if the I/O generator had a solution, it's independent from those 3. For more complicated problems we sometimes go higher.

Sample Incorrect Solutions

If one of your aims with a problem is to make sure a certain kind of solution doesn't succeed, don't rely on running-time calculations unless
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 If one of your aims with a problem is to make sure a certain kind of solution doesn t succeed, don t rely on running time calculations unless
they're very conclusive: implement your own version (preferably optimized) and run it on every input file.

Creating the Problem

Much of the problem-creation process is an art. Bitter experience has taught us that at least part of it can be a science. Read on for our take.

Difficulty
The key thing is to aim your difficulty at your audience. If you're running tryouts for your school's ACM team, you want everyone to leave
feeling like they've accomplished something. You need one incredibly easy problem. Not much past "I'll give you two numbers, you add them";
the people who only solve this might not come back, but they won't feel too hard-done-by. After that go for a problem with simple code and a
somewhat more difficult algorithm (something that requires a little insight). The ones who solve this problem are the ones with potential, and
surmounting this challenge will motivate them to come back for more. Then you can move on to the stuff that your people with some
experience will solve, and finally the problems that will distinguish your top performers from each other. This is (very roughly) what we aim for
in Code Jam's qualification round: easy; easy, but insight required; medium-hard.

On the other hand, if you're running an ACM regional or Round 1 of Code Jam, you know that most of your audience has at least a little bit of
experience. You still want an easy problem or two, but you want some nice mediums for your audience to sink their teeth into, and a hard or
two to pick the winners.

Overall your objective should be to have everyone solve at least one problem; probably for a few people to solve every problem; and for there
to be a significant gradient in skill near your cutoff (if 1000 people advance, you don't want 500-1500th place all to have the same score).

At the end of the day, nothing is going to go exactly as you expect. Particularly if you're an inexperienced problem writer, you aren't going to be
a very good judge of how hard your problems are. Sometimes, especially if you start with the solution and make a problem out of it, you'll think
about a problem for five minutes and come up with an algorithm that takes an hour when approached from the other direction; other times

you'll spend days working on something that turns out to be straight out of CLRS. That's why it's important to have other people look at and
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 you ll spend days working on something that turns out to be straight out of CLRS. That s why it s important to have other people look at and
solve your problems, and give you a sense for how tough they are.

Floating-Point Numbers
Round-off errors will happen. If you floor, they will happen. If you round, they will happen. Even if you compute your answer to 12 decimal
places and round to the nearest integer, they will still happen. Always accept some margin of error, unless you are doing something
particularly unique and creative.

If your problem requires rounding to make discrete decisions, you are almost certainly doing something very wrong! If your problem cannot be
solved exactly using only integers, you should never require contestants to make decisions by comparing floating-point numbers. Don't ask
them to print "Yes" or "No" depending on whether a triangle is inside a circle, unless all of the triangle vertex and circle center coordinates, as
well as the circle's radius, are integers.

There are exceptions here in the case where you're willing to make a guarantee about the structure of the input, but these are extremely
difficult to get right. We did this in Ninjutsu: ("There will be at least one value of r that gives an optimal solution and has the property that a
rope of length r - 0.999999 also gives the same solution"). We tried to prepare it for the 2009 finals, but couldn't settle on a way to avoid
floating point issues after a 96-email-long thread. We ended up preparing a 1-dimensional integer-only version as a backup. 80 emails later we
ended up running it in 2010. We wrote 8 correct solutions (including two using the gmp library for high-precision decimals), 2 deliberate wrong
answers (including one that was wrong because it used doubles instead of int64s to check collinearity) and a 1640-line-long generator that
made sure ropes of length r - 0.9999991 didn't work. The short version: don't do this unless you really want to run the problem and there's no
other way to ask it.

As an alternative to spending a hundred person hours preparing your problem, you could recast it. Instead of asking "What should I do to get
the highest probability of success?" ask "What's the highest probability of success if I act optimally?" That way they aren't making a binary
decision (is A better than B?) but it's clear the contestant has solved the problem, and you don't create unnecessary and difficult work for
yourself.

There's research to support this: look up condition numbers to learn more. A decision problem causes an infinite condition number; artificially
eliminating ill-conditioned regions (like in Ninjutsu) makes the condition number finite but often high; and asking for the maximum of two
numbers is well-conditioned. Roughly speaking, the higher the condition number is, the worse errors will be.
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 numbers is well conditioned. Roughly speaking, the higher the condition number is, the worse errors will be.

Big Integers
Most modern programming languages have a built-in library to deal with integers greater than 264. Unfortunately C++ doesn't have one built
into the language, and it's by far the most common language used by contestants.

Because of how common C++ is, and the fact that most programming contests steer away from BigInteger questions, we recommend against
using problems that require, or are significantly helped by, a BigInteger library. Contestants won't expect it, and it will create a significant bias
in favour of those who are comfortable with a language other than C++.

Besides, are you sure you need large numbers? Is your problem going to be significantly less interesting if you ask for the answer modulo
1000000007? If not, you keep everybody happy by going with the modulo version and letting your contestants focus on the interesting,
algorithmic part of the the solution instead of worrying about finding and using an appopriate BigInteger library. If, on the other hand, big
integer arithmetic is the heart of the problem, then perhaps you should avoid it altogether -- "Implement square root for BigIntegers" is not an
original idea, and contestants with minimal non-C++ experience will have a huge advantage over those without any.

With that said, in Code Jam we've made the decision that we will ask problems that require a BigInteger library if the problem requires it, and
the BigInteger aspect isn't central. We've given fair warning; contestants have Internet access during the contest, and are welcome to use
other people's libraries; the rounds are long enough that a few minutes spent digging up a BigInt library aren't going to kill someone's chances;
and, all things considered, in our particular contest we don't mind giving a tiny edge to people who know more than one language.

Enforcing Tight Limits

How do you distinguish between O(n) and O(n log n)? You don't. This is never a good idea, and you will never get it to work. Don't make
problems that require bucket sort or radix sort. Don't expect people to use hash tables instead of binary search trees. Don't insist on Fibonacci
heaps for Dijkstra's. Don't attempt to force anyone to use union-by-rank in Union-Find. Don't design a problem where your intended solution
uses some particular heuristic (see "guess what the contest organizer is thinking" elsewhere), or some particular constant-factor speedup.
Any number of factors can overwhelm the particular thing you're looking for, including how well-optimized the implementation is and the
user's choice of programming language.

Multiple Correct Answers

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Multiple Correct Answers
Some contests can have problems with multiple correct answers. In that case the problem preparer will have to write a judge that can
precisely differentiate between correct answers and incorrect answers. In Code Jam we have such a mechanism, but if it's possible to recast
a problem in a way that gives it a unique solution we'll often do it. Since problems with unique solutions are much more common in
algorithmic contests, you should probably mention in the problem statement that there may be multiple correct answers, and you'll accept any
one.

Conversely, you should make sure that every correct solution gets accepted. If the problem is to find a shortest path through a maze, then
every shortest path should be accepted as correct, not just one of the shortest paths. Take extra care to make sure that you have thought of
all possible correct solutions. Sometimes, it makes sense to ask for the lexicographically smallest correct solution, if you want to ensure
uniqueness. If you go that route, watch out: the problem may become orders of magnitude more difficult. For example, finding the
lexicographically smallest shortest path through a maze is tricky, and it requires a precise definition what it means for one path to be
"lexicographically smaller" than another path. Make sure to include this definition in your problem statement.

Problem Preparation Roles

For Google Code Jam we have roles that problem preparers sign up for. They are:

Statement Author: Creator of the problem statement, who works with the other two "Statement" jobs to make sure the problem
statement is as good as it can be. This person's main task is to care more about the final result than the effort he put into the paragraph
that really needs to be rewritten.
Statement Veri er: A second person to read the problem statement and to make sure it's clear. For Code Jam, we require that either the
Statement Author or the Statement Verifier is not a native English speaker. This helps us simplify problem statements (we introduced
this requirement after a particularly verbose round drove our contestants crazy). A note for the person who does this job: sometimes it
can be challenging to take a problem from your friend and tear it apart. Problems change as they're created, and that can sometimes
result in a badly-written problem statement. It's your job to fix it, and your friend will thank you when he doesn't have to deal with 20
clarifications and a lot of unhappy contestants during the contest.

Statement Finalizer: A native English speaker who will do the final check for grammar and clarity. For a smaller contest you probably
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Statement Finalizer: A native English speaker who will do the final check for grammar and clarity. For a smaller contest you probably
don't really need three people to look over the statement. In Code Jam We require that the "Statement" jobs are performed by three
different people.
I/O Generator: This person writes the I/O generator after the Statement Author is done. This person is the most likely to realize if the
limits need to change, so we require that this job is done or almost done before asking for sample solutions.
Input Consultant: This person helps the I/O Generator come up with test cases and classes of test cases.
Input Veri er: This person writes the input verifier.
Solutions 1-4: We typically require 3 (in addition to the output generation step of the I/O generator), but for trickier problems we
sometimes look for more. We ask solution writers not to talk to each other until they have finished implementing their solutions. This
means not sharing any algorithm ideas or even interpretations of the problem. We have seen cases of three people chatting about a
solution and then all making the exact same mistake in implementation.
Fake Solutions: For some problems we throw in some incorrect solutions. We serve different input files to different contestants, and we
need to make sure that incorrect solutions fail on every input file. Sometimes this is a matter of implementing a slow algorithm;
sometimes it's using the wrong epsilon, or throwing in an off-by-one. If one of your test solvers made a mistake when first trying your
problem, that incorrect solution should become a fake solution.
Editorial: This person writes the analysis after the contest to explain how to solve the problem.

At the end of a contest, we publish the analysis and publicly give credit to people who worked on the problem. "Written by" credit goes to the
person who came up with the problem in the first place; "Prepared by" credit goes to the Statement Author and the I/O Generator.

Clarifications
During the contest, contestants will ask you a variety of questions about the problems. These will range from the wholly inappropriate ("how u
do prob A?") to the helpful ("I think problem C's sample input doesn't obey the limits.") Even with a small army of people preparing problems
you can make mistakes, so be prepared to go looking for them.

Some questions you clearly shouldn't answer. If the contestant wants a hint, or wants to know how to solve the problem, the answer is no. As
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 Some questions you clearly shouldn t answer. If the contestant wants a hint, or wants to know how to solve the problem, the answer is no. As
people who have run into the wall of "Please read the problem statement" a few times too many ourselves, we have a few other canned
responses: "Sorry, we cannot give hints." is one we use in this context.

Some clarifications seek to understand the problem better. A legitimate question that is unclear from the problem statement should be
answered, possibly to all contestants, and corrected if possible. Some questions may be requests for clarification that you don't want to
answer (Q: The next number after 80 is 800 or 88? A: Please read the problem statement.). A lot of questions will be near that line: a poor,
confused contestant will ask you something that is in the problem statement, but maybe not stated directly, but you intended to be clear. We
tend to be more lenient about those in the qualification round and get rapidly stricter after that.

Other clarifications may seek to understand the contest better. Some of your contestants have probably never competed in a contest like this
before. We get asked about floating-point formats (Q: Is 6e-05 a valid answer? A: Yes.), validity of input (Q: Will the numbers in the input have
leading zeroes? A: No. Q: Does every test case have a solution? A: Yes.) and the like. Those are generally no-brainers: answer them.

What If I Make a Mistake?

Every contest organizer's nightmare is making a mistake in problem preparation. Your biggest responsibility, and the only point on which we'll
be unequivocal, is to admit it and do your best to fix it. At the end of the day, if the issue is anything beyond a minor rewording, chances are
that someone is going to feel screwed. Members of the Code Jam team speak from experience when we say that it feels a lot better when the
people responsible admit it and apologize.

The most common errors are unclear and mistaken problem statements. Your contest platform probably has a clarification system, and
chances are you found out about the error because a contestant used that system to tell you. The right move here is a matter of policy; in
Code Jam we'll correct the issue in the problem statement and move on, possibly announcing that to contestants if the correction isn't tiny. If
you're running a long contest and you're most of the way through, however, you might decide that the unfairness it creates is worse than the
improvement it would add and choose not to fix it. The "fix it" option isn't great for the people who were already working on the problem, but it
probably results in the best contest experience for as many people as possible, and at least it's a fair policy you can decide on ahead of time.

The exception to this is if the change significantly changes the meaning of the problem. Then you're going to have to decide what the most
fair thing is: do you perhaps clarify, and annul any wrong submissions that came from the confusion? Is it bad enough that you need to cancel
the contest?
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 the contest?

Another possible problem is invalid input data. This is pretty unlikely if you wrote a good verifier, but it can happen. The effect of this depends
on the contest format. In contests like Code Jam and ACM where contestants get the data, or feedback about it, during the round, the effects
of this are very similar to the judges simply having the answer wrong. In contests without immediate feedback (e.g. TopCoder) the invalid
case can hopefully just be removed or ignored (unless it's a data-based contest, and the error might have caused a submission to time out).

The biggest problem is when you, the contest organizer, got the problem wrong. If you're giving any feedback during the round, this will result
in wasted time for contestants who were misinformed. This happened to us once in Code Jam, in the problem Star Wars. In our case, the only
prize for the round was to advance to the next round. Of the contestants who weren't in the top 1000, there were only two who were
misinformed (told "Correct" when the answer should have been "Incorrect" or vice-versa) during the round. We decided to give them a bye to
the next round, where they were eliminated. That was the best we could do to make it fair without cancelling the round and running it later, and
we decided it was the better of the two choices.

As a judge, you have the important responsibility of deciding whether an issue that affected some contestants was small enough that the
round can continue without changes; large enough that some contestants should get special consideration; or huge enough that the contest
needs to be cancelled and possibly re-run with new problems. Just don't pretend it didn't happen: that violates the integrity of your contest,
and if it's ever discovered your contestants will be furious.

Google Code Jam-Specific Rules

If you're running your own contest that isn't heavily based on Code Jam, this would be a good place to stop (or jump to the "Comments?"
section at the end). Otherwise, read on.

Code Jam Basics

Some of the rest of this section won't make sense unless you know a bit about Google Code Jam. In a Code Jam round contestants usually
have 2.5 hours to solve as many problems as they can. Problems are divided into Small and Large datasets, each of which has some point

value chosen by the contest admins.

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 value chosen by the contest admins.

When a contestant has coded a solution to a problem, she can download the Small input. She then has 4 minutes to run her code and upload
the output. We'll tell her whether she got that right immediately. If not, she's subject to a small penalty (if she eventually gets it right) and can
try again. Once she gets the problem right, she can download the Large input. She then has 8 minutes to run her code and upload the output;
we won't tell her whether she got that right until the end of the contest.

Philosophical Musings
We decided to go with a data-based contest (download/run/upload) rather than a code-based contest (submit code/server runs it) because
we like contestants to be able to use whatever language they want, in whatever environment they're comfortable with. Part of that is that we
didn't want to try to sandbox (and keep our sandboxes up to date for) a ton of programming languages, particularly not since we'd have to do
it securely enough that we'd feel comfortable running them on the same machines that hold your registration data.

We also liked having the two kinds of input: Small, which has smaller limits and is judged right away, like ACM ICPC submissions; and Large,
which has larger limits and is judged at the end of the contest, like TopCoder submissions. The first gives contestants a second shot at stupid
mistakes, while the second builds suspense. 4 minutes lets a contestant with a slow connection and a confusing file system solve a problem
with some time to spare; 8 minutes lets your computer crash and come back up.

Input and Output File Sizes

Because some people have dialup, we have tight restrictions on the size of the input and output files: we don't let input files go over 200kB
(though we may change this, since modern browsers download a zipped version of the file anyway). Correct output files could theoretically be
arbitrarily long, since we ignore whitespace, but in the rules we guarantee that there will always be a correct output file under 200kB in size.

Running Time
Small inputs

A correct solution must be able to solve any input dataset in under 2 minutes on a 2GHz machine using at most 256MB of RAM. In reality,
contestants will have up to 4 minutes on their own hardware.

The reason we require a large margin of success is that contestants will be very unhappy if the difference between getting a problem right and
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 The reason we require a large margin of success is that contestants will be very unhappy if the difference between getting a problem right and
getting it wrong boils down to a choice between scanf() and fread_unlocked(), or a choice between using an STL set and
implementing a custom heap from scratch.

A solution that is too slow should run for at least 10 minutes on a 4 GHz CPU with 2 GB of memory. This may be very difficult to do as a
problem writer; for small inputs you may have to bite the bullet and allow slower algorithms than you intended to.

Large inputs

Contestants will have 8 minutes to run their programs on large inputs. The requirements remain the same: at most 2 minutes for a correct
solution and at least 10 (if possible, 30) minutes for a solution that is considered too slow.

If a contestant manages to massively optimize a sub-optimal solution and squeaks it out on the first try in 7 minutes, that's fine. It's not worth
the effort at our end, or possibly scrapping a problem, to ensure that such things are impossible; and it does come at a cost of time and risk to
the contestant.

Small vs. Large

How do you make two different sets of limits for the same problem? Sometimes the only answer is to have your super-clever optimal
algorithm be the only solution for the Large, but for some exponential brute force to work on the Small. We try to avoid that when possible,
because Smalls are worth points on their own and nobody wants to spend the last hour of a contest bashing out brute-force solutions to the
last three problems.

The best Small/Large divisions require some amount of insight to solve the Small, and perhaps more insight on top of that for the Large. In
Hot Dog Proliferation we require one insight to go from an enormous complete search to a single simulation, and then one more insight to
make this simulation fast. Another possibility is to let "case bashing" (enumeration of all possible cases) work on the Small, or even make it
small enough that someone could solve it by hand.

Judging

The parameters for Code Jam's judging system are laid out in our FAQ. Chances are you aren't writing a judge of your own, but if you are then
Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD
 The parameters for Code Jam s judging system are laid out in our FAQ. Chances are you aren t writing a judge of your own, but if you are then
we can say we're quite happy with how ours has worked. You're welcome to make use of our parameters.

Authors
This guide was written by Bartholomew Furrow and Igor Naverniouk, with input from David Arthur, Jorge Bernadas, Tomek Czajka, Ante Derek,
Luka Kalinovcic and Petr Mitrichev.

Comments?
We welcome any comments you have on this guide! You can reach us at [email protected], or on our mailing list.

Privacy Terms About Google Google Products

Create PDF in your applications with the Pdfcrowd HTML to PDF API PDFCROWD