Intro

First, a few words on the origin of this document. We've spent a lot of time playing, running, and talking about CTFs. We've also put a lot of time into the art, sport, and culture of CTFs. This addiction hobby, plus the recent open call for organizers for the DEF CON CTF, led to a number of discussions about what lessons we'd want to impart to whomever ends up running it, and highlighted a broader need for some general principles in running quality CTF events. To that end, here are some important maxims to live out when making a CTF.

Who is this paper for? First and foremost, it's for any poor fools who might be contemplating running a CTF (our condolences in advance). Secondly, it's for serious players of CTFs: other junkies who are equally hooked and will debate and discuss the points herein. Everyone else is welcome along for the ride though.

Some of this content is very high level and applies to a variety of competitions. Some of it is very specific and only relevant to certain styles of CTF. Regardless, we hope it is beneficial to many and improves the overall quality of future CTF events.

Maxims

1. We hack for fun, not for frustration.
2. The scoring mechanism should always be the easiest challenge.
3. Solutions might be a surprise, but recognizing when you have one shouldn't be.
4. When the next step requires a leap of faith, be sure to include a bridge.
5. An homage honors, but duplication doesn't.
6. Learners always win even when winners don't learn.
7. Your point estimates are exactly that until calibrated.
8. Never rely on the survival of a vulnerable server.
9. Competitors are cleverer than you, they also have more time.
10. Learning starts where prior knowledge ends.

Lessons

• Keep it stupid simple
• A fun CTF is a good CTF
• Nerds like learning
• Sucky CTFs suck
• Work to the fourth

Keep it stupid simple.

As in all good design, there is great value in simplicity. In an ideal "simple" CTF, challenges might be complex or difficult, but the game structure and scoring system should not be.

Most importantly, the scoring system should not be a challenge unto itself. When players feel they don't understand the optimum way to play, they usually feel frustrated at the game, rather than challenged by it. Players expect to occasionally feel frustrated by difficult challenges, but not the game architecture. So stick with a simple and easy to understand scoring system.

Another important application of this maxim is in the flags themselves. It is expected and encouraged that flags should be difficult to find. What should not be difficult is knowing when a flag has already been found. Make flags or keys obvious in their format or structure. If that's difficult for a particular type of challenge, see if you can rework it. Incentivizing players to constantly spam a scoreboard helps no one and only makes the game less enjoyable.

A fun CTF is a good CTF.

Speaking of enjoyable, let's not forget that we play CTFs because they're fun. We enjoy solving challenges, we enjoy learning, and we enjoy having fun doing those things. Always keep this principle in mind as you design CTFs and CTF challenges.

Making a CTF fun doesn't mean making it overly easy. It just means that challenges should leave players satisfied upon completion and not frustrated with the path they found to victory. No one likes blind guessing games, and one of the most common critiques a CTF player will level against a particular challenge is that it "required lots of guessing".

One way to keep guessing to a minimum is to leave "breadcrumbs" in your challenges. Make separable, discrete steps in harder problems that allow the players to know they're making progress. It is always enjoyable to solve a number of challenges in succession, each one leading naturally to the next and requiring new skills and techniques. As a bonus as long as you don't do something silly like require an MD5 hash of parts of a key, you as an organizer can track the progress of participants through wrong keys submitted with this approach.

Finally, while we are all inspired by others, make sure that your challenges aren't derivative. Or, if they are, do it with twists, cleverness, or as an entertaining homage. Building on the state of the art is encouraged, duplicating it is not.

Nerds like learning.

We all started somewhere and, indeed, many of us have learned and improved our skills tremendously through CTFs. Unfortunately, it's easy to forget how difficult these challenges can be without experience. We love CTF in-jokes as much as the next CTF organizer, but make sure not to leave behind the newbies in the process.

Don't confuse challenges that are easy to solve with challenges that are easy to make. In fact, making easy challenges that are accessible to a wider audience but are still enjoyable can often be a particularly difficult task.

Think back to your favorite CTF challenges. Very rarely will they be ones in areas you were already very familiar with. Invariably, we are drawn to the challenges that required us to learn something new. Go new, go old, or go unique. Avoid commonplace languages, files, and formats. Force people outside of their comfort-zone. Don't do so at the expense of fun, but the more you can challenge players with things they have to learn in the game, the more fulfilling it is in the end.

Of course, some CTFs are primarily exercises in standard, practical skills (pure blue-team exercises like CCDC, for example). In those cases, there might be less emphasis on those unique and otherwise unusual topic areas. Make sure you advertise upfront which type of game players should expect.

Sucky CTFs suck.

There is no quicker way to ruin a good CTF than through problems with quality. It is incredibly difficult to run a flawless event, but it should be our goal nonetheless.

There is only one path to that goal, and it involves a lot of time and hard work. It helps to have organizers who have played CTFs before and are therefore more likely to "get it", but there's no substitute for just putting the time into testing everything. This means that members of your team should be testing challenges they don't know the solutions for. Don't let the authors of a challenge determine the difficulty and the points, let the play-testers that solved it without insider knowledge do that.

In fact, points are notoriously hard to guess right. Unless you plan on getting a large sample of play-testers to determine an accurate point representation, be prepared for a 500 point question that's solved instantly, and a 200 point question that stumps everyone for hours.

Running a quality game also means testing your infrastructure. This is extremely critical in attack/defense CTFs such as the DEF CON finals. It is immensely hard to prepare your network for a dozen teams of hackers trying some of the dirtiest tricks on the planet, but it comes with the territory.

Have you red-teamed your scoreboard? Your hosts? No matter how many rules you put into place prohibiting teams from attacking infrastructure, you're running a hacking game. It's going to happen. Make sure you put an appropriate amount of time into testing your infrastructure against determined attackers. More than one scoreboard has been compromised or exploited over the years.

Yes, that means having your challenges and infrastructure done early enough that you have time to test them. Make a schedule and stick to it. This also means you'll stand a chance of actually starting your event on time -- another quality metric CTFs are notorious for failing.

Work to the fourth

You think you know how much time it will take to make a CTF following these maxims? No you don't. Take your estimate and double it. Now double it again. You might be right now. Do not underestimate how much time it takes to organize a quality CTF. That goes extra for events with live server-on-server components. You need evidence? Ask any of the former organizers of DEF CON CTF from the past eight years.

Acknowledgments

First, thanks to Tyler Nighswander, the team captain for PPP for his early feedback on this document and especially for many of the maxim quotes. Secondly, thanks to the many that have gone before. Kenshoto, DDTEK, 1@stplace, legitbs, and of course our current and former compatriots on Hates Irony. Additionally, there are so many other good CTF teams we have competed against and played challenges from. We stand on the shoulders of giants. Finally, thanks to Howard Taylor's Schlock Mercenary for the document theme.

Feedback

We'd like to make this a living document, updated as the culture around CTFs continues to grow, so feedback is appreciated. We're available via email to either author's handle at this domain.