How to set up a new PC in 12 Steps, or How I spent my evening renewing my disgust with Windows

Step 1: Spend 30 minutes unpacking boxes, peeling plastic, and connecting cables.

Step 2: In breathless anticipation, press the power button.

Step 3: Spend another 30 minutes hunting for the Windows Product Key so you can access the computer you just bought. Find it, finally, on an indelible sticker on the far side of the computer's case.

Step 4: Enlist an assistant to type the Windows Product Key while you hang upside down under the desk using a flashlight to read it out.

Step 5: Insert CD to install hardware drivers, because Windows does not know how to use the network card in your PC.  Try to convince Windows that you know what you are doing and yes, you really want to run that program from the CD.

Step 5b (optional): Wonder at how Windows has not only failed to improve, but has actually gotten worse in the 10 years since you last bought a PC.

Step 6: Using a clunky-looking "wizard" from the CD, attempt to connect to wireless network. Be unable to find your wireless access point in the list because you live in a crowded apartment building, and the list is sorted randomly rather than by signal strength or even alphabetically. Notice that the list has multiple pages, and advance to page two. There it is.

Step 7: Enter password for wireless access point. Curse in frustration when it fails to connect. Blush with embarrassment when you realize CAPSLOCK is on. Turn CAPSLOCK off and try again.

Step 7b (optional): Curse the inventor of the CAPSLOCK key.

Step 8: Start Internet Explorer. Type "google.com/chrome" into the location bar to download a real browser. Try to convince Windows that you know what you are doing and yes, you really want to run that program.

Step 9: Sign into Google Chrome. All extensions and bookmarks are automatically synced. Awesome.

Step 10: Using Google Chrome, visit www.ubuntu.com and download the Windows Installer to install a real operating system. Try to convince Windows that you know what you are doing and yes, you really want to run that program.

Step 11: Let the installer reboot into Linux. Be amazed at how all the hardware is recognized immediately, including the wireless card. Feel like Ubuntu just gave you a warm hug when the wireless network manager pops up on the screen and offers to connect you to your very own wireless access point if you will be so kind as to enter the password. Check to ensure CAPSLOCK is off. Enter password.

Step 12: Click "Install Updates" when the update manager offers to do so. Wait.

Step 12a (optional): Write a blog post about your experience while waiting for updates to download. Feel sorry for people who have not yet discovered Linux.

A Framework for Innovation

How does a large company create an environment that encourages and leverages internal innovation? Here is my checklist of prerequisites for "enterprise" innovation:

Great people. You may think this goes without saying, but it cannot be emphasized enough. You cannot hire drones who put in 8 hours for a paycheck and then head out the door. You need passionate, creative people, people who love their work, people who are impatient with "getting by" and want to be the best at what they do. These are the Innovators. Without them, innovation does not happen.

A clear vision. Innovation happens at the edges. It is not a top-down directed process, it is an organic, bottom-up growth. In order for the innovators at the edge to produce innovations that are relavent to the business, top management must articulate and communicate a clear vision for the direction of the company. If the innovators can see the direction, they will innovate in that direction and get you there faster. If not, they will innovate in random directions, and you won't get the full benefit of innovation. A clear vision is the difference between innovation and distraction.

Spare capacity. Innovation is experimentation. Innovators need time to experiment, and they won't have that if 100% of their time is allocated to executing your current plan. This is the hardest thing for top managers to accept, but it is absolutely essential. You need slack time, or there simply will not be any innovation. Allocate one slice of your capacity for executing the plan. Reserve a second slice for unplanned work and process improvements. Allocate a third slice explicitly to innovation. The relative size of the slices will be entirely dependent on your own business and your desired outcome. My personal preference is 50/30/20.

Freedom to make decisions. Innovators by definition have to make decisions, make changes, form partnerships, and allocate resources from the pool of spare capacity. If permission is required to accomplish these things, then innovation will be quashed before it can succeed.

Accessible Business Intelligence. If you are going to give innovators permission to make decisions, you must give them the information and tools they need to fuel decision making. Innovators need transparent access to customer data, product data, sales data, cost data. Without it, they are shooting in the dark, and the chances of success are low. Innovators also need easy access to tools for gathering their own data, for evaluating experiments and measuring success vs. failure.

Freedom to fail. Innovation is experimentation, and experiments, by their nature, do not always have the expected outcome. When Innovators exercise their power to make decisions, some of the decisions will be wrong ones. Innovators need to feel secure that they will not be punished for taking a chance if it doesn't work out. Remember, these folks are corporate employees, not risk-taking entrepreneurs. They don't stand to make millions if their innovation succeeds, so they shouldn't have to give up their health coverage and pension if it doesn't. Make it clear that failure is a learning opportunity, not a firing offense.

The above are a few requirements for fostering innovation in large companies. Ultimately, innovation only happens where the culture supports it. Managers at all levels build company culture through their hiring and firing practices first, and management styles second. If your managers fear the new and different, your culture will never innovate. Ensuring the above factors at all levels of the organization should help to unchain your hidden innovation potential.

What's missing from this list? How does your company encourage (or discourage) innovation? Drop me a note in the comments.

Systems and Mental Deficiencies

I was surprised when I read some of the things writer Terry Pratchett wrote or said about developing PCA, a form of dementia. I cannot now find the original source that I read, but there are several similar articles. He described some symptoms of the disease slowly robbing him of his own mind. The inability to see certain objects when they are right in front of you. Walking into a room but having no memory of why you went there in the first place. Difficulty comprehending written text despite recognizing every letter and word. Difficulty recognizing people's faces.

I was surprised when I read this, because these "symptoms" have affected me, well, pretty much my whole life. I thought they were normal.

Since I was a child, I have had these problems. Until I have met people many times, I may have difficulty recognizing or remembering them. Sometimes I put something down, and then simply cannot find it again, despite the fact that neither I nor the object have moved at all. I stopped reading books on paper years ago because I just couldn't manage to read one through; I would get distracted in the middle of a paragraph and forget what I was reading. And as for walking into a room and forgetting why you're there? Hardly a day passes without such an event in my life. Sometimes I return to the room two or three times before I manage to complete the task I set out to do. Sometimes I never remember what I was planning to do.

Part of my obsession with systems is the result of this bizarre array of mental quirks that I have slowly realized are not entirely "normal" (whatever that means). Systems are a simple set of rules that I can keep in my head. Lapses in memory become less important when the system is in operation.

Ever lost your car keys? It's bad enough when you can't remember where you left them. It's doubly bad when you can be staring right at them and not see them. (My family calls this quirk "object-blindness" and it drives them crazy. "Why did you put away every dish in the kitchen except that one?!" I didn't see it!)

I no longer have to remember where I left my car keys; the rule says that keys are by the door. If I forget where I left them, I can remember the rule. If I notice them sitting somewhere else (which is rare), I move them into compliance with the rule. When I'm about to put them down, I remember the rule, and I put them by the door. One consistent rule to be applied in all situations. A rule that doesn't even have to be remembered, because it can be derived again and again. (What's the best place to leave the keys? What's the first place I could put them down after entering the house?)

I live my life by rules like this. I have a system for everything I do. A system for packing my laptop bag to be sure I don't forget anything. A system for loading the dishwasher. A system for making breakfast. It sounds ridiculous, but I have rules for all these things so that I don't have to rely on my memory to get them right.

The most important element of my life systems is an attitude I learned only in the most recent quarter of my life, a skill Buddhists describe as mindfulness. Mindfulness means being fully present, in the moment. It releases you from remembering the past or worrying about the future and focuses your attention on the present, the here and the now. Mindfulness is what enables me to obey the rules now, as I perform the actions which later may confound me. It is what allows me to think when I put my keys down and observe the rule of where to place them.

By knowingly placing my keys by the door where they belong, I ensure that later I can find them again. By paying attention to my actions in the present, I ensure success in the future.

Django Settings: Three Things Conflated

If you work on a large Django project, there's a good chance that you would describe your settings file as "a mess" (or perhaps you use harsher language). You may even have broken your settings out into a whole package with multiple files to try and keep things organized. We're highly skilled and organized developers, how does this happen to us?

I believe part of the problem is that the "settings" bucket holds three different kinds of things without differentiating between them. If you make a clear distinction between these things in your own mind (and in your code), dealing with settings will become easier, if not easy.

Project composition

The first class of settings comprises those used for project composition. One of the killer features of Django is that projects are composed of independent modules (apps). The most important settings in your project's settings file define what apps make up the project and how they interact with each other. In other frameworks this would be done with code (well, technically settings are Python code), but in Django this is treated as configuration. Things like INSTALLED_APPS, MIDDLEWARE_CLASSES, and TEMPLATE_CONTEXT_PROCESSORS define how the components of your project are combined to achieve the desired functionality.

Settings whose values are (possibly a list of) Python modules normally fall into this category.

External resources

The second class of settings comprises those used for connecting to external resources. This is the area most broadly recognized as configuration. Settings like DATABASES and CACHES fall into this category. These are the things that The Twelve Factor App says should be provided by environment variables, and in fact it's not that difficult to pull these values into your settings from the environment.

In addition to the obvious dictionaries defining pluggable back-ends, any setting whose value is a file system path or a URL likely falls into this category.

Tunable parameters

The final class of settings comprises tunable parameters, things that are mostly constants or variables that are abstracted out because 1) hard-coded values are bad, and 2) you (or users of the code) might want to change the values from the defaults. Things like CACHE_MIDDLEWARE_SECONDS, DEBUG flags,  DATE_FORMAT, and so on are examples of tunable parameters.

This is the area of greatest multiplication. Virtually every app you pull into your project is going to have some tunable parameters.

Conclusion

Armed with the understanding of the three kinds of values in your settings, you may now be able to devise a superior method of organizing them. You might start by sorting your settings.py file into three sections. Or you might decide to break them out into separate files in a package. Maybe you'll start using different tools to manage the three types of settings differently. I don't know, I don't have the solution to this problem right now, just this one nugget of insight.

What successful methods have you used to organize settings in large projects?

Heroku and the Twelve Factor App: Architecting for High Velocity Web Operations

A while back I wrote that infrastructure should be delivered as code along with every web application, because web applications are not run by users, they are operated on behalf of users, and are therefore incomplete without the infrastructure needed to operate them. In that article, I mentioned Heroku, a platform-as-a-service company that makes a living operating other people's web applications. Inspired by their experience in web operations, some of those folks recently wrote a guide to creating web applications that can be operated easily. They call it The Twelve Factor App.

There is a great deal to be learned from this 12 Factor guide and the platform Heroku has designed. Their business depends on consistent, repeatable, and successful deployment and operation of web applications, and they have this stuff precision-cut and well oiled. The guide, and the Heroku platform, make a clear distinction between what is part of the platform, and what is part of the application. Even if you are heeding my earlier advice and delivering infrastructure with your applications, you will benefit from understanding the points of separation 12 Factor recommends between your application and the platform on which it runs.

I had been planning to summarize each factor here, but the descriptions at the web site are sufficiently concise that summary seems redundant. Just click through the links for each factor and read, it will only take you a few minutes, and it will be well worth your time.

1. One codebase tracked in revision control, many deploys.
2. Explicitly declare and isolate dependencies.
3. Store config in the environment.
4. Treat backing services as attached resources.
5. Strictly separate build and run stages.
6. Execute the app as one or more stateless processes.
7. Export services via port binding.
8. Scale out via the process model.
9. Maximize robustness with fast startup and graceful shutdown.
10. Keep development, staging, and production as similar as possible.
11. Treat logs as event streams.
12. Run admin/management tasks as one-off processes (see 6 above).

Natural Laws

I figure any phrase that people deem to be a "law" and find important enough to attribute to a specific person (even if incorrectly) probably contains some real wisdom. Here's a collection of Eponymous Laws from Wikipedia, all of which I have found to be true in my own experience.

Amara's Law: We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.

Conway's Law: Any organization that designs a system will inevitably produce a design whose structure is a copy of the organization's communication structure.

Gall's Law: A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system.

Parkinson's law: Work expands so as to fill the time available for its completion.

Law of the Instrument or Maslow's Golden Hammer: It is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.

Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law. (I've also heard this restated as "Every task takes longer and costs more than originally estimated.")

Occam's razor – "Entia non sunt multiplicanda praeter necessitatem." Literally, entities are not to be multiplied without necessity. When two explanations are offered for a phenomenon, the simplest full explanation is preferable. (Or in modern terms: Keep It Simple, Stupid!)

Pareto principle – 80% of consequences stem from 20% of the causes.

Schneier's law – Any person can invent a security system so clever that she or he can't think of how to break it.

Sturgeon's law – Ninety percent of everything is crap.

Efficiency: Enemy of Innovation?

The science of management in the industrial age was all about efficiency. It had to be. The whole concept of capitalism is based on efficiency. An entrepreneur acquires capital at a cost, and that capital must be made to produce profit at a rate higher than the cost of capital. If you borrowed money at 10% to start your business, you had to make it earn 11% at least. That meant controlling costs ruthlessly and milking every bit of productivity from every penny's worth of capital.

But talk to a systems administrator about efficiency. She'll tell you that, in terms of percentage of server utilization, there are two numbers you never want to approach, numbers that will cause midnight pages and pale-faced panic. The first number, of course, is 0%. Everything is down! The second, more surprising but equally frightening number is: 100%! At 100% utilization, everything breaks, because you have no more capacity for work.

Now a capitalist might look at a well-run data center, and his first instinct is, "Look at all this waste! Half these servers are sitting idle most of the day." But the clever sysadmin will tell him that spare capacity is what keeps the data center running. If your capacity is 100 requests per second, a 101st request can bring the whole system to a screeching halt. 100% and 0% are equally disastrous. If you want your Internet business to operate, you must have spare capacity.

Now, I'm not arguing that efficiency is somehow evil. If you are in a capital intensive business today, you still need to use that capital efficiently. Of course the capitalist theory goes that capital + labor = profit, but what often gets lost in the quest for efficiency is the fact that people are not labor. That's a false assumption, and that formula was never correct (which should not surprise anyone given its source). It's not labor that turns idle capital into profit; it's creativity and its more productive sister, innovation.

In order to innovate, in order to create, you need some very special ingredients. First, you need people. Smart people, with a desire to solve problems, the ambition to tackle big ones, and the hubris to believe that they can do something better than everyone who has come before them. These people then need time to analyze the problem and devise or improvise solutions, and they need resources (read: money) to test those solutions.

So no, efficiency is not necessarily the enemy of innovation. Saving time and money on existing processes creates spare capacity that can be allocated to innovation. The extra people, time, and money that are not being used to operate the existing business, instead can be applied to solve the next big problem and give birth to new lines of business. But too many business leaders still see people as labor which, if not making capital productive, they label as "waste". Spare capacity is inefficiency in their eyes. They see the tools of innovation as inefficiency, and so they attempt to eliminate it. With the result that they eventually become irrelevant because the industry has passed them by.

Don't fall into this trap at your company. In established processes like manufacturing, efficiency creates value. In exploratory processes like innovation, efficiency destroys value. Use efficiency to generate spare capacity from your established processes. Then, use that capacity to tackle big problems. To stay relevant and keep growing, accept that creativity is inefficient, and pay the cost to gain the future rewards.