ICST Paris 2010 – Innovation in Test


Last week, the ICST Wednesday morning keynote was given by Patrick Copeland, senior director of engineering at Google. The talk was about testing, culture, and what Google is doing to improve TTM and quality through innovative approaches to testing. Productivity and Innovation in a hyper-competitive cloud computing world were the key themes of the talk.

Innovation
There are those who are changers and those who are maintainers. Changers are people who are more likely to innovate. Maintainers will keep the existing running well. Innovators are changers who can take ideas and make them work in the real world. To innovate however, you need feedback mechanisms that let changers test ideas. Failing fast and early is the key. Mr. Copeland introduced the idea of “pretotyping”. A pre-prototype can give you early feedback. One example he cited in his talk was about voice recognition and dictation on a computer many years ago. You talk, the computer writes. The first demo of it was done with a microphone and a screen and as the subject spoke, the text would appear on the screen, but the text was being typed by a human who was listening in another room. What they learned from this is that talking into a computer is tiring and natural speech does not render well as written text. It provided some feedback that would have been obtained much later if a real prototype been built.

Mr. Copeland and I spoke afterwards and I described our A-TDD research proposal and he also put this in the category of pretotyping – taking a set of high-level requirements and building acceptance test cases, and sharing those with customers early, within days or a week of identifying a requirement. The Acceptance Tests don’t have to be very detailed, but enough to describe the behaviour of a feature to a customer. Describing a feature as a set of test cases is specific, and allows customers to see a development team’s interpretation of what is to be built. This will trigger conversations and collaboration to refine requirements and Acceptance Tests. Customers don’t know what they want, and therefore, providing Acceptance Tests, describing feature behaviour can help them identify what they don’t want, and eliminate waste in the development, and trigger conversation about the value of requirements.

Change
To make change happen, you need a company culture that supports the right behaviours:

  • Avoid plausible deniability
  • If you don’t have power to make things better you don’t even think about how to do it. A corollary: You may have more power than you think. If you THINK you don’t have any power, there is no motivation to try to drive change.
  • Darwin rules – and therefore a necessary component to surviving change is to have a diverse group of people with differing opinions who are not afraid to argue and debate. Homogeneity is a recipe for extinction.

The Virtual Physics of Teams
Mr. Copeland shared a metaphor of flight, and how the elements of drag, lift, and thrust creates an equilibrium that allows planes to fly. Change any one of those forces, and flight is compromised. Teams also have a natural equilibrium that allows them to perform, innovate, and deliver value for customers. So, understanding the virtual physics of teams is vital to achieving the right balance of forces for high performance. Think about the main factors that drive team performance. They include co-location and active communication, access to information and resources, impediment removal by management, quick feedback, and a normal workday. Good physics also include constraints, like time boxes, and limiting the work to avoid multi-tasking. It also includes enforcement of simple rules that constrain behaviour, but also drive innovation.

Organizational Structure
Simply put, flat and autonomous is the way to go. It can feel like chaos, and managers no longer have the ability to micromanage anything, because they have large groups of people reporting to them. The downside includes some potentially tribal team behaviour. However, on balance, this is proving to be the most productive way to work.

Code Submissions
Google has a single source repository for entire company for 1500 concurrent projects – everything builds from the head – single stream all the time – 20 code changes per minutes and 50% of the code base changes every month. The size of the code base is in the millions of LOCS.

Testing
In 2008, the average developer spent 37 hours a month waiting for tools to spit out results. The focus was on making this shorter. Google globally executes 65k builds per day or 20 million per year. Object file output is 698 TB of data, cached for 7 days. Incremental linkers are used for builds. And on these builds, Google runs 7.5 million tests every day. Most builds take about 5 minutes, but somewhat longer at the tail end when different build targets are done. 66% of the builds require no user intervention at all – fully automated. Google invested heavily in infrastructure to make this possible. With the new infrastructure, Google saved 604 engineer years – the average developer got back 2.7 months of productivity, which could be invested in innovation. Productivity jumped from 50% to 191%. Developers get feedback in minutes, not hours, if something is broken.

Testing Philosophy
Google uses humans for what humans are good at and CPUs for what CPUs are good at. You have to keep the feedback loop very fast or developers will ignore the results. Always keep the code at production quality. Integrate the tools so that one click does it all. Keep the test vocabulary simple. Google puts testes in buckets; small, medium, and large categories. Write tests for all new code and automate. Iteration over iteration, improve coverage. There is a cost, but the payback is visible and significant.

Fascinating talk and some great ideas to explore.

Advertisements

One thought on “ICST Paris 2010 – Innovation in Test

  1. Pingback: Retrospective & Happy Holidays « rajile

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s