Continous Integration and Testing Conference – by Jamie Longmuir

Field Report from Raleigh-Durham, North Carolina – April 16th-17th, 2010 – (CitCon –

By:  Jamie Longmuir

Note from Raj: Jamie just returned from Citcon and he graciously agreed to share what he learned there with all of us. Thank you,  Jamie!

This conference was an excellent opportunity to hear from other designers, testers and release managers who had adopted agile, CI, TDD, and other practices.  While the attendees all brought different perspectives, they were all very passionate about improving their software development practices.

Note: The points raised in my notes below, are just that…my notes.  Feel free to discuss any points you disagree or have something to add to in the discussion page.

Interesting Quotes from the Conference…

“Gartner’s analysts (Thomas Murphy and David Norton) predict that by 2012 “agile development methods will be utilized in 80% of all software development projects”. –

“If you’re not doing TDD, you’re not doing professional development” – Paul Julius

Conference Format – Open Spaces

I found the open-spaces concept very effective for this type of conference.  When you have many people, from different backgrounds, looking to share their individual experiences this is a great format.  Unlike a traditional conference, where sessions are pre-scheduled and presenters lecture on a particular topic, open-space sessions are scheduled by the participants during the conference, and frequently turn into a round-table discussion, rather than a lecture.


  • After introductions, all participants are invited to suggest topics for discussion.  You need NOT be an expert on a topic to suggest it.  You just have to want to discuss it.
  • All topics are placed on a schedule grid with sticky notes
  • All participants are invited to the board to indicate their interest in particular topics with a tick mark on the sticky note
  • The notes can be re-arranged by anyone at anytime before a session is to begin
  • During the sessions, participants are encouraged to leave a session that doesn’t interest them – it is better to have a session with 2 interested people than 20 bored people.


  • Topics of discussion are directed by the group as a whole – this makes for a great exchange of ideas, and makes it difficult for one person to dominate the discussion or attempt to preach/sell their methods without being challenged
  • Encourages an interaction between delegates – there is little distinction between the speaker and the audience
  • Delegates are encourged to leave sessions where they are getting no value – I never saw anyone looking bored
  • Frequently the most interesting discussions, aren’t planned…or even aren’t at sessions
  • Claim: If you don’t get what you want out of the conference, you only have yourself to blame for not brining it up.


  • Topics of discussion are directed by the group as a whole – it is unpredictable where a particular session will go, and difficult to predict which sessions will be most valuable…some luck is involved!
  • Some participants should have knowledge to share – would not work well if nobody had anything to share
  • Participants must be willing to get involved – an open spaces conference of only introverts would fail quickly
  • Having enthusiastic participants is a must – a conference of people who are there only because they have to be, would also likely fail

Pair Programming

Paul Julius had a very good session on Pair Programming, where a number of us tried different techniques for pair programming.  He has worked with the teams where designers really do program for a full 8 hour day.  I don’t foresee (or want) us to adopt such practices, but I do feel Pair Programming could be an valuable training tool.

Benefits of Pair Programming

  • Improved Quality is inevitable (PP is a continuous code inspection)
  • Biggest Benefit:  “Reduce the bus factor”
  • Transfer knowledge, spread code knowledge around
  • Improves communication skills between designers
  • Productivity over the entire development cycle may be improved… though this isn’t the focus

Biggest Challenges in Pair Programming:

  • Eliminating the dominant partner
  • Ensure both parties contribute
  • Not enjoyable


  • One person writes a small test-case, the other then writes the code to make it pass
  • Work in small increments – fast compile and test necessary

Ball and Board:

  • Senior and junior designers working together
  • Senior designer is only allowed to use the mouse (ball)
  • Junior designer is only allowed to use the keyboard (board)
  • Senior designer forced to explain all work with junior designer
  • Takes a hours (days?) to co-ordinate

Batting Practice:

  • Senior designer writes test cases first, then junior designer writes code
  • Guided ping-pong

TDD and Pair Programming:

  • Provides unit test coverage that will be necessary when refactoring
  • May end up with messy code from small incremental development before refactoring
  • Frequent Refactor  – e.g. combine structures, algorithms, Re-arrange code, improve the architecture.
  • Automated tests allow code to be refactored with confidence
  • Rework while coding = less rework later
  • The Goal: Better implementation than designing architecture and objects first
  • Downer Note: Often lose one team member who won’t convert on teams that adopt TDD, Agile, PP…ironically, they are often the first to be converted on their next team!

Fear and Rage while adopting Agile, CI and TDD Practices!

The greatest challenge when deploying CI or TDD comes from teamwork/emotional/”the soft stuff”, rather than technical problems:

  • Not money, resources, time, technical know-how…
  • E.g. The best test framework in the world becomes useless the moment the team decides “We’re too busy to write tests”
  • Need to find ways to mitigate these risks

Christopher Avery’s “Responsibility Model” provides a framework for understanding how people react to a problem:

  • RESPONSIBILITY -> take ownership of the problem, and fix it
  • OBLIGATION -> We’ll do it, because we have to do it
  • Outward acceptance, inward rejection of the problem
  • SHAME -> self blame…can lead to QUIT.
  • JUSTIFY -> This is the way it is, because… and thus it’s ok.
  • BLAME -> It’s someone else’s fault!
  • DENIAL -> Denying the problem exists

Learning increases as you move up the model.

All of us spend time at each of these stages on any given day, the amount of time spent at each stage is what separates people.

Cannot impose this model – but awareness of it helps!

This model only works for self-improvement…can’t tell a team “let’s all be responsible now”

BUT, you can be supportive of those who find themselves in a responsible state, particularly with practices that require team support to succeed (e.g. fixing broken CI builds, writing test cases using TDD, etc)

NOT GOOD – positive reinforcement for obligatory behaviour (e.g. working late)

Ways Management Can Help:

  • People need to see the value in new practices – difficult early, as value of some practices not seen quickly
  • More information to team members – information to justify the transformations, and regular progress updates
  • Provide the time and support required…and frequent re-assurances
  • Not wavering…even when things get painful…and they always do

Selling Continuous Integration Discussion

Benefits for Designers:

  • More frequent commits -> simpler merges
  • More frequent build and testing -> Anyone can build and test anytime (Everyone should be able to use the CI system)
  • Cause of broken builds quick to identify
  • Easy to demo code
  • No fear of late-cycle changes
  • Controlled test environment important
  • Should not be more work for developers

Benefits For Managers:

  • More Metrics..better metrics?
  • Precise knowledge of who submitted what and when
  • Need time to grow into Continuous Integration
  • Scrum practices important – use retrospectives to identify pain, make changes

Team Needs to Take Charge:

  • CI system must be maintained by the team – they take responsibility for the state of the build – spreading of knowledge key
  • Admins may be necessary in more complex systems, but the team still runs the show
  • Large products (like ours!): build/release team admins the system
  • Frequent environment changes a challenge, need to be dealt with promptly
  • Red/Green computer monitor to show-off build state…teams take pride in a green status indicator.
  • Nightly builds OK – many teams lack resources to have pure build/test on submit

Test Driven Development

Though TDD was mentioned frequently during the conference, I had difficulty generating discussion on specific TDD practices.  A quick pole revealed about ¾ of the participants had practiced TDD, and most people I spoke to claimed to practice it religiously.

From other discussions, I do have some thoughts on TDD, related to our team’s attempts:

  • TDD is a technique for writing code, and basic unit tests.  It does not replace integration or acceptance testing.
  • TDD is not about writing large test scripts before beginning development work, but about writing small amounts of test code before writing small amounts of development code
  • Ideal TDD Situation:
    • Write a very simple test case (a few lines) – confirm that it fails (no syntax errors)
    • Write the absolute minimum of feature code to make the test pass – confirm that it does
    • Refactor code already written, test again
    • Repeat
  • Impediments with our practice: slow build & test – inefficient to build & test too frequently when compile and test takes 30 minutes
  • Does NOT prevent TDD from being practiced
  • Important: Off-Target testing

My Point for our Team:  Designers don’t have to write all test scripts before beginning feature code – just write as little test code as you can test at one time, write the feature code, test, repeat.

You can force a designer to write test scripts, but very difficult to force TDD!

It is difficult to force designers to practice TDD… unless they are paired with a TDD guru or directly coached:

  • Can force designers to write test scripts, but it is difficult to force them to write them using TDD – only they will know if they cheated or not (though the resutling code and TCs will usually tell the truth)
  • Designers must be personally motivated to practice TDD, and have some discipline to make it work – will fail if designers are in “obligation” mode.

When TDD is successful…

  • Development time should be equal or shorter compared to pre-TDD practices.
  • Code should be simpler, not over-engineered (A persistent problem in waterfall design)
  • Code designs should be easier to maintain – code can be more easily refactored
  • As a bonus, some bugs may even be caught earlier!

Discussion on Agile Metrics – Joined Partway Through

If metrics are used to measure performance – people will perform to the metrics.

  • Most participants were not fans of code-coverage metrics (e.g. GCov)
  • Preferred C.R.A.P. – “The C.R.A.P. (Change Risk Analysis and Predictions) index is designed to analyze and predict the amount of effort, pain, and time required to maintain an existing body of code.”
  • An index based on a formula that weighs both  cylcomatic complexity (complexity of code) and code  coverage.
  • Time spent on maintenance versus time spent on new code also a good metric
  • Cycle Time -> time from idea to deployment

Re-affirmed importance of sprint retrospective – delays or surprise metrics can be explained there.

Story Points

  • Some teams do not allow stories greater than 5 – anything greater is considered “don’t know”
  • Others only allow stories to be rated 1, 3 or 5 to simplify estimation process.

Agile the new Waterfall?

This was a small discussion about how some companies may be adopting Agile processes, claiming to be practicing Agile processes, yet not be doing anything close to Agile processes.

  • Scrum certification very easy to get – bought and paid for
  • Agile a current buzzword – organizations want to be considered Agile for business reasons
  • The word “Agile” couldn’t be more vague to describe the practice
  • Conclusions:
    • Are we agile?  Who cares as long as our practices are adding value?
    • Companies claiming to use agile practices, while not, will suffer in the long run.

Lean Discussion

One of my regrets from this conference was not attending all of a discussion on Lean practices – it was suppposedly very good.  The summary is here:

More Information including Sessions I Missed

While I attended some great discussions, there were many I was not able to attend – some of which I heard were very good.  The Conference wiki page here has summaries of some of the other sessions.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s