Paul Duvall

Thanks a lot Paul and Andy...will you stick around?

I enjoyed hearing from each of you. I plan to stick around as much as I can.

Originally posted by Ilja Preuss:
That same developer today can't even imagine how other teams can work without a CI server...

Nice!

Actually, for some time now, we are using a homegrown software to show the build state that supports not just the two states "success" and "failure", but also "failure noticed", "fix in progress" and "fix committed".

I really like this. One of the techniques I've used with the Ambient Orb and Continuous Feedback is to notice trends rather than finite states (i.e. Success or failure). For instance, as the code duplication increased the color of the Orb would change from green to blue to purple to pink to red - as it got worse. Code becomes unmaintainable a little bit at a time and this can be an approach for noticing it while it's occurring.

Resource dependency of Junit tests:[/b] the issue of junit tests that I can run locally but not on the CI machine

Can you expand on this? Why are you unable to run on the CI machine?

Another interesting first reaction to Cruise Control was that one of our core developers installed an email filter so that he wasn't any longer "pestered" by all those emails telling him that a build failed...

Ah-ha! I like to call this antipattern "Spam Feedback". There are two sides to this. The first is whether this person is acting professionally and whether the development team's culture has embraced the concept that broken builds are high priority. The second side, and more interesting to me, is whether using an e-mail only approach is an effective communication mechanism for this person or team. In chapter 9, Continuous Feedback, I go over many types of feedback mechanisms including, Task bar notifications, RSS, large displays, sounds (more for co-located teams), Ambient Orb, Lava Lamps and so on. A team may implement some combination of these that works with their team. I rarely find that only one communication mechanism works for a team

Developers committing code that breaks the build?
Happening from time to time. That's one of the things that make me wonder whether a manual CI process might work better.

This is a good reason to go with a manual CI process as long as the team is disciplined to run the integration build *every* time. Other teams use a private build on their workstation that runs the same build that the integration build server will run (this prevents many of the common errors). Other teams may choose not to run a private build (because of the time involved) and will, instead, segment their project into smaller teams that each have their own build so that the build errors are smaller. There is a master integration build that occurs based on the success of these other builds (Owen Rogers from ThoughtWorks discusses similar concepts in a paper called "Scaling Continuous Integration")

Another thing that is rather annoying is that CVS commits aren't atomic, so that every now and then we have a build failure simply because of an inconsistent checkout.

Too bad you can't switch to Subversion Have you tried increasing the quiet period time in CruiseControl?

Thanks for sharing Ilja!
[ August 30, 2007: Message edited by: Paul Duvall ]

For those of you that are implementing CI on your projects, I am interested in hearing the challenges that you may be experiencing. Here are some of the challenges I've seen. Is anyone experiencing these and how do they manifest?

1) Getting developers to commit all code daily?
2) Developers committing code that breaks the build?
3) Integration builds continually broken or staying broken for several hours or even days?
4) Lingering broken tests in Integration Build (tests not failing the build)
5) Others?
[ August 30, 2007: Message edited by: Paul Duvall ]

What are your thoughts, if any, about BDD versus TDD?

I think Andy will be replying to this later, but for now, have a look at his blog article on the topic here: http://thediscoblog.com/2007/08/28/is-bdd-tdd-done-right/

The only thing I'm not familiar with there is what you mean by building system components separately. Could you elaborate on that?

If you are able to segment parts of your project into discrete components with well-defined interfaces that can be built separately you can significantly reduce your build times. Let me know if you need further elaboration on this.

I'd advise to make sure that you don't measure something just because it's easy to measure. Think hard about how you will use the information you will get. Having the process produce information that isn't acted upon has a high risk of reducing the perceived significance of the whole process.

I agree with this and an approach I've used and discuss in the book is to start by checking nothing and then add the built-in checks from the ground up (one-by-one) as necessary for the project. This is especially beneficial because running something like a CheckStyle with all its built-in rule checks can quickly overwhelm a team to the point where they never look at or, more importantly, *implement changes* based on the reports.
[ August 30, 2007: Message edited by: Paul Duvall ]

Others, though, voice the opinion that ideally, CI isn't done using a tool, but manually.
What is your point of view in the book?

I completely agree with Jim about CI being an attitude. Because of personal preference, I like to use an automated CI server. I suggest that if a team can pull it off, manual CI can be a very effective approach (and how CI was originally implemented). In the book, we say as such and in the text, we give more examples that relate to automated CI since this is how it is implemented most often (and our personal preference).

More often than not, some feel that once they've installed a CI server, they are "doing CI". This is simply not true and I recently blogged about how "Continuous Integration is NOT about the CI server" at http://www.nofluffjuststuff.com/blog/paul_duvall/2007/08/continuous_integration_is_not_about_the_ci_server.html

The property file contains things like the database url, the
mail server details...

These things vary in each environment.

What I've done is to use a .properties file for each environment: dev.properties, qa.properties, staging.properties and so on. If I want to override anything locally, such as from my developer workstation, I'll read a local.properties first (the values of this file are not checked into the version control system). Because of the immutability of properties in Ant, this works well.

The problem is that as our project grows large, build times are hovering around 20 minutes. Is that abnormal for a CI process?...what do you see as an acceptable maximum for the CI build time?

In my experience 20 minutes is not that abnormal, but you will also begin to lose the benefits of CI the longer the build takes. The longer it takes, the less people want to run it, the less people want to run it, the less they commit their code, the less they commit their code, the less feedback you get...and on and on. It's all about the technique. There are many techniques we discuss in improving the performance of long-running builds, particularly in chapter 4 (Building Software at Every Change), such as:
- Using a dedicated integration build machine
- Increasing the integration build machine's hardware capacity
- Building system components separately

Many of the teams I've worked on or observed that use CI tend not to move onto their next task until the integration build is successful (a good practice). If your build starts taking a long time then there will be more waiting. It really depends on your team environment, but I think a good rule of thumb is what Kent Beck suggests in "XP2", which is keeping your integration build to less than 10 minutes or so. Another suggestion I have is to establish a build pipeline ("staged" builds) in which you run a "commit" build - which is a real lightweight build - followed by a secondary build based on the success of this commit build that may take much longer and run things like component and functional tests against a database, etc. The lightweight commit build may include only compilation and unit tests (that don't touch a database) so that it is extra fast and you still learn a majority of the most common errors such as not committing a source file to the repository, failed unit tests and configuration errors.

We were using cruise control but it seems like it was slowing down the server. Then we installed Pulse, but it has the same problem. Now we are thinking of Hudson.

Do you folks use any better?
Any recommendations from Author/s?
Any good about Hudson?

I tend to use CruiseControl, because it's given me what I've needed over the years and I've gotten it to work with many projects (on a single server) and large projects as well. At times I've seen Out of Memory errors due to large log files, etc. If it's happening with a couple of servers (CC and Pulse, in your case), I'd suggest looking at what type of output the automated build is generating (perhaps decreasing verbosity?), increasing the allocated memory allocated to the Java VM running Ant or perhaps increasing your hardware resources. But, I'd hesitate to suggest that until you can find out exactly where it's slowing down (i.e. does it only happen when it's running the build?). I'd hate to suggest moving to Tool X (be it AntHill, ParaBuild, BuildForge or whatever) and it turns out that it's because your build scripts were generating large log files for some reason. In other words, you may have the same problem whichever CI server you use.

You should be able to stand up many of the servers quickly so if you want to throw Hudson or some other server at it and see if your problems go away, then go for it. But, it still may not solve the problem.
[ August 29, 2007: Message edited by: Paul Duvall ]

could they use the book as a �check-list� to see if the CI is used "correctly" or if there's something missing?

I think this would be a good way to utilize the book and, in many ways, this is how I organized it. At the beginning of (and throughout) the writing process, I thought to myself "if I were to list the effective practices I've used on projects, which would I choose?" There are 46 CI-related practices in the book. Most of the 46 practices are chapter subheadings and are based on this list of practices that are useful from our experiences. Of course, not everything on this list of effective practices made it to the book because certain practices weren't relevant to Continuous Integration. And, I definitely don't think you need to implement all of the practices from the book to be successful at CI, but "Continuous Integration" demonstrates the possibilities of what you can implement using examples.

For dev teams with an existing CI environment, I will assume they have a fully automated build with minimal dependencies (chapter 1 and 4). Next, they can use the development team practices from chapter 2 as the cornerstone for moving forward to incorporating other build processes such as database integration (chapter 5), additional automated testing (chapter 6), static/dynamic analysis (chapter 7) and deployment (chapter 8) to a target environment
[ August 28, 2007: Message edited by: Paul Duvall ]

The team I work in uses continuous integration, but only within a system. We run an automated unit test suite every time code is checked in. However, a lot of issues arise in the interfaces between systems, i.e. database access, remote updates, interface changes etc.
Is this something that CI could be applied to? That is CI appropriate for inter-system as well as intra-system use? If so any tips on how go about putting tests (test harnesses?) in place?

Yes, you can use a CI system to do this, but the more you can control these inter-system dependencies, the better. This means you might get a snapshot of this system/database, etc. on some scheduled basis otherwise you'll be spending all your time managing someone else's project. A better option may be to mock these system interfaces so that it works according to an established "contract". However, you'll want to sync with the real thing periodically. I'll assume that these system interfaces are an important aspect of the value of the system, so the sooner you find any interface problems, the better. But, you don't want it to "continuously" slow your main project to a halt either, so mocking with a periodic check against reality can be a good approach to this.

I am new to Java. What is the best process to get an Application executable to run on a client PC. We are using Ant where I work is this the best step to take.

Also, when you build an app in Eclipse why is it you can save a program to a .exe instead of have using javac.

Yes, despite its problems, using Ant to build your software is a good choice especially as compared to running a build from only your IDE (Eclipse). From Eclipse, you can execute this Ant build script. Just make sure you can also run your Ant build without needing to use Eclipse (i.e. from the command line; this is necessary for CI to work)