wood burning stoves 2.0*
The moose likes Agile and Other Processes and the fly likes Problems in large-scale IT projects Big Moose Saloon
  Search | Java FAQ | Recent Topics | Flagged Topics | Hot Topics | Zero Replies
Register / Login


Win a copy of Murach's Java Servlets and JSP this week in the Servlets forum!
JavaRanch » Java Forums » Engineering » Agile and Other Processes
Bookmark "Problems in large-scale IT projects" Watch "Problems in large-scale IT projects" New topic
Author

Problems in large-scale IT projects

Hu Chong
Greenhorn

Joined: Jun 30, 2003
Posts: 17
I am currently working in a multi-million dollar project with the project team size of around 150+. There are many problems as with other IT projects
of a smaller size.

Care to share your experiences in projects of such size?
Kishore Dandu
Ranch Hand

Joined: Jul 10, 2001
Posts: 1934
Observations from my previous experience.

1. Project needs to have clearly defined, well thoughtout, preferably long deliverable timings.
2. Definitely needs infrastructure choices that will direcly/indirectly supports distributed development efforts(some times across multiple continents)
3. Be ready for turnaround of the troops.
4. Great communication & understanding(on technology approaches and execution) among project leaders & up in the hierarchy.


Kishore
SCJP, blog
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
I don't have any personal experience with teams of that size, but I'd advice you to take a look at http://www.jeckstein.de/agilebook/index.html


The soul is dyed the color of its thoughts. Think only on those things that are in line with your principles and can bear the light of day. The content of your character is your choice. Day by day, what you do is who you become. Your integrity is your destiny - it is the light that guides your way. - Heraclitus
Hu Chong
Greenhorn

Joined: Jun 30, 2003
Posts: 17
Originally posted by Kishore Dandu:

4. Great communication & understanding(on technology approaches and execution) among project leaders & up in the hierarchy.


I have thought long on this issue of communication. Do you think using a team requirements gathering tool like Rational RequisitePro helps?

The way i see it, some people join in the project and got lost because the project is too large, some people left and take away their knowledge of the requirements. Although there were use case documentation and such, nothing beats attending the requirements meeting with the users.
[ March 27, 2005: Message edited by: Hu Chong ]
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Hu Chong:
I have thought long on this issue of communication. Do you think using a team requirements gathering tool like Rational RequisitePro helps?

...

Although there were use case documentation and such, nothing beats attending the requirements meeting with the users.

Here's a question: How does using RequisitePro improve upon the use case documentation?

Not much, I'm afraid.

Documentation by itself is not communication. Documentation encapsulates information but someone still needs to interpret and transfer that information in order for the knowledge to spread from one head to another.

Tools that encourage collaboration by making day to day routines easier can indeed make or break collaboration, but don't fool yourself to rely on tools alone to keep things on track...


Author of Test Driven (2007) and Effective Unit Testing (2013) [Blog] [HowToAskQuestionsOnJavaRanch]
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
There are several problems regarding using software tools for communication:

- you need to have the information in the heads of the people, not in some tool. Using software is not always the best way to get the information into the heads.

- communication using software has low bandwith. In a face to face conversation you can ask for a clarification in a matter of seconds, with nearly no effort. For a small question, even hitting a reply button and formulating the text might feel like too much effort, especially if getting the answer will take minutes (and might raise follow up questions) or more.

- communication using software is "cold" - it is missing a lot of subtle modes of communication. In a face to face discussion, you often can (for example) imply from tone/body language etc. how important something is to your conversational partner, or how certain he is about something. Much of that is lost in written communication.

- much of the important knowledge in software development is "tacit knowledge" - knowledge that you would have trouble documenting, because you are not really aware of it. That knowledge is much more effectively transferred using close collaboration (where you can notice you coworkers applying it).

Jutta Eckstein (the author of the book I mentioned earlier) uses "communication teams" in large projects - teams who's members sole responsibility is to foster communication between other subteams. They will go around, speak to people and connect them with other people to talk to. "You are working on Foo? Bob has thought about that too, recently - you should talk to him about it."

Alistair Cockburn propagates the use of "information radiators" - big, simple posters/charts etc. about important aspects of the current state of the project, displayed at a public place (such as the hallway) where "passerbys are hit by it".

He also speaks about "osmotic communication" - communication that happens just because people sit closely together in earshot. I strongly recommend his book "Agile Software Development" on this topic.
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
I just remembered something else Jutta does regarding requirements (which she told in a keynote last year).

For every subteam (of around a dozen developers) she has a flipchart listing all the features to be implemented in the current iteration. There are also three columns with "checkboxes" for feature: started, finished, and approved. That's a nice Information Radiator on the progress in the current iteration!
Kishore Dandu
Ranch Hand

Joined: Jul 10, 2001
Posts: 1934
Originally posted by Ilja Preuss:



Jutta Eckstein (the author of the book I mentioned earlier) uses "communication teams" in large projects - teams who's members sole responsibility is to foster communication between other subteams. They will go around, speak to people and connect them with other people to talk to. "You are working on Foo? Bob has thought about that too, recently - you should talk to him about it."

Alistair Cockburn propagates the use of "information radiators" - big, simple posters/charts etc. about important aspects of the current state of the project, displayed at a public place (such as the hallway) where "passerbys are hit by it".

He also speaks about "osmotic communication" - communication that happens just because people sit closely together in earshot. I strongly recommend his book "Agile Software Development" on this topic.


Osmotic communication has near zero value for distributed teams with more than 100 developers(likely situated across continents).

The management team above and including the team leads in a large scale project we are discussing will turn out to be the main part of 'communication teams', since many organizational heads will frown at having a special team for this purpose.

BTW I have observed that if the team is across multiple continents the execution tends to be successful if the team members share something in common. It can be culture, language etc. This is from my personal experience by working for a firm with most of the development and management teams are well versed in Hebrew and has other productive developers who are good technically.
Stan James
(instanceof Sidekick)
Ranch Hand

Joined: Jan 29, 2003
Posts: 8791
We use requisite pro as a communication tool with a team in the 35-50 range. The requirements database lets you add your own columns. We set the assigned developer name to tell them what they should work on next. We mark requirements ready for testing to tell QA when they should test them. QA marks them passed or failed to tell management the status of a release and developers when they can stop working on something.

It's bureaucratic, cold, cumbersome, slow and fairly painful. Rather than this, I'd recommend scrums, face-to-face conversations, 3x5 cards, burndown charts or nearly any other alternative.

To the original question for 150 people I'd recommend The Mythical Man Month and any way you can find to turn this into 10 teams of 15 or 15 teams of 10.


A good question is never answered. It is not a bolt to be tightened into place but a seed to be planted and to bear more seed toward the hope of greening the landscape of the idea. John Ciardi
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
Originally posted by Kishore Dandu:
Osmotic communication has near zero value for distributed teams with more than 100 developers(likely situated across continents).


Well, it certainly would have value, but is hard, if not impossible, to achieve.

The management team above and including the team leads in a large scale project we are discussing will turn out to be the main part of 'communication teams', since many organizational heads will frown at having a special team for this purpose.


Not sure what you are arguing here. Certainly a practice that you don't implement won't help you. I don't see that as a failure of the practice.


BTW I have observed that if the team is across multiple continents the execution tends to be successful if the team members share something in common. It can be culture, language etc. This is from my personal experience by working for a firm with most of the development and management teams are well versed in Hebrew and has other productive developers who are good technically.


Certainly. Frequent visits also help, as far as I have heard.

Not sure how the subject of this thread changed from "large-scale" to "distributed", though.
Hu Chong
Greenhorn

Joined: Jun 30, 2003
Posts: 17
Originally posted by Stan James:

To the original question for 150 people I'd recommend The Mythical Man Month and any way you can find to turn this into 10 teams of 15 or 15 teams of 10.


Currently, the project team is divided into many teams of 6-10 members, with each working on their individual modules. However, communication is hard among teams, as each team lead is focused on their individual module. There are scenarios when a module needs to pass information to another module in order to complete a transaction.

The architect is completely overwhelmed with such a massive project.
[ March 28, 2005: Message edited by: Hu Chong ]
Ilja Preuss
author
Sheriff

Joined: Jul 11, 2001
Posts: 14112
You have only a single architect?

I have to resort to Jutta again - as far as I know she advices to have an architecture *team* of around ten people at that project size. The team acts as a *service* team to the other teams - that is the other teams tell them what they need, and they work on providing it in a way consistent and coherent for the whole project. With other words, they develop the basic framework for the project based on the concrete needs of the subteams.

Another good idea (again from Jutta) is to rotate members between the teams on a regular basis. That greatly improves communication and understanding (and thereby collaboration) between the teams.

Where a team depends on the work of another, it also might be beneficial to have members of both teams pair program when it comes to developing the interface between the modules.

Hope that helps...
Lasse Koskela
author
Sheriff

Joined: Jan 23, 2002
Posts: 11962
    
    5
Originally posted by Ilja Preuss:
[Jutta Eckstein] advices to have an architecture *team* of around ten people at that project size. The team acts as a *service* team to the other teams - that is the other teams tell them what they need, and they work on providing it in a way consistent and coherent for the whole project. With other words, they develop the basic framework for the project based on the concrete needs of the subteams.
I was once a member of a 140-man project that had an "architecture team". It didn't work at all. Why?
  • Because the team had the wrong people in it and the "senior" folks sitting in the architecture team had to delegate everything to more technically competent people, i.e. certain members of the "component teams", and
  • because the architecture team did prescriptive work, i.e. didn't respond to real needs but instead worked on the boxes-and-lines diagrams without understanding the real problems the component teams were having.

  • In other words, be careful when creating the architecture team. It's a people business and if the roles are filled based on politics and seniority, you're probably better off reading Monster.com than reading the architecture team's work products.
    Warren Dew
    blacksmith
    Ranch Hand

    Joined: Mar 04, 2004
    Posts: 1332
        
        2
    Hu Chong:

    The architect is completely overwhelmed with such a massive project.

    Seems to me like what you need is a 10-15 person architecture team responsible for the interfaces between the modules. The chief architect can be responsible for the overall organization of the modules, and the junior architects can be responsible for working out the details of the interfaces between the modules.
    Stan James
    (instanceof Sidekick)
    Ranch Hand

    Joined: Jan 29, 2003
    Posts: 8791
    Organization by module may be a problem. Some organizations have a small number of design experts writing interfaces and an army of generic coding units writing implementations. If you don't like being called a generic coding unit, you're not alone.

    Organization by functional area might be better if each small team can drive a new feature from beginning to end with little interaction with other teams. This may be impractical if most new features require changes in common underlying modules. It is possible to design to minimize this kind of change (DIP and IOC are my favorite hammers) but it's probably not possible to completely eliminate it.
    [ March 28, 2005: Message edited by: Stan James ]
    Hu Chong
    Greenhorn

    Joined: Jun 30, 2003
    Posts: 17
    Originally posted by Stan James:
    Organization by functional area might be better if each small team can drive a new feature from beginning to end with little interaction with other teams. This may be impractical if most new features require changes in common underlying modules. It is possible to design to minimize this kind of change (DIP and IOC are my favorite hammers) but it's probably not possible to completely eliminate it.


    What are DIP and IOC?

    In my current project, there are 2 kinds of team: common systems teams such as payment, form/letter printing, etc. and the domain teams which uses the common systems teams' components. The common systems are unaware of many new requirements because there are so many domain teams.

    Is there a better organization structure to support such a large-scale project?
    Ilja Preuss
    author
    Sheriff

    Joined: Jul 11, 2001
    Posts: 14112
    Originally posted by Hu Chong:
    What are DIP and IOC?


    Dependency Inversion Principle and Inversion Of Control (I prefer the term Dependency Injection).

    Googling should give you many relevant hits.
    Stan James
    (instanceof Sidekick)
    Ranch Hand

    Joined: Jan 29, 2003
    Posts: 8791
    I like Dependency Injection, too, cause that captures the aspect I focus on, but that gave me DIP and DI which even confused me. Sorry for dropping the acronyms in with no reference. Here's my meager contribution to understanding these things ... some examples borrowed from Robert Martin's book and bent to look like what my team is doing nowadays. It goes off topic into testing at the end.

    http://www.surfscranton.com/architecture/DependencyInversion.htm
    Lasse Koskela
    author
    Sheriff

    Joined: Jan 23, 2002
    Posts: 11962
        
        5
    Originally posted by Stan James:
    It goes off topic into testing at the end.

    What? Testing off topic? Never.
    Warren Dew
    blacksmith
    Ranch Hand

    Joined: Mar 04, 2004
    Posts: 1332
        
        2
    Hu Chong:

    In my current project, there are 2 kinds of team: common systems teams such as payment, form/letter printing, etc. and the domain teams which uses the common systems teams' components. The common systems are unaware of many new requirements because there are so many domain teams.

    That actually sounds like a pretty good organization to me. The trick is to set the common systems up so that they can expose a clean interface that provides the services needed by all the domain teams.

    Given you have this organization, an alternative to having an architecture team would be to have a junior architect, or at least an interface oriented engineer, on each common system team in charge of that component's public interface.
    Stan James
    (instanceof Sidekick)
    Ranch Hand

    Joined: Jan 29, 2003
    Posts: 8791
    I like the core team and domain function teams. We organized our code that way so we could add new user groups and functions without touching the core. Unfortunately we didn't enforce the rules well, and "bad" stuff sneaked in there. If your core team can lock down the core and maybe treat it like a shrink wrap or black box product with regular releases and guaranteed backwards compatibility they might be onto something very cool.

    Something that worked well a couple projects back was for each team to have a facade. When I wanted a new service from another group they could hold a short design session together and put an empty method on the facade to return a canned result very quickly. Then they could make it do real work at their leisure. You want the API that comes out of that short session to be as stable as possible, but "forever unchangeable" is very hard to achieve.
    [ March 29, 2005: Message edited by: Stan James ]
    Udayan Patel
    Ranch Hand

    Joined: Oct 14, 2004
    Posts: 94
    Originally posted by Hu Chong:
    I am currently working in a multi-million dollar project with the project team size of around 150+. There are many problems as with other IT projects
    of a smaller size.

    Care to share your experiences in projects of such size?


    My experience with projects like this is
    1. project is broken down in modules.
    2. each module is distributed to individual team. team size depends on module. but i believe that there will be no more than 10 developers on a team.
    3. if you have 10 developers on a team, makes life way to easier. as far as communication goes, I have seen scrum 10 minute morning meeting most efective as it inculdes road blocks and issues. and let tech lead of each team hashout those issues and road blocks.
    well this is just my experience.
    3.
     
     
    subject: Problems in large-scale IT projects
     
    Similar Threads
    Eclipse very slow, any way to speed things up ?
    GWT Java AJAX Programming Prabhakar Chaganti - lots of examples?
    Craig Larman
    Eclipse : Buid Path Configuratin
    Resume 1page or 10 page