Wood for the Trees

Project Management and Best Practices in Retrospect-iva

Posted by Mathew Abonyi Sat, 28 Jul 2007 02:18:35 GMT

I don’t hear a lot about project management, even though there’s a lot about how to manage a project. Testing, deployment and source code management get the most attention, and project management seems to get the least. Maybe that’s because it hasn’t been done properly yet and all the solutions out there only address pieces of the overall problem.

So I’m going to try to clarify, for myself mostly, the kind of project management that is needed and why it is so important in development.

Getting It Right

When I say management, I mean a combination of something like Lighthouse and Basecamp, with a serious overhaul of perpsective. An integral part of good development is developing ideas hand-in-hand with the code. Management is all about keeping this communication as agile as the coding process, sticking to priorities, and addressing the right things at the right time.

Is Basecamp sufficient for managing a project? No. DHH even says it is not meant for managing Rails projects; it’s for marketers and managers. It’s only a piece of the puzzle, because it provides no way to easily track code.

Is Trac sufficient for managing a project? No. It is too much like a big todo list and a bug tracker combined. It is very developer-centric—even when the developer is also the designer and manager, there’s no way to make known the other roles. Trac too is only a piece, because it provides no way to easily communicate ideas. A wiki doesn’t cut it.

Are help desks sufficient? No. They are too customer/support-centric. They have no way to easily communicate the ideas of developers and designers.

What about Lighthouse or Unfuddle? For a hosted solution, Lighthouse and Unfuddle combine Trac and Basecamp. That’s going in a decent direction. Anything which integrates different parts of the development process is addressing the need for management. But it’s not enough, because it has no integration with the customer.

What about [insert megolithic answer to everything]? No. It’s too complicated, has too many options and forms, too much information on each page. Something as complicated as Google Analytics, for example, is pushing the boundary of what is acceptable. Complicated applications get in the way of communication and understanding, even if they integrate everything. Simplicity first.

What all of these solutions lack is a focus on the different kinds of users for a project, ways of easily communicating their needs and ways of addressing those needs. Even when the developer, designer, marketer and manager are rolled into the same person, it is important to separate the roles, make them clear and integrate each one’s concerns at the right points.

It’s all about the development process.

The Development Process

I see that there are 9 stages in the development process:

1. Management: find out the next need to address
2. Specification/Testing: specify how the need is addressed
3. Coding: code until the specification passes
4. Continuous Integration: combine the efforts of multiple coders
5. Refactoring: clean up the code
6. Graphic Design (if needed): make the new feature appealing
7. Deployment: release the latest revision
8. Marketing: advertise the latest feature
9. Customer Feedback: find out what is going well and what isn’t

If you see this process as organic, the importance of management becomes much clearer. In fact, I think management is the most important stage, more important than the code itself, for a number of reasons:

• Management is the first step; without management, testing/coding is arbitrary
• Management brings everyone into the development process
• Management gives everyone an overview so they can see the wood for the trees
• Management encourages communication between everyone
• Management naturally focuses on the most important aspects
• Management reinforces and rewards good development
• Management operates organically, reflecting needs and their importance
• Management begins the specification/coding cycle
• Management draws from and feeds into all other stages of development

Disconnecting from these aspects of development is a serious mistake because it denies the organic element of development. Everything needs to converge at some point and management is the most natural way of acknowledging and converging all of a project’s members, roles, ideas, problems and concerns.

Poor management will try to force the development process into a linear pattern. It will approach things as ‘things to do’, ‘features to have’, ‘milestones to reach’, ‘deadlines to meet’, ‘code to test’. Everything will have its place, need to be addressed by a particular person… in short, it’ll look like a Trac installation. The code will feel strained, regimented and will generally be a rather boring thing to deal with. The developer is being strait-jacketed.

On the other hand, a lack of management will result in the process becoming chaotic. The developer will code whatever takes his fancy. Occasionally e-mails or posts containing feedback will find their way into the code, but mostly the code will diverge from the customer’s interests. The organic element has gone mad in this case because the developer is too isolated.

But good management acknowledges the organic aspect of development and lets the code flow. It translates ideas into specifications just as test-first translates specifications into code. Good project management will create and maintain strong channels of communication between developer, designer, marketer, manager and customer. The real needs for the project will appear of their own accord as different ideas converge in one place.

But that sounds much easier than it actually is. There isn’t yet an application out there which integrates all of those roles together, but some are closer than others. I think Retrospectiva could be the one which gets there first.

What is needed for good management

The three major aspects of project management are development, collaboration and integration. There needs to be a way to develop, track that development, and focus it. There needs to be collaboration and communication surrounding that development. There needs to be a process of integration between management and the other stages of development.

Ideally, a project management system will have the following aspects:

• Stories: isolated stories to be resolved (bug, feature, question, idea)
• Dynamic properties: status, milestone, persona, feature, assigned user, assigned group
• Reinforcement: aspects of the application NOT to change (positive feedback, robust code)
• Personae: ability to define personae, like power users, buyers, sellers, novices, etc.
• Milestones: rough organisation of stories and deadlines

• Messages: site-wide (like Basecamp) and for each milestone
• Roles: developer, guest, manager, designer, customer, marketer, administrator
• Groups: optional story development by groups
• Interfaces: different interfaces for each role and/or group

• Cross-referencing: referencing between stories, messages, milestones, source
• Testing integration: update stories with progress on tests (e.g. Tesly)
• Coverage Integration: stories for area of test coverage, whether 100% covered or not
• SCM integration: update stories through commit logs
• Continuous Build Integration: create stories for failed builds
• Error Notification Integration: create stories for application errors
• Customer Integration: create stories for customer feedback (positive & negative)

Most importantly, the interface(s) needs to be extremely clean. Lighthouse goes a long way in making a highly readable, even pleasurable interface. Most of the aspects I list above can exist on their own, meaning the application itself will have many facets, but each very easily understood. Cross-referencing is probably the most important of them all, since it will bring together the various aspects.

The shift in perspective I suggest for project management is to not focus on managing people (like Basecamp) or code (like Trac) or users, but ideas. Those ideas will never disappear from view, unlike tickets on Trac or todo lists on Basecamp. As stories grow and connect with new stories (like associating tickets), everyone will see the evolution and development of features, in which way the project is going, and be able to react better to the movement of the project. In a way, project management also begins to document the project, but more importantly, it shows in black and white how ideas become code and how they evolve. At the centre of the project should be a cloud of ideas which each role can see differently.

Just to give a little hypothetical situation: at the beginning of your project you had a simple user authentication system. Over time, users talked about adding Open ID. Management wanted an authorisation system and an admin interface. Designers wanted a cute widget that pops down with AJAX. Developers wanted to extract it into a plugin. All of these ideas would be associated and appear together in a good management system, showing the time each one was added and completed, the role which initially suggested it, and the group or user responsible for implementing it. All the bugs, notes, support questions, requests and feature stories will clump together and naturally point towards what is needed next, if anything.

This form of project management could very well revolutionise the way development is perceived. Or is it a load of bullshit?

Posted in Rails, Ruby, Retrospectiva | 3 comments | no trackbacks

Capistrano 2.0, upgrading & fitting into a size 0 dress

Posted by Mathew Abonyi Fri, 27 Jul 2007 14:02:09 GMT

The improvements to Capistrano are much welcomed. My deployment recipe is now half the length it used to be and it is much easier to follow what is happening for my many types of deployment. I love the new features added, mostly dealing with manipulating scopes and enhancing the user’s ability to extend the core framework.

Review of new features

namespaces: Like Rake, you can namespace your tasks and group them together more sensibly. This feature alone is worth upgrading for just to make your scripts more sensible and easier to read.

events: Like Rails, you can now perform tasks before or after other ones rather than using the hacky ‘before_something’ and ‘after_something’. Much cleaner and much faster too.

strategies: In addition to checkout, you can now deploy via export and copy and use different strategies for deployment, such as using export for your copy_strategy rather than zips and tarballs.

scoping: All sorts of scoping has been introduced in Capistrano 2.0, from namespacing to single execution of “run” and “sudo”, allowing you to define specific roles or hosts in which your commands run.

help: Capistrano 2.0 now has a more verbose way of explaining tasks with cap -e task_name. You’ll realise how useful this is when you use it for the built-ins as well as your own.

All in all, Capistrano is pretty simple, but it is the way it is written that makes it appear so much simpler than it really is. Capistrano 2.0 takes that to a new level, not groundbreaking perhaps, but definitely a lot cleaner than its previous releases.

Upgrading from 1.4.1

There is no need to change config/deploy.rb out of the box. Capistrano 2.0 is nicely backwards compatible, unlike other things out there, and, at least for me, nothing broke because of the upgrade.

You can look at Capistrano’s instructions for upgrading, if you want to know what is being done, but for the impatient, here are the steps you have to follow before we can start drying up your deploy script.

sudo gem install capistrano

~# cd projroot
projroot# capify .

projroot# cap -f upgrade -f Capfile upgrade:revisions

4. Rinse and repeat for each of your deployment targets

Getting your deploy.rb into its new size 0 dress

You may now have the very understandable urge to slim down your deployment recipes. With the introduction of Capistrano 2.0, I found my deploy.rb reduced to less than half the size. Below, I cover the areas which you should focus on to get that deploy script into its new size 0 dress.

Anatomy of my deploy.rb

1. requires: capistrano-ext, mongrel_cluster, etc.
2. global, stage and custom variables
3. event chains
4. rewriting built-ins: web:disable and web:enable
5. extra tasks: fixing permissions, copying mongrel confs, etc.
6. custom deploy tasks: long, normal, quick
7. maintenance tasks: backup, restore

Variables

More than before, variables are the lynch-pin of slimming everything down. The first thing you should do is look over every task rewrite or custom task and see how it can be turned into a simple set :var, true/false/whatever. Capistrano 2.0 will make it very easy to do this.

With Capistrano 2.0, you should use the set command religiously, both for built-in and custom tasks.

I personally set the following at the top of my recipe.

• Global variables: stages, deploy_via
• Application specific: application, repository, user, scm_username
• Deployment specific: deploy_to, rails_env
• Custom variables: serving_via, suexec, suexec_user, suexec_group, disable_template

Deployment Strategy

I would personally suggest using xport for your deploy_via strategy unless you have a reason for using heckout or copy.

Using Namespaces

Namespaces make it dead simple to group common tasks, like different restart methodologies. I use a serving_via variable which translates into the reload:whatever task to run for restarting the application. For example:

namespace :reload do
  desc "Default reloading procedure"
  task :default do
    mongrels
  end
  desc "Reload an FCGI application"
  task :fcgi, :roles => :app do
    sudo "#{current_path}/script/process/reaper -a graceful -d #{current_path}/public/dispatch.fcgi"
  end
  desc "Reload an LSAPI application"
  task :lsapi, :roles => :app do
    sudo "/usr/local/litespeed/bin/lswsctrl restart"
  end
  desc "Give the mongrels a bath"
  task :mongrels, :roles => :app do
    restart_mongrel_cluster
  end
end

Note: I warn against using restart as a namespace because it clashes with the built-in task and, in certain instances, results in infinite recursion.

Maintenance Splash

The biggest change in Capistrano you may need to worry about is the removal of delete and render. Don’t despair, though, because creating a maintenance splash is still easy. This is my rewrite:

desc "Generate a maintenance.html to disable requests to the application."
deploy.web.task :disable, :roles => :web do
  remote_path = "#{shared_path}/system/maintenance.html"
  on_rollback { run "rm #{remote_path}" }
  template = File.read(disable_template)
  deadline, reason = ENV["UNTIL"], ENV["REASON"]
  maintenance = ERB.new(template).result(binding)
  put maintenance, "#{remote_path}", :mode => 0644
end

desc "Re-enable the web server by deleting any maintenance file."
deploy.web.task :enable, :roles => :web do
  run "rm #{shared_path}/system/maintenance.html"
end

Using events

Like the before and after filters in Rails, you can now cleanly chain together tasks. I’m a sucker for one-line solutions and these are really so simple that it makes my heart bleed:

before "deploy:restart", "fix:permissions"
before "deploy:migrate", "db:backup"
after "deploy:symlink", "deploy:cleanup"
after "deploy:update_code", "deploy:web:disable"
after "deploy:restart", "deploy:web:enable"

capistrano-ext & multistage

I highly recommend the use of multistage. It comes with the capistrano-ext gem (which has been upgraded to Capistrano 2.0, of course).

Basically, it separates the concerns of different deployments. If, like me, you like having a few other versions of your application out there, like a staging area, a testing area for bleeding edge features, and, of course, the production site, separating these in Capistrano before 2.0 was very irritating. Multistage sorts that out very nicely.

By default, you must specify the stage you wish to deploy. This behaviour can be overridden by setting the default_stage variable, but I like being explicit. This is what using stages looks like:

# cap production deploy

If you don’t provide ‘production’, it’ll complain and abort.

Using multistage is dead easy. Put this at the top of your deploy.rb:

  require 'capistrano/ext/multistage'
  set :stages, %w(staging production testing)

Run the task for generating your stage deploy files:

projroot# cap multistage:prepare

This will create a recipe file for each stage in a new config/deploy directory (exactly like Rails environments). Now, in each stage recipe, add all of your stage-specific tasks and variables. For example:

set :rails_env, "stage"
set :application, "staging.example.com"
set :deploy_to, "/var/www/#{application}"

Now switching between different deployments is a breeze. Just make a new recipe file for it with the necessary variables and you’re set.

Posted in Rails, Capistrano, Ruby | 1 comment | no trackbacks