The Best Software Development Lifecycle

Or, at least according to me :)

So you want to start a project huh? Hopefully you’ve been on the internet long enough to have seen this comic about software projects, but it bears repeating:

Perhaps you’ve been a part of one (or more!) project that’s been a train wreck from the very beginning. Hazy requirements if any, changed mid project, then discovered by chance talking to another engineer that it’s been built all wrong requiring throwing out half of what you’ve done, shipped in twice the estimated time, only to find out no one wanted the feature and it gets canned.

Projects like that suck to be a part of. No two ways about it, they’re just extremely demotivating to work on as a software engineer.

I’ve been on dozens of projects throughout my career and seen some great successes and some painful failures. From all that, here’s my take on how projects should be run, taking into account the size and maturity level of the company.

A quick note on what exactly I mean by a project though before we get started. I’m talking about a significant new feature or refactor that takes several weeks (usually months) for a team to complete. This can be something like integrating a new payment processor to a checkout flow, building a whole negotiation workflow for trading grain, or adding a new sport to a Sports Voice query infographic platform. It’s not adding a button with functionality that takes 2 days to implement or fixing a bug.

Small Companies#

Your main advantage as a small company is that you can move nimbly, experiment to find what works, fail the first couple times, but then move onto the next hypothesis. You need to dance past those mega corps weighed down by their heavy armor and seize the advantage.

So little to no structure for projects is appropriate here. The risk of spending way too much time planning and documenting (wasting time and money) is much higher than the benefits gained by being methodical with project execution at this stage.

Typically whoever’s acting as the PM will come up with an idea for a feature, chat awhile with the engineer about it, and let them take it from there while collaborating on any issues or questions that come up during the process. As the lead engineer, go huddle with any other engineers who you’ll be collaborating with and come up with an attack plan and then get to work!

This is a nice simple and quick way to work when each individual has more responsibilities and autonomy to experiment. It breaks down as the number of lines of communication grows exponentially as people are added to the project though.

Mid Sized Companies#

This is where I’m going to focus since it’s both where I’ve spent a good amount of my career as well as the place individuals still have leverage to improve the processes. Early stage companies are great for lightweight processes and more agile workflows. As the company grows to have several independent teams that don’t really know what each other are doing you start to need more process and structure to ship larger projects.

Who’s Holding the Steak?#

First let’s talk about the… ahem, steakholders. Everybody’s got a steak these days but the ones I want to focus on that are the core trifecta: developer, designer, and product manager. I won’t bore you with their job descriptions since you should know or can look them up, but just know that these are the key roles to shipping large features that must be aligned throughout the project.

Depending on the team composition there may be folks wearing two of the hats, but if you find yourself wearing all three then you can gently take them all off to give yourself a break, then get back to work instead of reading the rest of this since you clearly have lots to do. There are cases where you’ll want to double up on a role, see below for cross team projects, but try not to go above 6 people for the core decision makers since it’s too easy to get sidetracked and lose the tight communication that makes for successful projects.

1. Prep For Kickoff#

Before really getting started it’s important that some prep is done ahead of time to use everyone’s time most effectively for the next step. Most of the work here falls to the Product Manager to validate and research the project before bringing it to the team, but naturally they will have a few questions for engineering while doing their research which is totally fine as long as it’s not expecting designs or estimates.

Here’s what each party needs to do before kickoff:

Product Manager:

• Have a clear understanding of the business problem trying to be solved, and the user jobs to be done (what are the requirements and nice to haves)
• Research competitor solutions to the same problem
• Have loose ideas on how the new work will fit in with the existing product
• Understand how much business appetite there is in time and therefore cost
• Ideally set goals for metrics to influence

When the research is nearly complete, the PM should schedule the kickoff ~1 week out and include a short description of the problem statement requirements so engineering and design can start their own research.

Engineer:

• Make sure you have a solid understanding of the architecture of the existing product in the area that will be changed.
• For complex projects research design patterns, open source packages, and competitor architectures

Designer:

Find design inspiration on how similar features in other products work specifically looking at:

• Their user interaction flows
• What edge cases and error conditions exist
• How many button clicks there are to accomplish the task
• What touch points there are outside the product
• If there’s any tutorial on how to use the feature
• What sort of messaging is used

2. Kickoff time#

It’s time to get this party started and gather the troops!

The goal of this meeting is for the Product Manager to lay out goals of the project in detail and to share their research with the team such that the trifecta understands the project as well as possible. A good chunk of the time should be spent as an open dialog with the PM answering questions about what’s needed and how things might work. Engineering and design should be asking as many prodding questions based on their own research to fully understand the problem and what solutions are feasible.

3. Solutioning#

After a short break gather the trifecta again to start talking solutions. This could happen right after kickoff but there are cases where engineering or design needs to do some more research based on what was discovered in kickoff to get a good enough idea on how to build a solution.

The goal of solutioning is to all agree on user flows and low fi designs that show how the new thing is going to work all the way through. Use a whiteboard, figjam or balsamiq to create the user flows and designs so everyone’s on the same page. Just saying the user flows out loud isn’t good enough. It’s too easy to miss edge cases and have two people end up with two different ideas of what should be the same thing. The designs need to cover every page (and emails or other touch points) and workflow the users will go through. This means every new button or change to functionality, every decision point and what happens in each case, with each user persona (admin interactions for example).

Remember to keep it high level, this does not mean every bit of text appearing on the pages, detailed metrics, testing, API design, or anything like that. That all comes later.

Ideally the trifecta will just lock themselves in a room for an afternoon, or every afternoon for a week, or however long it takes to get to agreed upon flow diagram and designs. Doing this async is possible but from my experience this just drags on week after week with lots of back and forth as the requirements are negotiated and the tradeoffs weighed. Getting to agreement should also include the discussion that the design is feasible within a time frame the business will tolerate, otherwise you need to de-scope something from the design or just can the project.

Once you get to agreement it’s time to put the diagrams and agreed upon requirements in confluence or wherever as the first artifact of the process.

4. Collaborate#

At this point the lead engineer on the project needs to write down the high level plan for accomplishing the project. This includes any new pieces of infrastructure needed, what API’s need to be added or changed, what data will need to be stored or processed, and what frontend changes are needed. It does not mean detailed API / DB schemas or detailed UI component design. Going through this exercise you may also uncover edge cases that weren’t thought about in solutioning. Meet on them as the trifecta and if a simple solution can be found great, otherwise do some more research and go back to step 3 for a deep dive to solve the issue.

Once the high level plan has been created it’s time to share it with the the rest of the engineers. That’s right, send it to all the other engineers so anyone who’s interested can take a look and give you feedback. People will skip the one’s they don’t care about so don’t worry about sending it too many people as long as the culture is established that every engineer is not expected to review. This process is where you find your blindspots in designs. We all have blindspots since it’s impossible to know every technology out there and every nuance to every part of a large product. Think of it like trying to find a tennis ball in a big field at night with only a flashlight. No matter how big your flashlight is, you’re better off with a big team all equipped with flashlights all looking at once. You might be surprised to find out another team built something similar, or an engineer has worked on a similar feature at another company and knows the pitfalls.

Set a time deadline on reviewing (a week or two should be sufficient for all but the largest projects) so you’re not stuck in an endless change cycle responding to comments. As the high level design is being iterated on based on feedback, keep the rest of the trifecta in the loop. Discuss any surprises or major changes that affect scope, you may find certain “requirements” that turn out to be really hard suddenly aren’t so required anymore.

At the same time Design can be brainstorming different ideas on data presentation, optimizing the user flows, and how to adapt the feature into the existing product. It’s possible to start on the detailed designs at this point knowing there’s risk in wasted work (see below).

The artifact coming out of this process is a signed off high level plan as well as an engineering estimate measured in person-weeks with also a sense of how parallellizable it is. Review again as a team to make sure the ROI is there with the business.

5. Figure out the Details#

At this point the project should be signed off to start and the only possible way there’s an outcome where the project doesn’t ship is if something very out of the ordinary happens (keep a mental note if this happens more than once as it’s a sign of company instability).

It’s now engineering’s job to refine the high level design into a detailed low level design with API contracts, DB schema’s, and how the edge cases will be handled. Design will also need to finish the UI/UX design ideally with copy finalized though that should be easy enough to change.

This may seem like overkill at this point but for companies with more than 3 engineering teams there should be a formal architecture review meeting held to present the low level design. This let’s people know exactly what’s coming so they have context on what’s going on outside their team and it also gives them a chance to ask any remaining questions, especially on how it relates to what they’re working on. Most importantly though it’s engineering sign-off on the project to close out any other “why didn’t you do it this other way” questions that come up after the projects started or finished.

This architecture review should be a breeze if you followed the process and had the other engineering stakeholders review the high level design earlier on and incorporate their feedback. There shouldn’t be any surprises by the time you hold the meeting!

Once the low level design is complete it’s a good time to create tickets for the work and estimate them as a team to get a more accurate idea of time.

6. Start Building!#

It’s finally time to start executing! Execution is not the focus of this post which is already much longer than I thought it would be 😅 but I’ll leave you with one thought here. Next time you’re starting to execute a new project build the hardest (and scariest) part first. This by definition is also the riskiest, since it comes with the most uncertainty, so getting it out of the way first reduces the chances of the project slipping late in the game. Once you get over the hump it’s also a great feeling to have the rest be a nice gentle downhill ride.

Optimizations / Skipping Steps#

There’s a lot of stuff going on with the whole process and since mid-sized companies covers a lot of different possibilities going through every step may not be the best use of time. Be warned though, everything has a tradeoff and often times what you think is saving time by skipping a step comes back to bite you and causes more wasted time, hence the whole need for this process in the first place!

Engineering can skip the prep if they know the application area front to back, design can skip the prep if they’re very familiar with building similar features, but the Product Manager should never skip the prep. This is the most crucial part of the job and the most effective way to waste time is to build the wrong feature!

Skipping the Collaborate step might save you some overall time on the project but comes with a big risk. If a team starts building a feature then finds out 4 weeks into the project another team is also working on a project that conflicts in design you’re going to cause a major kerfluffle and delay for all involved while you sort out how to reconcile the two. There’s also the possibility you missed using another teams system that you could have used so there’s more wasted time. In the mostly remote-work tech world we live in now it’s hard to find time to interact with other teams anyways so you really should go out of your way to make it happen.

Design and engineering may be very tempted to go straight to detailed designs either before the trifecta agreement on the low level designs or the collaborate step but this is a surefire way to waste time when (not if) the requirements change 6 (or 11, or 29…) times during the process.

One reasonable optimization is starting to build the backend in parallel with the final designs being finished since they’re not dependent on each other. During this time the frontend teams can also be setting up their mocks based on the API contracts which can be used for tests and getting started on the UI before the backend is ready.

Cross team projects#

Projects that cross team boundaries really up the complexity ante. More people working on the project means exponentially more communication channels and ways for things to go wrong. The more I think about it the more I think it’s a topic that deserves it’s own post. There’s just so many permutations of how teams could be structured, what additional parties are in involved (like a project manager), and how much work falls on each team. So attempts to summarize here would surely be inaccurate in some situations.

If I could only say one thing about them though, it would be this: spend the majority of your time making sure the interfaces between teams is 100% clear. Both the functional requirements of what each team is responsible for as well as the technical (API, functional, data) interface.

Large Companies#

I’ve only worked at one mega-corp in my career thus far so my experience is more limited here but as far as I know one of these two scenarios will apply:

1. The SLDC has been well established and everyone must follow it so good luck changing anything and this article doesn’t apply to you

OR

2. You work in a semi-autonomous business unit that still has flexibility on ways of working and most everything from the Mid Size section applies

If you’re lucky enough to be in the latter category there’s only a couple differences I can think of to point out. Big companies need more documentation at each handoff point for a variety of reasons (most of which are legal compliance) so make sure to spend some extra time there and cross your i’s and dot your t’s. You may also be asked to get sign off from higher management in which case the high level plan with diagrams you’ve already created should be suitable.

The company is also too large at this point to send the high level plan to all engineers. It would just be spam to tune out if all the engineers did that and then no one would be inclined to review anything. Limit the scope to teams adjacent to yours that are working on similar things. If your company supports the concept of chapters or guilds you should also send out the relevant portions of the design for them to review as well.

Conclusion#

There’s a lot of ways to do project management wrong, and probably no way to do it perfectly. That shouldn’t keep us from trying to constantly make it better though, and this is my attempt. If you’re interested in reading more check out this article on project management in tech.

And as always, let me know in the comments what’s worked well for you!