Two of my systems design projects went live during the past three months. It doesn’t always happen that two of these projects reach their final stages together. This simultaneity gave me a busy start to 2024 and taught me a few lessons regarding the design decisions I made early in the projects. In this post, I share some lessons from my system design process and participation in these projects.
Context
If you’ve stumbled upon this post and have not yet been introduced to systems design, you can read up on why you should solve complex problems using systems design.
I’ve worked as a systems engineer and designer in air traffic control for the past four years. What I do depends on the type of projects that have the highest priority within the organization. I write code, maintain Linux systems, write documents, or collaborate with my peers in meetings.
My favorite projects are those where the business needs to expand, replace, or integrate entirely new systems into the operational Air Traffic Management machine.
These projects usually require a systems designer. I’ve been fortunate enough to play my role as a systems designer in a few projects. During the past three months, two of these projects reached fulfillment:
- A system replacement project where I created the initial design in November of 2022.
- A system expansion project where I created the initial design in August of 2022.
As an engineer, it is an incredible feeling to see the end products of something that started as a rough sketch and evolved into a functional design, a technical design, implementation, and finally, concluded after twenty months.
Being part of these two projects from start to finish does not always happen, as it is not uncommon for organizations to onboard a designer at the beginning of the project and utilize others for the implementation.
I wore many hats in my case, but an entire team still performed the implementation. At least 20 – 30 people were required at different stages of each project.
I’m also on the team supporting the new system for one of the projects (at the time of writing this post). This means I see how the design decisions influence the end users and the support team.
Looking back at the previous twenty months, I’ve learned a lot. Even though some things might sound obvious in hindsight, your focus is not always straightforward when you are in a rocking boat with a few people.
Lessons Learned
Be a team player – know your role.
I like to do things myself; however, I don’t have enough time to perform all tasks alone. Significant change happens when you play as a team. Remember also that being a team player means you are responsible as a team, and the work is not yours but the team’s or the organisation’s. I’ve found that team ownership is complicated to master, as some of us like individual recognition.
Being part of a team means that you are also a leader in your specific discipline. If you are a DevOps engineer, advise the team early on how to adopt DevOps best practices and plan for CI/CD infrastructure.
If you are a software engineer, ensure you use the right development tools, such as development platforms, static code analyzers, code style guides, documentation best practices, etc.
Understanding your role within the team is essential. Whenever I tried to perform tasks outside my role, I got frustrated. Communicating this early on within your team or with management is usually much better. This links to my next point, which is the most important one.
Communication
It isn’t easy to overcommunicate in any project team role, but it is extremely easy to under-communicate. You can grow in your role by working on your communication skills. This can lead you into new roles quicker than you might think.
In practice: inform your team of the progress of your tasks; ask others how they are doing and if you can help; detect impediments early on and shed light on them; leverage written thought – documentation is essential; ask questions to understand the goal and objectives; be consistent; be open; be honest; don’t gossip and don’t be scared of making mistakes.
Luckily, a few tools exist that assist in supporting this process.
Use systems to build systems
In today’s Agile world, teams have already adopted task management tools such as Asana, Jira, Monday.com, and many more. These tools help track task progress and identify dependencies and impediments early on. Once you get into a good team rhythm, it also assists in predictability, but this is certainly no easy task.
Use systems to check coding style, audit documents, deploy new software versions, and automate inventory administration. This also includes building checklist-style reminders into your team’s workflow. Workflow (Process) automation is a silver bullet that allows you to achieve quick wins.
Process saves us from the poverty of our intentions.
Elizabeth King – Sculptor
Being a team player, having clear communication, and using systems to automate and support processes are core to operating effectively. However, these things are ineffective if you don’t utilize good systems design practices.
Spend time on systems design
If I could do it over again, I would spend more time working on the system design and document every detail I can think of. This might sound like overkill. But I remember one of the on-site engineers calling me and asking which ports should be used for the network cabling. I only realized then that some ethernet ports were configured for specific purposes, and a wiring diagram was essential.
This occurred twelve months after I submitted the design, and believe me, most of your short-term memory and system context have been long forgotten by then.
It helps if everything is documented for implementation and maintenance purposes. If you have moved on to another project and someone needs to know why something is as it is, making the implicit explicit is crucial for continuity.
After the project, I also realized that the system design would need to be updated and kept current. This hadn’t dawned on me before, but including the system design as part of the maintenance instructions is essential.
Build with maintenance and support in mind
The design should consider maintenance and support. We initially were much more user- and client-focused. Now that I’m on the support team, I’m happy we accounted for certain things and wish we had accounted for others.
The cost of change is high after a project has been delivered. Temporary fixes also become permanent features. Changing things for end-users is significantly less cumbersome if maintenance and support are accounted for early in the systems design process.
Try to standardize and purchase new components as much as possible. This will make finding spares more manageable and reduce maintenance effort down the line.
One of my regrets was reusing computers and network appliances in the system replacement project. I argued that the systems were not yet end-of-life (EOL) and could operate for several years. However, the EOL date for some of those components is now within the next twelve months.
Work within the constraints
The reality is that only some things are always that easy in practice. To ensure compliance with international aviation standards, we needed to implement a new feature on the legacy system before we were ready to go live with the new system. This meant we had to introduce a workaround method built on Java 5.
Using the latest Java release and operating system would have been great, but implementing that for legacy operational technology (OT) systems can be challenging. This did not mean we could skip out on development best practices. We incorporated the new software into a build pipeline, wrote unit tests, utilized peer review, created training material and maintenance documentation, and ensured a team supported it.
Remembering that we can only fix some things as we go along or implement the newest technology is essential. However, working with the goal in mind and within the immediate constraints could get you one step closer to the finish line, even if it is not a pretty or perfect solution.
Account for collaboration complexity
So, you have a design based on multiple sessions with stakeholder input, research, and requirements analysis; now it is time to implement. You must account for the increased time required when various teams or stakeholders are involved during implementation.
Our team recently had a system release that required us to change two IP addresses in a configuration. This was due to the introduction of a new international network into the corporate infrastructure.
To test the entire data chain, we needed to collaborate with the service provider’s engineers, the company’s network team, our team, our supplier, our first- and second-line support, our operators, and our clients.
It took six months for a seemingly simple change. This is due to the cost you pay for the complexity of working across multiple teams and company boundaries.
This could also influence your system design decisions. While outsourcing or doing it yourself might be cheaper, consider the complexity of the cost of collaboration.
Celebrate the victories
When you work on a twenty-month project, celebrate the little victories. It is great to reflect on a long period of hard work at the end of a big project, drink some beers, and eat cake with your colleagues.
But we sometimes forget that we spend many hours building things for society to use. Why not celebrate when version 1.0 of the system design has been completed, the order has been made for the bill of materials, or the first component of the system has been installed?
I did, and it kept me going.
What’s next?
With these lessons and experience in my pocket, I’m continuing this journey toward the next project and constantly searching for more ways to improve my craft.
What I’m taking with me is to remember to:
- Be a team player
- Communicate effectively
- Use systems to build systems
- Spend time on design
- Build with maintenance and support in mind
- Work within the constraints
- Account for the cost of collaboration complexity
- Celebrate the victories
What’s next for you? Feel free to share the lessons you learned from your experience, what stood out for you in this post, what you can relate to, or what I can improve in the comments section below.
Leave a Reply