It’s okay to challenge meeting organizers to ask them for agendas, your intended contribution, and intended outcomes.

It’s okay to schedule long blocks of uninterrupted time on your calendar for deep work to get into the flow state.
Mark them as “DEEP WORK - DO NOT SCHEDULE!”.

It’s okay to step away from your desk to take a break and not be available.

It’s okay for you to not reply to an email or chat after your “normally available” hours.

It’s okay to challenge and clarify the priority of an invitations/discussions/initiatives.
Is this important? Is this important AND urgent? Where should this land against other priorities?

It’s okay to vent frustrated.
If you're a leader, be a thoughtful listener, don’t jump to conclusions and solitons. Process offline and and think about how you can take one step forward at a time together to remedy.

It’s okay to disagree and challenge directly, while caring personally. Including disagreeing with leaders!

Technical Debt

I’ll usually refer to technical debt as a tool, like taking a loan out, we can do it, it might help accelerate efforts or reach the finish line sooner, but we’re leveraging ourselves. Unless we put a “payment plan” in place, we’ll ultimately end up bankrupt.

Technical debt is a tool we use sparingly.

How do we decide to take on technical debt? We're deliberate:

Product Managers and Engineering Managers work to ensure:

1. We’re explicit when we take on debt, “Hey guys, we’re taking out a loan and incurring some interest.”
2. We’re explicit about how we intend to pay back the debt, “Here’s where we’re reserving capacity on roadmap/plans to back this debt back.”

How do we pay down technical debt? We favor doing it opportunistically.

When a product team is taking on new work, we should think about each software change/additions/improvements we’re planning, and think about the systems we need to change, and then ask ourselves a few questions:

1. What’s the right way to make this change? IOW, without cutting corners and incurring technical debt. (Avoid getting trapped by the sunk cost fallacy on existing code.)

2. What would that cost to do it right from a delivery timeline perspective?

3. Is there some technical debt in our way we haven’t fully paid back yet?

   • Is now the right time to “accelerate our payments” and fix this for the new change?

4. Can we do this right and deliver on the customer outcome with a reasonable delivery timeline?

   • If yes, then we should always prefer this option and not take on any technical debt.

Thoughts on Managing Legacy Software

Avoid automatically calling any old or arcane software technical debt.

I've seen many technology leaders make this mistake and sometimes leads to poor decision making and ultimately wasted effort rebuilding systems that don't provide real and measurable value to customers or the organization.

"This system was designed years ago, nobody understands it, let's rebuild it the right way!" -- Some Software Engineer (also, me, at some point early in my career)

We must prioritize changes to systems and technology with specific and measurable outcomes for customers, the team, or the organization.

What do we instead?

If technology/software/systems are working and providing commensurate value, and there we aren't necessarily planning changes in this area, why are we devoting any energy to rebuilding it?

Let's instead make it easy to manage and change! We can do this by:

1. Ensuring the right instrumentation and monitoring are in place.

2. Ensuring the right build-and-deploy automation is in place.

Questions to guide whether we should consider rebuilding/investing:

1. What's the rate of change for this system/technology?

2. Is there an inherent risk of maintaining this technology? Vendor support, warranty, community support, etc.

3. If the answer is "yes" to either above, then it's probably a good candidate to start evolving toward more modern modular technology.

"Poised to Deploy" as an Operating Philosophy

Finally, here's a great keynote from the 2016 Velocity Conference (that I attended!), where Richard Cook talks about the value of adapting systems continuously.

He argues that it's not the system that we need to focus on; it's the ability to make changes and adapt the system that leads to value creation. He calls it being "Poised to deploy." "Being unable to deploy is an existential threat."

I regularly emphasize the importance of instrumentation and build-and-deploy tooling for all systems as a first step. This makes it inherently easier to change the system and evolve and adapt it to customer needs.

Poised to deploy: The C-suite and adaptive capacity

"Resilience is sustained adaptability. Being continuously poised to deploy is the ideal state to grasp opportunities and cope with challenges. This sustained adaptability is expensive and requires full-on DevOps. We are exploring how this all works, and the results are surprising and encouraging." -- Richard Cook

Poised to Deploy

1. Knowing what the platform is supposed to do;

2. Knowing how the platform works;

3. Knowing what the platform's behaviors means;

4. Being able to devise a change that addressees 1,2, and 3;

5. Being able to predict the effects of change;

6. Being able to force the platform to change in that way

7. Being prepared to deal with the consequences

So, what's the problem?

As eSpecialtyCorp grew from a small ecommerce startup to a company serving millions of customers, our engineering teams started to suffer from typical growing pains:

• a proliferation of patchwork technology
• a massive monolith, strongly coupled to several front-end applications
• ample amounts of "glue and duct tape" holding the pieces together

Once an environment that was easy for a small team of engineers to manage and reason about, it had ballooned and become unwieldy for even our most experienced senior engineers to work, iterate, and operate effectively. Our database footprint alone grew from three databases to several dozen (multiplied by integration, QA, and production environments). We knew that if we wanted to continue scaling, we needed to adopt modern patterns while also leveling up our engineering practice.

Monolith to Microservices; Maturing an Engineering Organization

eSpecialtyCorp had embraced microservices. We were on our way to decomposing our monolithic applications opportunistically as we'd designed new features. We'd built lots of automation around provisioning infrastructure, configuration management, and build-and-deploy tooling. Lots of elements were moving more smoothly than before, except for the database.

Remember those three databases? Nearly everyone in the technology group (and several folks outside of the group) had access to production environments, including access to sometimes sensitive and personally identifiable information (PII). We had a serious problem on our hands that posed too much risk and had to solve before our next PCI audit. The Payment Card Industry (PCI) mandates that merchants have authorized third-parties to audit their security and operating practices around how they store and manage PII.

We immediately relinquished access to the production databases from most of the team, sans our systems administrators, which immediately backfired. We'd hamstrung our teams; they couldn't as quickly diagnose problems with their apps, and they couldn't process customer support tickets, which routinely required query access to production databases. Our SLAs slipped, and our customer service started to suffer.

/gmdb to the rescue.

After quickly reinstating production access to a few engineers on each team, we knew we needed a better system in place long-term.

We started by establishing the following guiding principles for our approach.

• No one should have permanently elevated or administrative privileges to the systems that support our software. To reduce risk, we should operate in a mode that provides the least privileges possible to get our jobs done.
• Teams must have elevated access to systems when they need it.
• Elevated access must be intentional and deliberate.
• Elevated access must be temporary.
• Elevated access must be transparent and auditable.

We'd start with our database environment first, then move onto the remaining infrastructure components, servers, load balancers, firewalls, etc. Ultimately, helping our teams get the resources they needed to build and maintain good software.

In a matter of days, we'd shipped version 1.0 to our engineering teams.

The solution.

Our teams were already well-versed in collaborating asynchronously utilizing tools like Mattermost (an open-source Slack alternative). We'd started to adopt the "ChatOps" philosophy; most of our builds were reporting into their respective channels, helping create transparency around how our teams operated and helping them ship code more frequently.

We developed a series of Powershell scripts and access control lists (although fairly rudimentary, more than sufficient to test our assumptions for v1.0!) and tied them to what Slack calls "slash commands" allows users to execute commands right inside of a channel.

Here’s how the basic interface was constructed in v1.0:

/gmdb --database-server “[database-server]” --database-name: “[database-name]” --access: [read | write | pii | admin]

After entering the slash command into the channel, a Powershell script would run performing the following actions

1. Validated input
2. Validated the user was permitted to request access
3. Provisioned temporary access for the user; automatically expiring after 30 minutes of inactivity
4. Logged the request in the channel

By implementing this simple solution, we'd quickly achieved all of most of our goals. People no longer had permanent access to production environments, limiting inadvertent changes or mistakes. We now had more sophisticated logging and controls in place in support of an eventual PCI audit. And most importantly, our engineering teams could serve their customers effectively while limiting risk!

Delivering a solution like gmdb was a critical milestone in helping our teams understand how to make progress while maturing our engineering practice. That we should expect to encounter some rough patches along the way, but if we remained disciplined and could take a step back to help us better understand the problem, that we'd be more likely to succeed.

Team,

I want to take a moment to thank you all for the hard work and dedication you put in every single day to support our incredible group. In what seemed like the most challenging and tumultuous year, you all persevered and embraced the grind with the hope of making things better. I've always had confidence that we could turn the tide with our teammates' support; I believe it's begun.

So, how do we continue making progress? In what ways do we have to change to get better? How will each person on this team contribute, in concrete terms, to making us the highest-performing team?

Be radically candid with each other. Candor must override hierarchy. Build and nurture relationships with your peers and your business partners to enable a continuous loop of thoughtful and honest feedback. Without open communication, we rob each other of the chance to experiment, fail, and, most importantly, learn from our mistakes.

Embrace change. ​Be respectful, be empathetic, and ask, "can this be done differently?" I believe change begets growth. It may be uncomfortable at times, but if real and valuable learnings result from the change, it was worth the pain.

"The challenger needs support to find its footing. Protection of the new, not the future, not the past, must be a conscious effort." -- Ed Catmull, Creativity Inc.

I have confidence in our team and our leaders that next year, with continued growth and progress, we can confidently and proudly say that we're one of the best engineering practices in San Diego (baby steps).

Best,

Reza

"And it ought to be remembered that there is nothing more difficult to take in hand, more perilous to conduct, or more uncertain in its success, than to take the lead in the introduction of a new order of things. Because the innovator has for enemies all those who have done well under the old conditions, and lukewarm defenders in those who may do well under the new. This coolness arises partly from fear of the opponents, who have the laws on their side, and partly from the incredulity of men, who do not readily believe in new things until they have had a long experience of them. Thus it happens that whenever those who are hostile have the opportunity to attack they do it like partisans, whilst the others defend lukewarmly, in such wise that the prince is endangered along with them."-- Machiavelli - The Prince, Chapter 6