Designing for Brobdingnag

| categories: sysadmin, design

On the topic of talks, while at Google I was involved in an interesting SRE Classroom event in late 2013. These events were run by many of the SRE offices as a way of sharing design approaches with software and systems engineers who might lack exposure to large-scale systems.

I later gave my talk from the Dublin event to devops.ie in February 2014. PDF with speaker notes from that event:

Designing for Brobdingnag: considerations in scaling large web systems.

Today, we're focusing on technical designs in large-scale software environments, and patterns which we've found work well.

These can be relatively difficult to get exposure to.

This won't be an exhaustive treatment, but we hope to give you:

  • an appreciation of the kinds of constraints and opportunities we have at scale;
  • some tools for approaching design;
  • and a better idea of the kinds of systems SREs engage with.

Note the "Resources" slide towards the end: lots of good links on design and distributed systems there.

Based on presentations at the London and Mountain View SRE Classroom events by Matt Brown, John Neil and Robert Spier. I also include ideas from Jeff Dean's talk Building Software Systems At Google and Lessons Learned. Many thanks to Niall Richard Murphy, Alex Perry, Laura Nolan and Pete Nuttall for assistance, ideas and review.


monendi te salutant

| categories: sysadmin, fun

Talking with Niall on IRC today, I had a brainwave: with just one letter changed, the famous Latin quote morituri te salutant - "those who are about to die salute you" - could become monituri te salutant - "those who are about to be paged salute you".

A noble and fitting handover note for oncall engineers?

Sadly, as is often the case when I take enthusiastic flights into classical translation, I'm off here: moriturus is a somewhat irregular thing from the verb morior. moniturus is active, not passive: it means "about to warn/advise/notify".

To get this right, we need a future passive participle, which it turns out is supplied by the gerundive. So that gives us monendus.

monendi te salutant

Not as sweet a solution as the one-letter change complete with "monitoring" embedded, but not bad. Vale!


Alerting in production systems

| categories: sysadmin

Anyone who has been (or is going) through the wringer of a noisy pager rotation may enjoy this talk, delivered at intercom.io in Dublin on July 1st:

Pager Bound: high-signal alerting in production systems.

  • Good alerting is something that needs to be designed: organic growth tends to not go so well;
  • page, ticket or log: eliminate email alerts;
  • if we must be paged out of bed, it should be for something that really needs human attention;
  • we can only handle ~2 events well per shift;
  • service-level objectives are a really useful way to orient our alerting to customer experience & business priorities;
  • page on the symptom as it relates to our SLOs, not the cause.

Following Rob Ewaschuk's philosophy on alerting.


ICs, managers and bears! Oh my!

| categories: thoughts

Some considerations in transitioning from an "individual contributor" to a management role in production engineering.

This is based on a document I wrote a couple of years ago at work. The question of my becoming a manager came up, so I had a series of brief discussions with people who know a thing or two about it. This is my synthesis from notes; any errors are mine.

Some of this is perhaps specific to a large company, but I think a lot applies generally to engineering management. As it turned out, I stuck with being an IC, but discussing it and writing up helped me decide. I hope others find it useful.

What's good about being a manager?

You can make a big difference to your team and your organization, and you can influence what is happening in the company at large. You are able to help people more than as an individual contributor, and it's rewarding to see the people in your team grow as a result. You can enable people to do cool things.

You will almost inevitably increase your impact (for better or for worse). You have a high level of responsibility and investment; the success or failure of your team is on you. You can sharply define your identity, to yourself and to the rest of the company.

The network you build is good from a promotion point of view, and it may be easier to be promoted on the engineering management track than the (senior) engineer track. Remuneration is good.

What's difficult?

You have to deal with sensitive, interesting problems and need to be prepared for difficult conversations and interactions. Dealing with under-performers or firing people can be really hard. You need to learn how to do this effectively to avoid serious team morale and performance problems.

You must adjust your expectations of what it is to feel productive: you lack independent progress and quick feedback on your performance day by day. Progress and results are apparent only over very long periods - 6 months or a year. You need to take a longer view of your achievements and those of your team. When your work does bear fruit, your part in it may not be visible.

You may have to play a role at work more than as an individual contributor. What you say and how you say it affects the performance of others directly, and as such it can be harder to be yourself. It's significantly harder to relax with the team when ultimately you're responsible for assessing their performance.

Career options can be limited. For example, moving around in a big company as an engineer is relatively straightforward; as a manager it can be difficult - teams relatively rarely have openings for managers. Similarly, your team winding down or merging with another in the same location can leave you at a loose end. The way you need to manage your career is different.

In any case, transitioning from individual contributor to manager is likely to be personally difficult, although the vector for the difficulty varies greatly between individuals. If you're doing it right it tends to be a growth experience; and like all such, likely to hurt.

What makes a good manager?

Good engineering managers are technical. This is important. Having a background as a technical contributor means you have some insight into the day-to-day work of your team. It's easier to delegate difficult tasks if you've done the same kind of work yourself.

You need to be interested in taking on a high level of responsibility. Managers make judgement calls and bear the burden if things go wrong. You have to be reliable. People need to be able to count on you.

You have to think in a larger frame, set team priorities accordingly, and assign work that is appropriate, useful and high-impact. You need to know everything that is going on in your team.

You need to enjoy your job: if you are not happy about what you're doing as a manager everyone around you will suffer.

How can you avoid being a bad manager?

Finding good mentors (including but not limited to your direct manager) is important. The learning curve as a new manager is steep.

Delineate responsibilities so it is clear what part your team plays in the organization. Consider logical ways of organizing responsibilities rather than historical ones, and push for them. Your team must own its success or failure, and not be in a dependent position. Make sure you own your objectives and that the team has room to progress.

If you think of a spectrum between "working for the business" and "working for the team/person", you can be a poor manager by being at either extreme. There needs to be a good balance here.

Delegate; trust your team. Listen carefully to what they are telling you and make sure you put yourself in a position to help them. Don't take on too much external responsibility and lose interaction with your team as a result. Don't be a choke-point for information flow into and out of your team.

Have enough meetings to make progress, and learn how to make meetings successful.

Be prepared to drop technical issues that interest you, and focus on getting technical work done by organizing your team instead.

What are good reasons to become a manager?

You're excited by working more closely with people and nurturing them. You're interested in how organizations work, how teams work, and how large things get accomplished. You believe that you will contribute more than you could as an individual contributor.

And bad ones?

You think it will look good on your CV. You love the feeling of power. You want to build a glorious empire. You're being pushed into it by someone else.

How could you explore being a manager?

Training can be useful, both at the individual contributor and manager levels. That said, there's big switch and learning curve when you go from having no reports to having some.

The nearest prior experience you can get is management-style work, for example leading a project with others or finding a project where you can take significant responsibility. Getting to know other managers and finding out what they need help with at the management layer can be useful.

In any case, you need a transition plan. Talk with your manager(s)/mentor(s) about what your plan could look like. Note that doing management does not necessarily equate to having a management job title. There are plenty of senior people with management responsibility who don't.

If being a manager is something you want to try, understand that if you can't make it work for you in the long run you can always step back.

Should you do this?

You can think about the difference between junior and senior people as junior people adding force where senior people multiply it, i.e. help everyone to be more effective. One way to "multiply" in this sense is to manage people; another is to continue as a senior individual contributor. It's comparatively easy to be a multiplier as a manager, but it's a different skill set. Consider the best way for you, but understand that companies do need senior ICs, and going into management is not the only way to progress.

Make sure this is something you want to do. It needs to be driven by you, not by circumstance or your managers. It needs to be the direction you want to go in.

With thanks to Astrid Atkinson, Colm Buckley, Dave O'Connor, Dermot Duffy, Kate Ward, John Looney, Niall Richard Murphy, Rob Ewaschuk and Sarah Magee.


Beyond Corp

| categories: sysadmin

For the last couple of years at work I've been part of "Beyond Corp", a programme to

Re-architect corporate services to remove any privileges associated with having a corporate network address.

Doing this at a large, 15-year-old company with an extensive legacy IT infrastructure is hard. It's been interesting.

Earlier this month, my colleagues Jan Monsch and Harald Wagener presented a talk about the programme at LISA '13. It's a detailed overview of our background, vision and architecture, along with a discussion of challenges we've met along the way. I had planned to present, but family life intervened and Harald heroically stepped in. :o)

It's a great talk; strongly recommended to anyone with an interest in modern security and management of large, mobile client networks.


Next Page ยป
Site managed with Blogofile.