Where do you work and what's your role?

I currently work at Cisco as a Network Consulting Engineer (NCE). This is a role that combines consulting services and networking technology into a customer facing position within Cisco's Advanced Services division. The role, in my experience, is a good mix of design and implementation. Those in the role are expected to be technical experts in a number of technology areas and have strong design skills. Some NCEs support one customer on-site full time, while others spread their time among many customers and offer high-level design support. I fall into the former category, but occasionally support other customers when the need arises.

What kind of roles and work did you do before arriving at Cisco?

I worked as the Network Architect for a US Government organization. I now support this customer full time as a Cisco employee; I sit in the same chair and interact with many of the same people. The work in my previous role was sometimes that of a senior designer or architect, but rarely so. I spent much of my time doing low-level implementation (great for technical skills) but many of the design decisions from last decade were unshakable, even in the face of needed change. This is not a ding on the organization, but rather the cemented mindset of what the network "is, was, and will always be".

What have you learned in your career so far that you would like to tell the younger you?

Often times, people mistake Microsoft Office skills and meeting attendance for being a good architect. That seemingly grizzled "architect" in the conference room, reclining in the fancy chair with crossed legs, should be able to actually deliver value to the organization. Give them the benefit of the doubt and take their advice at face value; never more. Believe 50% of what you read and 0% of what you hear. When you find yourself in an architect's position 10 years from now, always remember that mastery of technology against specific business needs is what makes a good architect. See through the chaos and the political maneuvering that grips many organizations in both public and private sectors. Present rational, fact-based arguments either supporting or challenging a given design.

Additionally, "failing hard" is more than a phrase. You actually have to break things to fail. Don't be afraid to make significant changes to a production network during working hours provided you've mitigated outage risks and sought approval to do it. Presenting a good plan will go farther than you think. You miss 100% of the shots you don't take.

What are the most important skills you have picked up in your career so far?

As Russ White says, learning to think. I do not claim mastery on this topic, but I've met a lot of senior engineers (highly skilled ones, too) who know how to configure/troubleshoot something, but do not understand how or why it actually works. Many engineers can configure, troubleshoot, and explain NAT/PAT without much trouble, yet only a small subset can actually describe the NAT order of operations. Even fewer can explain why the order of operations was built as it was. Another example includes understanding multicast delivery trees related to PIM. Many engineers cannot explain which PIM routers are roots or leaves for given delivery trees and instead opt to describe the routers of PIM routers in more vague terms. This isn't about memorizing message types or data communications exchanges. It's about understanding how the technology provides a service or solves a problem, and why. It's about being able to compare and contrast different technologies around the context of business problems.

What's your opinion on degrees? Are they useful for someone in the networking industry?

It depends. I could make some probably controversial statements that having a degree in Fine Arts could be more valuable than one in IT. The former teaches you to think and evaluate seemingly arbitrary things, lacking a discrete measurement scale for success or failure. The latter might do the same, but is more likely to teach you skills that you could otherwise learn after graduation. I earned a BS in Computer Science from RIT in 2008. The school is considered to be a good one for this program, and I graduated with High Honors. I considered myself a very good programmer upon graduation. I worked at a large company as a co-op/intern through college writing some pretty advanced automated test suites for a large product line. These days, I meet developers with near zero formal education yet are extremely skilled. They would absolutely embarrass my skills from 10 years ago when I was in my programming prime.

I never studied IT in college and graduated only knowing the high level differences between TCP and UDP; I consider myself a highly skilled networker today. Bottom line is that universities, in my opinion, are a superior option for teaching people how to reason, and less so for hands-on technical skills that might be antiquated within a few years. This is to be taken within the context of IT, not something like electrical engineering, where circuit knowledge doesn't change every few years.

What about certifications? Are they losing their value?

See comments above. Certifications, unlike degrees, are typically more malleable and subject to change. They (should) target an ever-changing market. They adapt faster and are offered by a variety of vendors, neutral organizations, standards bodies, and others. Certifications, in name, are self-evident; it certifies that you know a specific set of things outlined in the syllabus or blueprint. Degrees are meant to be similar, except again, I believe the true value of a degree is learning to think, not the concrete skills retained after graduation. Holding two CCIEs, CCDE, and several other certifications, I can say that they have been massively beneficial to me. They've taught me grit, determination, and a whole slew of technical skills I didn't think I'd ever know.

Is the skillset of network engineers changing? What skills are important to have in the coming years?

Everyone you interview is probably going to give you the same answer here. I'll stop talking about "learning to think" since its probably clear that I strongly believe that is a critical skill. I also believe that networkers should have familiarity with programming, automation, and general systems knowledge. I also believed this 10 years ago. Hands-on in the best way to learn it, however, don't become consumed with the current fads. Python and Ansible are powerful, but are fads, and will probably not be tools of choice in 10 years. I would suggest learning to use both (as I am actively doing right now), but while doing so, understanding programming concepts more than Python and YAML syntax. Basic programming concepts like object oriented design, polymorphism, inheritance, functional decomposition, and error handling are important to writing good software. I see a lot of network guys claiming to be "learning to code" just copy/pasting from other people's Githubs and Stack Overflow. I see them pasting chunks of code in a single .py file without much thinking, modifying a few fields and calling it their code. This is not coding. While it might be an appropriate solution to meet immediate business needs, we should not confuse this with mastering a new skill.

On the topic of general systems knowledge, this is mostly targeted at Linux environments. You should be able to easily navigate the Linux shell, including managing files, users, permissions, processes, and computer resources. You should know how to add/remove disks, partition hard drives, and set up network interface. I would also recommend learning basic shell scripting, sed, and awk. Some of these tools require serious wizardry to understand in depth; the goal is to know how they can help you. Just yesterday I used a handful of shell commands (including sed) from an Ansible playbook to perform backups of all IOS devices in a network.

What skills are important for Network Architects?

I think I answered this above, but I'll clarify it more. An architect's job will undoubtedly require meetings, politics, and other undesirable realities. Those do not define the job, though. A good architect or designer (I'm not going to split hairs on the differences here) should be able to understand the organizational requirements and present options based on them. The requirements will often come with constraints around cost, timeline, lifecycle, and things that are not necessarily technology related. The best technical solution is often not the best business solution, and good architects will appreciate this.

A critical skill for architects, in my opinion, is the ability to model. Think about a civil engineering architect that builds a sports stadium. Chances are that this individual and his/her team has built a small 1:5000 model or something of the sort. In building the model, the architect actually built what was going to be made on a smaller scale, and had to consider many things during this process: building foundation, surrounding environment (trees, power lines), avenues of approach such as roads or walking paths, vehicle parking lots, and other things. Doing this in Powerpoint is not enough, yet I see it all the time. If you are an architect, BUILD A MODEL. If you are rolling out a 5,000 node Internet VPN WAN design, set up 10 routers in a lab at a minimum. Today, it might even be possible to build the entire thing. Personally lead the effort with your architecture team, if it exists; if it doesn't, build it yourself. You MUST be able to clearly articulate how the system works. This is not meant to be a "dust off your fingers for CLI retraining" drill. It's meant to show the organization that you do, in fact, understand exactly how this new design will work. It shows that you've considered all angles of the solution, identified and discussed trade offs, and made adjustments based off your actual findings, not feelings of confidence.

Will the need for networking experts go away? Is it better to be a generalist than an expert?

A good team will always have both. Consider a handyman's service truck. It's likely that in the truck, he has many specialty tools, such as pipe and tubing cutters for occasional plumbing work. He will also have tools that serve more than one purpose, but still a small set of purposes, such as hammers and pliers. Then, he has general-purpose tools like duct tape, WD-40, and maybe a Leatherman/Gerber style multi-tool or Swiss Army knife. Sometimes the multi-tool is all he needs; when I was in the US Marine Corps, almost every Marine had one on his belt. It was a low cost and lightweight 70% solution to most mechanical problems observed in the field. It doesn't make sense to carry unwieldy and expensive specialty tools for jobs that don't require it, akin to hiring a CCIE to rename VLANs in your campus access layer. Alternatively, it doesn't make sense to use the screwdriver or a multi-tool when there are many screws in need of securing (use a drill).

To better answer the question, I would recommend about half of an IT organization should be generalists and the other half should be a collection of specialists. The generalists can be spread across the different silos (oh wait, I thought we busted these down ...) like network, server, storage, and customer service. Individual experts will only exist in roles where their skills are maximized.

What do you think of soft skills? Do we need them in the networking industry? If so, which ones are the most important?

The number one soft skill is the ability to listen. Just shut up. Understand the viewpoint being presented. Don't listen for the sake of arguing or waiting your turn to speak. Sometimes you may find yourself humbled by what others have to say. A good engineer or architect will see it as an opportunity for learning rather than an external threat. Those with more feudal mindsets towards corporate organization will probably struggle with this advice, though.

Other skills like presentation, public speaking, eye contact, are important too. But these come with practice, experience, and naturally improve over time. The will to listen takes discipline.

With SDN, orchestration etc. can we throw the "traditional" networking knowledge out the window? Why or why not?

I'm laughing out loud over here. Absolutely not. SDN isn't a single thing, it's a continuum. Russ White and Denise Donohue summed this up nicely in their book "The Art of Network Architecture". I also covered it briefly in a free whitepaper on Cisco Evolving Technologies. Paramount to understanding SDN is to define it simply. There are many competing definitions of it, but I strongly believe that in order to achieve SDN at all, you must separate the control plane from the data plane, effectively centralizing forwarding decisions off-box. I wholly reject the belief that "SDN is management" and that a fancy UI with some point-and-click tool qualifies at SDN. If so, we've been doing SDN for 15 years already. The degree in which routing decisions are centralized reinforces the belief that SDN is a continuum. Cisco IWAN provides central policy application but retains individual device control planes for failover. This comes at a dollar cost; other fully centralized solutions which are touted as "the only SDN solutions" fully centralize routing decisions, leaving the forwarders dumb and cheap. Neither is better than the other in a general context. Within the context of specific business constraints, however, one solution might be a more appropriate fit.

Another interesting aspect of fully centralized SDN designs is that the distributed control plane ever really goes away, assuming there are multiple controllers. Generally speaking, state still has to be synchronized between controllers, which means issues like consistency, availability, and partitionability (CAP theorem) still apply. You've just moved the distributed control plane from your "legacy" network to being between your controllers, which is probably atop a small, distributed control plane network itself. This isn't a bad thing in and of itself, but its important to divorce yourself from the belief that distributed control planes are dead. They've just been moved.

Should someone in the networking industry learn to code? Why or why not? What is your language of choice?

I mentioned it above. I believe the answer is yes. Not just for the massively hyped automation craze of late, but because programming is both an art and science that requires a lot of individual, unconstrained thinking. This is healthy for any engineer.

Language of choice is a question I can't really answer. Whatever I say is only going to be relevant for a few years and I feel like I would be inflaming a fad by saying Python. Specifically within the network industry, this one is the most applicable and is probably worth learning ... for now. But I would implore networkers out there to explore other languages. I've done quite a bit of work in C, C++, C# .NET, and Java. Each of them has some interesting behaviors and a unique set of trade offs.

One thing I liked about C# was that compiling code and exporting a Windows .msi installer file was trivially easy. It was a convenient way to write custom applications for myself and run them on my home PC 10 years ago. I wrote one that helped me manage MP3 files because I couldn't afford to buy Apple products. I had a cheap MP3 player and needed a way to sort tracks, rename files, and edit song data. C# seemed like the right tool for this application. For networking applications, Python is often the right tool ... again, for now.

What's your best advice for staying updated in the networking industry? How do you stop the sipping from the firehose?

Focus on what matters to you and branch out slowly. Example: I wanted to get into programming (after 10 years of not doing it) and automation, so I started with Python and Ansible. I also signed up for AWS and decided to do most of my testing there. These were good choices and I was able to grasp the core concepts quickly. I've written several production-ready playbooks in Ansible for work. I've tested routers in AWS tying in via IPsec VPN to our network. I've solved a few Google codejam challenges on my Github in Python.

The guys at Network To Code probably know how comparatively weak I am with Ansible. Real AWS experts would laugh at my VPC design. Real developers probably wouldn't even read my codejam solutions. The goal isn't to be an expert outside of your specialty area; it's to understand the basics and integrate them to your daily routine.

Recently, I decided to wade into OpenStack, Puppet, Chef, and a bunch of other things that are currently out of my league. I could probably hack them together to turn them on, but where is the value? I wasn't using these things at work, and they didn't seem to fit the use cases that I envisioned. So, I've personally decided to table those for another day, perhaps when I start to find use cases for them. I always find that when a certain tool or technology has a direct impact on something I am doing, it is much easier and more enjoyable to learn. This is ultimately how, after finishing my first CCIE and vowing never to do it again, that I earned a second CCIE followed by CCDE. I needed a reason to do it, and the reasons began to present themselves over time.

Before we close out. What would you want to give as a final piece of advice to the NetworkCareer readers?

Talk is cheap.

Subscribe by adding "www.njrusmc.net" to your RSS/Atom reader!

All Blogs -- Main Page