Configuration drift on the network

Every network starts with a standard. Devices are built the same way, with the same authentication, the same logging destinations, the same access lists. Then a change is made at two in the morning to restore service, another is made to accommodate a supplier, a device is replaced and configured from memory, and eighteen months later no two switches are quite the same.

This is configuration drift, and its cost is that you no longer know what your network does.

How it happens

Emergency changes. Something breaks, a change fixes it, the change never goes back through review. The fix stays, undocumented, and it is now part of the running configuration.

Replacements built by hand. A failed device replaced under time pressure, configured by someone recalling how it should look or copying from a neighbouring device that had already drifted.

Temporary exceptions. A rule added for a project, a port opened for a migration, an access list relaxed for a supplier's engineer. Almost none of these are removed, because removal requires someone to remember they exist.

Vendor defaults. Firmware upgrades occasionally reintroduce settings or change behaviour. A device upgraded at a different time from its peers can end up materially different.

Multiple people, no single source. Two engineers with the same intent will configure a device slightly differently. Neither is wrong. The estate is now inconsistent.

What it costs

Troubleshooting takes longer, unpredictably. When a device does not behave as the documentation says, the first hour of an incident goes to establishing what is actually configured. That happens during the incident, when it is most expensive.

Changes have unexpected effects. A change tested against the standard configuration behaves differently on a device that has drifted from it. This is a common cause of changes that work in test and fail in production.

Security posture becomes unknown. Access lists that were tightened once and loosened since. Management interfaces reachable from more places than intended. Logging that stopped being sent somewhere useful. Each is individually small and collectively means you cannot answer what is exposed.

Audits become archaeology. Being asked to demonstrate a control is applied consistently, when it is applied inconsistently, turns a routine request into a week of work.

Recovery is unreliable. Rebuilding a device from a documented standard produces something that does not behave like the device it replaced, because the device it replaced had accumulated changes the standard does not contain.

Detecting it

The mechanism is straightforward: hold a definition of what each device should look like, collect what it actually looks like, and compare.

What makes it work in practice:

Collect configurations automatically and often. Nightly is reasonable. Store them somewhere with version history, so you can see not just that something differs but when it started differing.

Compare against a defined standard, not against each other. Comparing devices to their peers finds inconsistency, which is useful. Comparing to a standard finds inconsistency and tells you which one is wrong.

Normalise before comparing. Configurations contain timestamps, counters and ordering differences that are not real drift. Without normalisation the report is noise and gets ignored.

Alert on change, not just on difference. A configuration that changed outside a change window is worth knowing about immediately, regardless of whether the result matches the standard.

Fixing it without an outage

The instinct is to reconcile everything to the standard. Doing that quickly, across a production network, is how you cause an outage.

A safer sequence:

First, capture the current state properly. Before changing anything, have a complete, versioned copy of every device's configuration. This is also your rollback.

Then classify each difference. Every drift item falls into one of three categories: it is a mistake and should be reverted, it is a legitimate change that should be added to the standard, or it is legitimate but specific to that device and should be recorded as a documented exception.

That triage is the actual work, and it needs someone who knows why things are the way they are. Expect to find changes nobody can explain; those need investigation rather than automatic reversion.

Then update the standard. Frequently the standard is what is out of date, not the devices. If eleven of twelve devices share a setting the standard does not contain, the standard is wrong.

Then remediate, in small batches, through change control. Least critical devices first. Confirm each batch before proceeding.

Keeping it from returning

Drift is a process problem, and it returns unless the process changes.

Every emergency change gets a follow-up ticket. Raised automatically at the time, requiring the change to be either formalised into the standard or reverted, within a defined period.

Build from the standard, never from memory or from a neighbour. Device replacements should start from the current standard configuration, with device-specific values substituted.

Give exceptions expiry dates. Anything added for a project or a supplier should have a review date attached at the moment it is created.

Report drift where it is seen. A weekly summary that goes to the team, showing what changed and what differs. Visibility is most of the control; drift accumulates in the dark.

Reconcile the device inventory. Devices that exist and are not being collected from are invisible to all of the above. Compare the collection list against the network inventory regularly, because the gap is where the worst drift lives.

The realistic goal

Zero drift is not achievable in a network that is being operated. Networks change, and some changes are correct and urgent.

The goal is that every difference between the standard and reality is either known and documented, or detected within a day. That is achievable with modest tooling, and it converts an unknown network into a known one, which is the whole point.