The use of sophisticated information technology to control nation-wide utility delivery increases efficiency but introduces new hazards. There are lessons to be learnt from those with long experience of testing such critical infrastructures, according to David Hill, Vice President, EMEA, Spirent Communications.
According to the British government’s energy.gov website, the term ‘smart grid’ generally refers to a class of technology people are using to bring utility electricity delivery systems into the 21st century. It’s a great idea, but with one major drawback: the 21st century is becoming a dangerous place for innocent newcomers, however smart. According to a US Defence Secretary, attackers are targeting the computer control systems that operate chemical, electricity and water plants and those that guide transportation throughout the country. A cyber-attack perpetrated by nation states or violent extremist groups could be as destructive as the terrorist attack of 9/11.
The idea of a smart grid is being applied across a wide range of public utilities, transport and industrial control systems, but the most obvious example is electricity, where a ‘national grid’ delivers power to homes and business via wires, substations transformers and switches. This is largely a one-way transfer, from power station to the consumer. Meanwhile data from the users and network infrastructure has traditionally been gathered by sending out staff to read meters and engineers to inspect the system for broken cables or faults.
A smart grid, however, uses two-way communication and computer processing to gather such information. This not only saves money by reducing the need for field staff, it also allows real-time data gathering that can increase efficiency. With the move to ‘green energy’ it becomes all the more important to respond quickly to changing consumer demand and faults and a smart grid makes that possible.
An electrical grid is fundamental to electricity delivery, but the use of electrical control grids extends to a whole range of national services: transport, water and waste, gas supply and industrial control systems all rely on instructions transmitted from a central controller to outlying switches or traffic signals. And in every case there will be potential benefits from upgrading these into two-way smart grids allowing, for example, the immediate detection of leaks in a water main.
What sort of communication is used on a smart grid? The default option for most computer communications is internet protocol, so that the smart grid becomes an intranet linking all its parts. And when it comes to connecting individual homes and far-flung sites it makes good sense to connect via the ubiquitous Internet rather than lay new cables. This is exactly where the greatest vulnerability can arise – welcome to the 21st century!
In theory, a utility smart grid could be totally independent of the internet, running on its own dedicated cabling across the nation. But in practice it often makes sense to use existing telecommunication lines rather than laying new cables, especially to far-flung sites. The same thing has been happening in business organisations, where previously independent systems such as fire and burglar alarms, smoke detection and industrial control systems increasingly run across the same corporate IT network.
But even if an attempt is made to quarantine the smart grid from the internet, it is not easy to maintain that state. Corporate IT networks have for years been facing the challenge of ‘network permeability’; whereas the earliest networks consisted of isolated computers linked by cables, today’s networks have to cope with mobile staff plugging in their laptops anywhere on the network, also with wireless access from smartphones and with data transfer via USB memory sticks.
So, instead of sealing the network from the internet, the main focus has been on ways to run secure services over the internet. A host of solutions are available, including firewalls, intruder detection systems and deep packet inspection devices to examine all the traffic on a network and look for anomalies. These forms of protection are very necessary, especially for a nation’s critical infrastructure where so much is at stake. Don’t be fooled by the argument that public utilities rely on highly customised systems, with no two alike, so that hacking them would be impossible without insider knowledge. Hackers have long known how to get such knowledge. Specific instances where intruders have successfully gained access to these control systems are well documented. We also know these disruptive influences are seeking to create advanced tools to attack these systems and cause panic, destruction and even the loss of life. However, as everyone with experience in IT networks knows, every addition to a network, however necessary, increases its complexity. And that makes it harder to predict.
The testing imperative
So the real challenge for any national smart grid is this: how can we secure a highly complex system? Unless the engineers managing the grid have long experience with complex networks – the sort of experience gained over decades by telecommunications and IT network engineers – it is easy to underestimate this challenge. Security vendors will assure you that their products will make the network utterly secure, and provide good reasons to justify their claims.
So, consider this ‘thought experiment’, imagine that you have won a prize to be the first civilian to fly to the moon. It’s exciting, but also scary. So the NASA team spends a whole day showing you the blueprint of the spacecraft and explaining all the fail safe and security provisions built in. That is encouraging, but still pretty scary. Now imagine that you are also told that the same space ship has flown hundreds of moon missions and never once failed. That is far more reassuring, because we humans do have an innate sense of complexity and know that, however smart the design, a complex system can only be proven by repeated testing.
And that is how it is with today’s complex IT networks: you design in all the safety and security features that are needed, but you then submit it to rigorous testing under realistic operating conditions as well as extreme loads and attack situations in order to make sure it is secure and also to allow fine tuning of the network for optimal performance.
This sort of testing is critically important for the sort of smart grids being planned for our national utility infrastructure. The good news is that there are companies that have long experience in testing IT and telecoms networks and sophisticated tools are available to facilitate testing of highly complex networks under real-world conditions. The first lesson from years of network testing is that you need two types of test: security testing and performance testing.
The need for security testing against attacks and system faults is obvious, what is less obvious is that a complex network can develop surprising problems under varying loads. A telecomms network, for example, might be able to handle hundreds of gigabits of data per second during file transfers and yet fail at a much lower bandwidth when handling a mix of different types of traffic – e.g. video and VoIP. So it is necessary not just to test its greatest data capacity but also to test how it performs under a whole range of realistic traffic scenarios and combinations of traffic.
Security testing should include three main stages. First the sort of assessment that might be provided by those security vendors: an experienced eye looking over the existing or planned grid for obvious weak points or vulnerabilities, and making sure they are protected.
The second stage is to simulate actual attacks under real-world operating conditions. Today’s sophisticated test tools not only simulate all combinations of normal operating conditions but also combine these with state-of-the-art malware attacks. Most relevant to a national grid are the so-called denial-of-service attacks that could cut off users from the service and cause widespread panic. The most advanced test tools are integrated with a cloud database that is kept up to date with every new attack or virus as they occur, rather than waiting days or weeks for patches to be distributed.
The third stage is to explore further for unknown vulnerabilities and today’s smart test solutions have the flexibility to allow very detailed testing around the boundaries of normal operation. For example: what happens when a system requires a long pass-code to be input and an operator mistakes a capital ‘O’ for a zero? Does it simply report an error, or does the wrong type of character crash the system? The best test solutions allow for ‘fuzz testing’ – testing such variations from normal behaviour to anticipate problems that might accidentally arise.
And then we come to performance testing. Today’s networks have to carry many types of traffic – data, video, voice, control signals etc. – and a range of different protocols for each. It is not enough just to know the maximum bandwidth capacity but also how the network behaves under a whole spectrum of different operating conditions. The right test solution in the hands of an experienced network test engineer will be able to test the network to all its limits, and provide clear reports to show where problems could occur.
The ideal is, of course, to have a grid that can handle every operating condition and survive any type of attack or local fault, but this is hardly realistic. The real value of a comprehensive network test report is often that it spells out the system’s limits, rather than saying it is perfect. So, during a crisis, when a certain type of traffic is surging, the grid operators know where the danger point lies and can take precautionary steps before that point is reached.
What sort of standards exist for the security of a nation’s critical infrastructure? In the USA, the Department of Homeland Security is working with service providers. But information sharing alone is not sufficient. It is essential to work with the business community, to develop baseline standards for our most critical private-sector infrastructure, including power plants, water treatment facilities and gas pipelines.
In Britain, The Centre for the Protection of National Infrastructure is collaborating with the British Standards Institution to create standards addressing a number of critical concerns such as mail and food and drink security. The Office of Energy Delivery & Energy Reliability’s Energy Delivery Systems Cybersecurity web page also offers useful guidelines for securing the grid.
Meanwhile, it makes a lot of sense to speak to any company that has spent years in the business of testing the performance and security of the world’s most critical financial, government, medical and corporate networks. Today’s utility grids are getting a lot smarter, but to survive in the 21st century they also need to be streetwise – and that demands a whole lot of experience.