Why Internet of Things (IoT) Software Testing Matters

The term Internet of Things (IoT) refers to a system of network-enabled devices, components, or services that publish and/or consume data. The most familiar IoT environment is probably the home automation system in which lighting, security system, irrigation system, etc. can be controlled via a central device, such as a smart phone. But an IoT environment may also enable machine-to-machine communication for safer, more efficient industrial uses, such as a “smart” utility grid or for factory automation.

Transportation networks, such as the “connected car” and rails systems, are also IoT systems. Components within a car exchange data to provide monitoring and functionality to the driver. Trains within a rail system also act as a data consuming/publishing components and have been doing so for decades. Other examples include:
  • Defense networks and dedicated secure networks
  • Medical systems and devices in an emergency room that publish/consume data about the patient, including vitals and medication consumption

These components work together as a holistic system, simulating their complex interactions, reproducing individual components, and testing functionality and non-functional requirements is very difficult.

IoT environments typically connect over a public network. As a result, there are several characteristics beyond IoT device makers’ control:

  • Security: You can control the security at the software level, but the network might be exposed to attacks
  • Reliability: You can control software resilience, but network availability might be beyond your control
  • Scale: You can control scale of internal infrastructure, but not the public network.

It is up to the device maker to ensure that the above software characteristics are well tested at the component level. Scenarios should also be developed and tested to handle errors that result from events outside the system, which can be done with service virtualization and API testing.

Systems of Systems

An IoT environment can also be a system of a system. A connected car, for example, is treated as a component connected to the Internet. But within a car, there are also several smaller components that can either interact with each other or independently connect to the Internet.

The entertainment system, OnStar, etc., can connect directly to a network. Other components are connected to the CAN bus, which is a well-protected component that controls all of the specific functions in the car. The steering components, braking components, etc., need to communicate with the CAN bus to report when there are problems that need to be broadcast externally. This introduces a potential security problem, as well as potential conflicts as different systems transfer data through the CAN bus. As a result, there are different layers at which testing must occur, such as:

  • Discrete components
  • Component within the context of a subsystem
  • Component within the context of a larger system

Safety Critical Considerations

IoT systems represent a potential point of failure in safety-critical systems, such as automotive, avionics, etc. If you are working on an IoT project in a safety-critical market, you should assess the level of risk associated with each component to determine the level of required testing. There are two aspects: the risk associated with component failure and the recoverability.

You can still control most cars if the power steering or power brakes fail, but mechanical connections are increasingly being removed in favor of control-by-wire technologies, which seek to increase the overall efficiency of the system. While a failure of the braking system is catastrophic, it may also be recoverable in systems that do not have control-by-wire technology implemented.

The flight control systems (FCS) that control surfaces of the airplane are another example of changing risk assessments. The failure associated with the FCS represents a SIL 1 safety-critical risk. This is a catastrophic, but recoverable, failure. There is a hydraulic system that the pilot can leverage to keep the plane in the air. In some systems, however, the control surfaces controlled by the FCS are critical to the plane remaining in flight that a failure in the FCS represents a SIL 0 risk. The complexity of the control surfaces has reached the point where a human cannot control them without the FCS and is therefore not recoverable.

Both of these examples represent systems of systems within an IoT environment. In each case, there are critical components that must be tested at different layers.

To learn more about how to test IoT environments, download the full white paper: End-to-End Testing for IoT Integrity