The demand for automated and partially-automated driving functions raises one of the biggest technical challenges facing today’s automotive industry. Cars are no longer mechanical machines that simply serve as mobility extenders – they are becoming complex electro-mechanical systems that provide mobility experiences, with vast arrays of sensors, multiple communication networks and as many as 150 on-board ECUs.
This trend isn’t confined to car makers working towards high-level autonomy. Since 2022, it has been mandatory for all cars sold in the European Union (EU) – one of the largest passenger car markets in the world – to feature Advanced Emergency Braking (AEB) and a Lane Keep Assist System (LKAS). This means that even the simplest and the most affordable cars sold in the EU region must feature at least a rudimentary ability to apply braking inputs and steering corrections based on sensory data from vehicles’ surroundings.
For other markets, the targets may be even more ambitious. Numerous organisations are currently testing vehicles that are capable of full hands-off autonomy in complex situations. For OEMs working on developing these vehicles for the consumer market, the complexity of sensor fusion technologies (which involves validated software and hardware integrations) is exponentially greater.
Aside from fulfilling manufacturers’ legal obligations, there is evidence to suggest that Advanced Driver Assistance Systems (ADAS) could be a significant factor in consumer’s purchase decisions. Among decision-making criteria, safety is consistently identified as a major concern for car buyers. Thus, organisations such as Euro NCAP have begun grading cars on customer-facing performance ratings of various active safety systems.
In total, the value of the Connected and Autonomous Vehicle (CAV) market is forecast to reach close to $USD 13,632 billion by 2030. Exactly how close the industry is to true self-driving autonomy remains to be seen, but the quest to reach this goal is driving vast amounts of research and development – much of it based around virtual testing and simulation.
In this paper, we’ll focus our attention on Driver-in-the-Loop (DIL) simulators and the role they can play as automotive digital product development tools that support the engineering needed to design and deploy safe and effective on-board ADAS and full CAV systems.
As we’ll discover: It is DIL simulators’ innate ability to connect real people with real and imagined hardware and software that provides new opportunities for vehicle manufacturers as they explore future mobility solutions.
At first glance, vehicles with significant ADAS and autonomous capabilities might appear to cast human drivers and occupants as redundant participants in future mobility scenarios. But quite the opposite is true. As it turns out, the responses and bevaviours of humans interacting with on-board control systems become even more central for developing the market-acceptable vehicles of tomorrow.
User experience is key. There can be many questions about how customers inside vehicles will perceive assisted and autonomous interventions. Does the car’s behaviour instill confidence in its ability to transport them safely to their destination? Is it a comfortable and relaxing (or indeed invigorating) experience? Are control logic schemes fundamentally flat – or can code convey some sense of a car’s brand or character?
All these questions can be investigated with virtual product development tools – namely, tools that encourage human-vehicle interactions during the vehicle design and development phases. Driver-in-the-Loop (DIL) simulation is one such tool, one that provides a safe, repeatable, cost-effective and environmentally sound means of studying human interactions and intentions with vehicle systems.
In theory, many vehicle operating 
scenarios can be evaluated with offline simulations, which have historically proven to be effective engineering tools for iterating through the more straightforward aspects of vehicle design characteristics. But even the most basic CAV and ADAS functions involve human interaction of some sort, thus rendering offline, strictly-computational simulation tools ineffective.
For example, it’s a well-documented phenomenon that overly-intrusive vehicle assistance systems are frequently switched off by drivers. Something as innocuous as a Blind Spot Monitoring System (BSMS) that’s frequently illuminated on a busy road might be disregarded by most drivers if it is not effectively and informatively presented. At its worst, an unexpected intervention from a system such as AEB can be highly disconcerting, and potentially even dangerous if there are other vehicles following close behind. Offline simulation can generally validate vehicle responses to pre-programmed interventions and control strategies, but human involvement is required to assess how such things might feel.
For active safety systems, a major challenge is determining the proper point of intervention. Knowing at what point to intervene in a complex, unfolding scenario is key to striking the right balance between driver control and active safety – especially in performance car applications where a degree of movement might be acceptable or even desirable. Similarly, the system has to judge when it’s safe to hand control back to a driver.
There’s also the question of how much 
assistance to offer. Should the system discreetly prompt drivers’ senses in some way, or should it actually be capable of taking over some aspects of driving or directional control? This comes back to the idea of customer acceptance, and how much control a driver should relinquish, but it can also contribute to potentially-dangerous feedback loops.
Active safety systems are frequently set up to intervene in highly-dynamic scenarios, such as loss of traction during high-speed cornering. These scenarios are likely to prompt a reaction from the human driver (albeit, not necessarily the correct one). As such, it’s vital to ensure that an ADAS intervention won’t cause the driver to over-correct or accidentally oppose justifiable interventions. The only way to test this reliably is with a human driver in situ, and as we’ll discuss below, there are compelling advantages to using a DIL simulator lab in place of real-world testing for the bulk of this development work.
Even in the case of full autonomy, there are factors that simply can’t be evaluated offline. Occupants’ comfort and wellbeing inside a self-driving car may be perceived differently – and may actually result in different physiological responses. There’s ample evidence, for example, that drivers who don’t normally suffer from motion sickness may do so in an autonomously piloted car. Several suggestions have been put forward to mitigate this problem, such as altered suspension characteristics, altered outward visibility and revised control strategies.
Another point to consider is the level of confidence that automated driving functions instil in vehicles’ occupants. As humans, we naturally and instinctively tend detect cues in a vehicle’s behaviour that indicate stability, control and confidence. This could be slowing down properly during the approach to a junction, or it could be something more subtle such as lane positioning for an upcoming corner or hanging back when we sense that another vehicle is about to make a directional change. It’s been suggested that autonomous vehicles could exaggerate such cues to make passengers feel more at ease, or perhaps use more direct forms of messaging to communicate upcoming control intentions. Again, this comes down to a uniquely human sense of security that can only be evaluated by a real person.
In autonomous driving scenarios, virtually all of the points discussed above can also be extended beyond the ego vehicle and applied to other traffic. For instance, will an autonomous vehicle that’s under evaluation communicate clearly enough with other road users to put them at ease? Can it deal with the full range of varied and sometimes illogical behaviours from human drivers and sudden environmental situations that emerge on the road? Is there a danger that a vehicle would struggle to make acceptable headway in a busy urban environment where it might be subject to aggressive driving from other vehicles?
Sometimes the best way to explore these scenarios is to place a human being into direct sensory contact with a virtual test-driving experience that is powered by simulated environments and systems - in other words: use a DIL simulator. This approach, when set up and executed properly, provides a controllable, safe and repeatable engineering laboratory that also happens to have an extremely low carbon footprint since the usual resources are not consumed while testing.
Real-world testing of CAV and ADAS functionalities can be tricky – perhaps more so than evaluating any other aspect of a vehicle. From a financial perspective, this might include the cost of producing prototype vehicles equipped with various sensor arrays. Due to the large number of exploratory possibilities, this can actually limit the number of concepts that can be investigated within a realistic vehicle development timeline. Some of the required hardware and software components may not even be available in the early stages of a project, limiting the scope of physical testing at that stage.
With simulation, this is no longer a problem. The inherently scalable and modular architecture of engineering-class DIL simulators allows items such as sensors and electronics modules to be included where required as Software-in-the-Loop (SIL) models and/or as Hardware-in-the-Loop (HIL) test benches. This “virtual prototyping” approach enables testing with human touch points to begin at a much earlier stage of any given project. First-hand evaluations and assessments carried out at this stage enable engineers to make better-informed decisions on fundamental design parameters that may be difficult and costly to address later in a programme.
Along with the upfront cost reduction associated with eliminating physical prototypes, virtual prototyping mitigates risk to people and equipment. That’s particularly true of active safety systems, where learning algorithms need to be trained using potentially-dangerous scenarios; it is much safer to set up dangerous edge cases in a controlled DIL lab environment than it is to set up real-world tests.
There are, of course, ways to mitigate such risks in real life, but physical prototype testing always introduces its own challenges. Using collapsible targets instead of real vehicles for collision avoidance studies, for instance, raises a number of questions surrounding movement and appearance. For example, does a test target reflect radio waves or sound waves in the same way as a real vehicle if radar or ultrasonic sensors are to be used? Does it move in a realistic manner? Is the shape of the target – potentially a 2D board or a 3D inflatable – accurate enough to be used for a camera system?
In theory, one of the benefits of physical testing is that it can capture some of the variability of real-world use. However, this can be a mixed blessing. In DIL simulation, it’s possible to achieve 100 percent script-ability and repeatability, matching scenarios and conditions for each test, allowing freedom to choose what changes and what does not. In contrast, real world testing presents a constantly-shifting pattern of environmental conditions – weather and time-of-day effects are perhaps the easiest ones to identify. Other, non-ego vehicles may also follow subtly different trajectories in repeated tests – even if drivers and control algorithms are attempting to replicate exactly the same courses.
Certain engineering-class DIL simulators (some, but not all) are built around a scalable and modular architecture – which means that connected hardware and software systems can be, in a sense, interchangeable throughout the model, as are real and virtual (AI) drivers. Complex effects such as fog, water on sensors and lens flare can usually be accurately modelled in software, giving complete control of world-spaces and scenarios. Complex models can be switched with simplified models to test systems and support decision-making.
For instance, the presence of another vehicle or obstacle can be modelled directly or via an electronically-generated image could be fed to a car’s sensor systems for it to carry out its own object identification and classification. Similarly, LiDAR, sonar sensors (and others) can be represented by emulators or real components, in-the-loop with all the other simulation elements. As a typical vehicle development process unfolds, more and more of the software-based emulation can be replaced by real hardware. Connections over networks such as CAN and FlexRay can be used to communicate with the hardware on test, giving a truly authentic environment for electronics modules to operate within.
DIL simulation captures the varied and sometimes unpredictable actions of human beings, but it puts those in the controlled conditions of virtual vehicles and virtual worlds. It allows numerous design iterations to be quantitatively and qualitatively assessed, even in the absence of physical hardware. And perhaps most importantly, it does so without any physical risk to employees, vehicles or bystanders.
Another intriguing question for vehicle manufacturers and service providers looking into self-driving vehicles is related to brand feel. If a vehicle drives itself for some or all of the time, will it still be able to convey a sense of identity that ensures marketplace acceptance?
This is a complex topic that incorporates factors as diverse as exterior styling, ergonomics, user interfaces and even marketplace messaging. But there are also physical, dynamic aspects to consider. For example, since automated driving algorithms collect environmental information from sensor arrays that are used to inform vehicle directional controls (steering, braking and throttle applications), and since directional control gradients can be set anywhere within a vehicle’s performance capability envelope, vehicle engineers are faced with the daunting task of defining rational settings within an extremely large working space. This touches upon everything from establishing cruise control functions in well-structured platoon situations to mapping complex operation modes that involve safely navigating in complex, variable world-space settings.
One possibility for establishing and/or
preserving brand identity is to embrace these complexities with virtual development tools that can collectively serve as a vehicle development sandbox. DIL simulators are one such tool that, as mentioned above, have the unique ability to directly include human interactions.
By definition, the job of a DIL simulator is to convince human participants that they are experiencing certain sensory inputs. This can provide the perfect means for understanding and evaluating these parameters from a vehicle identity perspective.
Not only can today’s engineering-class DIL simulators give a true sense of the dynamic behaviour of vehicles operating in autonomous or assisted modes, but they can also be used to determine logical human-machine handover protocols. For partially-autonomous (SAE Level 3) systems, the moment a driver passes control over to the vehicle is likely to be a psychologically significant one. Even more crucial will be the point where control is returned to a driver.
Here, the vehicle will be responsible for clearly communicating the upcoming handover and ensuring that the driver is ready to take control.
At University College London’s Person-Environment-Activity-Research-Laboratory for autonomous vehicles (PEARL), research into this area has been significant. And PEARL’s DIL simulator lab has been a key part of creating a flexible environment for conducting experiments and research to improve understanding of complex human behavioral interactions with ego vehicles, other vehicles and pedestrians. Professor Bani Anvari, Professor of Intelligent Mobility and Director of IM@UCL, describes it as follows:
We have created a controlled, repeatable DIL simulation environment to explore any number of autonomous handover situations. It’s proving to be an excellent research tool for us.
Mass-implementation of ADAS and CAV technologies is unlike anything encountered by the automotive industry to date. In some instances, the core work is driven by traditional automotive companies, but the trend is definitely shifting toward external dependencies with external providers of sensors, ECUs, software providers and so on. In some cases, innovators may come from a completely different sectors, perhaps with experience in electronics or computing, but no background in automotive engineering per se.
A corollary to this is that there are now more potential stakeholders than ever before, and not all will have access to test vehicles or the appropriate hardware in the early stages of their research. DIL simulators provide a cost-effective research platform to those entering the automotive market with no existing test fleet to call upon. In addition, DIL simulation can serve as a bridge, providing a common sandbox, so to speak, where engineers from widely different disciplines can work toward common goals.
This could be something as simple as the freedom to simulate a more complex sensor array than those currently on the market, or it could be a whole new vehicle concept that doesn’t exist at present. In either case, it allows technology developers to fill capability gaps and press ahead with developments.
