Digital Transformation―or DX―has begun to have a significant impact on the mobility market, boosting connectivity, enhancing the sophistication and reliability of safe driving support systems, and rapidly bringing autonomous vehicles closer to real world operability, and much more. Amid this current trend, mobility service providers are faced with the challenge of using video-based driver assistance and appropriate measures to correct anomalies to ensure the safe and secure transportation of cars, trucks, buses and other vehicles.

In this paper, we introduce the Vehicle Interior/Exterior Monitoring Solution that combines NEC’s real-time on-board camera video streaming technology and AI-based video analytics to enable remote and advanced driving support to ensure the safe and secure transportation of people and goods.

1. Introduction

Fig. 1. shows a conceptual diagram of the Vehicle Interior/Exterior Monitoring Solution. In this solution, multiple cameras are installed in a vehicle to provide full coverage of the interior and the external surroundings. These cameras transmit video data via a mobile communication network to enable a surveillance operator at a remote monitoring center to experience the same vehicle conditions as the driver. Using AI-based video analytics technology to detect various problems that could occur while the vehicle is in motion, the surveillance operator can communicate directly with the driver and passengers should any problems arise.

One of the difficulties of remote monitoring is the need to pay attention to multiple video from multiple surveillance cameras installed in each vehicle. Moreover, the surveillance operator must monitor multiple vehicles simultaneously, exacerbating operational stress. NEC’s AI-based video analytics technology takes the load off the remote operator by immediately detecting any anomalies in the vehicle’s external environment and any danger to drivers and passengers inside the vehicle and drawing the attention of the remote operator to specific displays that need to be more closely monitored.

Conventional vehicle monitoring systems that use AI-based video analytics technology normally need to incorporate advanced AI chips in an on-board device, which can significantly increase the vehicle’s power consumption, as well as extending the lead time before the device can be introduced into the vehicle. On-board devices also are subject to processing constraints due to the performance limitations of on-board hardware and the difficulty of adding or modifying services despite rapid technological progress of the hardware itself. We have addressed these issues by implementing the video analytics technology in the cloud. This has the added benefit of accumulating video data in the cloud, which facilitates collaboration with other systems, and precipitates continuous improvement of services.

2. Component Technologies

Next, let’s take a look at the three proprietary NEC technologies behind our Vehicle Interior/Exterior Monitoring Solution.

2.1 Adaptive network control technology for real-time on-board camera video streaming

For the real-time remote monitoring of videos captured by multiple cameras in a vehicle, there is a problem of video quality degradation. This is because the bandwidth of mobile networks and the volume of video data fluctuate on moving vehicles (Fig. 2). To address this, NEC has developed adaptive network control technology which integrates communication quality prediction method that predicts changes in communication bandwidths and communication control method that automatically optimizes distribution of communication bandwidths by prioritizing incoming video streams from multiple cameras (Fig. 3).

The communication quality prediction method predicts future probability distribution of bandwidth in real time based on the past communication bandwidth data. It makes possible high-video-quality, disturbance-free video delivery by automatically controlling the compression rate of video based on the prediction result¹⁾.

The communication control method automatically optimizes the amount of data transmitted from each camera and the video compression rate in accordance with various vehicle-related conditions such as vehicle status, and constantly fluctuating video quality, as well as with the predicted communication speed. We are now engaged in R&D aimed at making this a core technology to automatically optimize network communication resources. We are also improving it to implement it to communication equipment. Thanks to this NEC-original communication technology, smooth, real-time remote surveillance and control are now feasible under a wide range of driving scenarios. With a view to further improving the efficiency of transmitted data and video quality, we have also now begun to work on improving clarification of region-of-interests (ROIs) in the video when video data is transmitted²⁾. By using AI to automatically control the position and quality of the ROIs, we can reduce the volume of transmitted data and improve video quality.

Case Study: Demonstration Test of Real-Time Remote Monitoring in a Self-Driving Bus on Public Roads

As part of our joint research project with Gunma University, we conducted a demonstration test of a self-driving bus on public roads between the university’s Aramaki Campus and the JR East Shibukawa Station on December 11, 19, and 24 (Photo 1)³⁾.

In this test, we transmitted video from twelve cameras installed in the self-driving bus to the monitoring server at the Center for Research on Adoption of NextGen Transportation Systems (CRANTS) of Gunma University using three LTE lines. We confirmed that in the demonstration test route there was significant bandwidth fluctuation between about 400 Kbps and 4 Mbps in the available bandwidth of upstream communication of one LTE line.

When video is transmitted over communication lines subject to significant bandwidth fluctuation using the existing system, the received video may be subject to delay increases and deteriorations that make it all but impossible to effectively monitor the bus from a remote center. By prioritizing video from front and rear cameras, we succeeded in transmitting all video from all cameras to the control center in real time keeping the video quality of the important cameras high. Photo 2 shows examples of surveillance screens of a self-driving bus traveling the section when the available bandwidth was the lowest in the route. NEC’s adaptive network control technology was incorporated for the first time in a demonstration test conducted on public roads to support remote monitoring, successfully transmitting high-quality, stable video while predicting communication delays and fluctuations of communication even in a rapidly changing and congested communication environment.

2.2 Pose estimation technology to detect anything unusual with the driver or passengers

To detect abnormalities in the driver or passengers, our pose estimation technology extracts people’s postures (skeletal structures) from video data. A diagram of how this works is shown in Fig. 4.

NEC’s pose estimation technology has been designed to work reliably with surveillance and crime prevention cameras and, for this reason, it can effectively extract people’s postures even when video resolution is low or in crowded situations where capturing the whole crowd on video results in each individual becoming very small.^{4) 5)}. Conventional pose estimation technology, on the other hand, is based on the assumption that individuals can be imaged at a larger scale as in snapshots and that people do not overlap too much. These limitations can lead to errors in pose estimation and confusing one person’s joint points with those of another person. We are confident that NEC’s pose estimation technology will be able to achieve high detection accuracy even when video data resolution has been compressed for streaming via a mobile communication network.

Utilization of NEC’s pose estimation technology will make possible the following applications, which will contribute to the reduction of traffic accidents and other incidents.

2.3 Scene recognition technology to detect potentially dangerous situations outside the vehicle

Our incident recognition technology is a useful tool to detect potentially dangerous situations outside the vehicle. It recognizes traffic conditions based on the information about traffic participants and road configurations derived from video data. A conceptual diagram showing how our scene recognition technology works is shown in Fig. 5.

Using video data transmitted via mobile communication networks, NEC’s scene recognition technology performs parallel processing to estimate locations and distances of vehicles on the road, as well as semantic segmentation of roads and lanes. Drawing on these results, this system executes a top-view conversion to show a bird’s eye view of traffic conditions⁶⁾. This makes it possible to discern traffic incidents and automatically detect potentially dangerous situations as listed below.

3. Conclusion

By applying NEC’s Vehicle Interior/Exterior Monitoring Solution to vehicles, such as cars, buses, and trucks, NEC aims to create various services that can detect anomalies inside and outside vehicles and enable prompt response in the event of an emergency such as a traffic accident or passenger falling down inside a bus. Through implementation of the Vehicle Interior/Exterior Monitoring Solution, we look forward to supporting the safe and secure transportation of people and goods.

Reference

Authors’ Profiles