Expectations for future driving technologies have recently been growing in the mobility domain; one of these is in the practical use of drive recorders for remote surveillance and another is that concerning remote driving systems. However, as vehicles keep on moving, the communication bandwidths of mobile networks and the amount of transmitted camera video data may change in a complex manner. This issue causes the videos to be disturbed when the real-time surveillance of videos from multiple vehicle-mounted cameras are processed. In order to solve this issue, NEC Corporation has developed the communication prediction/control technology that integrates the communication prediction and the communication control technologies. Communication prediction technology predicts changes in the communication bandwidths, and communication control technology detects important communications from multiple cameras and automatically optimizes the communication bandwidths using a rule-based AI. The present paper introduces a remote surveillance solution that enables real-time surveillance of videos from multiple vehicle-mounted cameras using the developed technology.

1. Introduction

Expectations for the practical use of drive recorders and remote driving systems are recently tending to be on the rise in the mobility domain. The drive recorder is used actively as a driving guidance tool for drivers by using records of accidents and near-miss events that auto transportation companies are encountering during every day driving. However, nowadays there are many hazardous driving events that are hard to identify from the recorded data or are hard to recognize using the records from the front and driver’s seat cameras alone. As more vehicles are equipped with the communicating - type drive recorders that transmit video data to a cloud computing system during driving, it is anticipated that guidance will be given to drivers at optimum timings by utilizing real-time driving video data. This new technology may thereby result in a decrease in the number of traffic accidents.

A future need of the real-time driving video data via cloud computing is for availability of a mobility service with an unmanned autonomous vehicle in certain areas. This would be expected to offer a solution for the issue of people affected by disadvantaged mobility in sparsely populated areas. With this service, a remote driving system allows a teleoperator who operates a vehicle in a remote location to check the videos of cameras mounted inside and outside the vehicle, with a delay time of less than about 0.3 seconds when observing the passengers. If there is any deviation from the threshold (ODD: Operational Design Domain) programmed by the system, the teleoperators can control the vehicle from the remote location. This service is planned to be put into practical use in Japan by 2020.

The need for real-time checking of the driving video data of multiple vehicle-mounted cameras from a remote location is increasing as described above. Nevertheless, wireless communications pose the issue of the impossibility of real-time video transmission due to the variations in the number of users who use the same frequency and of the radio wave conditions changing as the vehicles travel. To transmit videos in real time in an environment accompanied with variations of the wireless communications bands, NEC is tackling the communication prediction/control technology using AI. The present paper introduces a remote surveillance solution enabling the real-time surveillance of multiple vehicle-mounted camera videos using the communication prediction/control technology. Additional relevant details of the technology are also explored.

2. Remote Surveillance Solution

Fig. 1 depicts the outline of the remote surveillance solution in a mobility service using an unmanned autonomous vehicle. To allow the teleoperator to check the conditions inside and outside the vehicle just like ordinary bus drivers, the unmanned autonomous bus carries multiple vehicle-mounted cameras. Videos from the vehicle-mounted cameras are transmitted in real time from the vehicle-mounted communication device called the connected gateway to the multi-viewer via the LTE network, so that the teleoperator can monitor the driving situations. In the case of an issue with the vehicle driving, the teleoperator controls the bus by sending the control commands of the steering wheel, brake and gas pedals. Communication with passengers can also be achieved via the connected gateway. It is with these mechanisms that the unmanned autonomous mobility service is implemented.

To disseminate the mobility service with an unmanned autonomous vehicle as a solution for issues of people with disadvantaged mobility in sparsely populated areas, the use of the existing LTE network will be essential. However, since the wireless communication bands of the LTE network vary widely, smooth real-time transmission of vehicle-mounted camera videos is difficult. When cameras are mounted at the front, rear and both sides of a vehicle and real-time transmission of their videos via the LTE network is attempted, the videos are often disturbed as shown in Photo 1. Previous systems achieved smooth transmission by inserting delays of a few seconds between the video transmitter and receiver sides to detect the communication band variations and lower their video quality accordingly. However this method causes video disturbance when the delay time is limited to less than about 0.3 seconds. In addition, using multiple vehicle-mounted cameras causes further video disturbance due to the competitive scramble for the communication bands by the cameras. If the disturbance issue was to be solved using the previous method, it was necessary to prepare a dedicated LTE network that did not produce communication band variations.

On the other hand, NEC’s remote surveillance solution is capable of smooth video transmission without disturbance as shown in Photo 2. Section 3 will introduce the communication prediction/control technology that enables a remote surveillance solution.

3. Communication Prediction/Control Technology

The present section describes the communication prediction/control technology that is required for the remote surveillance solution by transmitting the videos from multiple vehicle-mounted cameras to the surveillance center smoothly in real time.

As shown in Fig. 2, the communication for remote surveillance is required to deal with the following two kinds of variations. One is the variation of the communication band of mobile networks. This is a complex variation that depends on the vehicle movement speed and communication congestion status. The second is the variation in the required amount of transmitted data of the vehicle-mounted cameras surveying the front, rear, and both sides of the vehicle. This also varies in a complex manner depending on the vehicle drive conditions including the environment, velocity and travel direction. These two variations produce packet losses and communication delays, and the resulting disturbance of the videos transmitted from multiple cameras makes it difficult to achieve smooth real-time remote surveillance.

To solve this issue NEC has developed an AI-based communication prediction/control system as shown in Fig. 3. This technology consists of a communication prediction method that predicts variations in the communication band and the communication control that identifies important communications from multiple cameras using a rule-based AI and automatically optimizes the transmitted data amount and the video quality.

During the video delivery, the communication prediction builds a prediction model of the communication throughput variations in the last one minute in real time and enables prediction of the probable variation width of the communication throughput until 10 seconds later. The research into this technology has achieved results as a core technology of media communication¹⁾. The communication control uses a rule-based AI to automatically optimize the amount of transmitted data from each camera and a video compression rate according to the conditions, including the vehicle drive conditions. The video quality and communication band prediction is thereby varied. The research into this technology has achieved results as a core technology of optimization of network communication resources²⁾³⁾, and it is being improved for application in the connected gateway. This AI-based communication prediction/control technology is unique to NEC and makes possible smooth real-time remote surveillance and control in various driving conditions.

4. Demo System

In FY 2018, NEC built a remote driving demonstration system using a large wireless-controlled vehicle as shown in Fig 4. The remote-controlled vehicle carries four cameras at the same height as the eyes of a driver in the driver’s seat so that the videos from the vehicle-mounted cameras could be checked with a sense of real vehicular scale.

Evaluations were made using this demonstration system. The remote-controlled vehicle and virtual cockpit are compatible with the LTE communications so remote surveillance and remote driving from a distance of more than 10 km was verified. It was confirmed that the vehicle could move at 40 km/h on the test course in the NEC premises.

5. Conclusion

To promote an improvement of the skill of teleoperators and to disseminate a mobility service with an unmanned autonomous vehicle in the mobility domain, the use of real-time videos from vehicle-mounted cameras promises much. To meet such a need, however, it is essential to transmit vehicle-mounted camera videos smoothly in real time under the existing LTE network environment. By using the remote surveillance solution making use of the communication prediction/control technology, NEC will implement real-time vehicle camera video transmission for the existing LTE network, thereby contributing to the development of the mobility society.

References

Authors’ Profiles