Self-driving car based on deep learning

Generalization: automated driving on a yet unknown complex track (compared to training tracks).
Note: “jumpy” steering reflects toy RC car limitations: it turns 45° to the left/right or drives straight ahead.
(Music: GoNotGently)

After struggling to make a neural net that would predict steering commands reliably for an autonomous toy RC car, only based on the current camera view (no history), I approached the problem systematically in a robot simulator, which allowed for faster experimentation, finally leading to success.

Training examples: manual driving with arrow keys to create a perfect left/right turn.
The purple “Trail” shows the driven path (geometrically clean after several tries).

With only two simple training tracks, one with a 90° left curve and the other one with a 90° right curve, I was able to teach reliable driving behavior. The neural net generalizes better than expected, such that the self-driving car stays on the “road”, even for tracks differing significantly from the training data.

Given more varied examples of successful steering, the driving behavior could become a lot smoother than the video shows. But interestingly, the convolutional neural network (CNN) seems to interpolate nicely between the provided training examples, and is able to handle unknown degrees of road bends.

It even manages to drive through road crossings (see after the break), if a little awkwardly, since crossings “look confusing” and were never trained. When positioned outside of the track facing it at a slight angle, the car also manages to steer in the “hinted” direction and aligns properly with the track!

Continue reading

Building an autonomous robot car

This project is about an autonomous vehicle, based on a modified toy RC car, that can drive along a “road” without any manual interaction required.

To this end, the car’s remote control is modified so it can be attached to a microcontroller, that receives commands from a Python program running on a laptop. The camera, mounted on the top of the car, streams its view wirelessly to a neural net on the laptop, that decides what steering commands are the most appropriate at every time step/frame.

In this post, I will present how to modify the remote control (soldering and mechanical changes), how to extend the car, and how to stream live video, with low latency, from the Raspberry Pi to a laptop using GStreamer and OpenCV. An upcoming post will show a reliable neural net model for automated steering.

Continue reading

Safety and formal proofs in robotics

Formal proofs and verification in robotics are a difficult subject to tackle, due to the unclear nature of the environment and the question of what constitutes a sufficient model to even be able to make valid proofs.

I stumbled upon research regarding safety in robotics, in particular about obstacle avoidance and surgery, which I have yet to read in detail, but seems interesting.

Since the NTSB, a US-government agency investigating transportation accidents, released their final report analyzing the causes of Uber’s autonomous car accident from 2018, the relevance of the search for a Driver’s License Test for Robotic Cars, that uses formal proofs, is highlighted again.

Continue reading

3D scan with a hobby LIDAR on a pan-tilt kit

Since I learned about point clouds a while ago, I wanted to gain some practical experience with them. The goal was to scan a terrain / landscape-like scene, using an as basic as possible hardware setup, not a depth camera.

I used a LIDAR-Lite (a laser based distance sensor), mounted on a pan-tilt kit, driven by servos to position the sensor. All those devices were controlled by an Arduino, which streamed the sensor data back to a PC. To avoid problems with outdoor light, I added some wavy terrain-like features to a room, and scanned it instead of an outdoor scene.

Finally, I wrote my own software to compute each voxel (3D point / pixel in space) from the sensor data and servo angles, such that I could render them as a depth map and a point cloud, with various methods.

Continue reading

Magnetic-core memory on an Arduino shield

Core memory is the simplest kind of main memory you could build from really basic components (no chips!), and easy enough to understand well.

I always wanted to understand computers to their core (no pun intended), and to build one myself from the ground up. The following kit seems to make it quite fun to do that for working memory (RAM), while being physically large scale enough to allow for inspecting and measuring what is going on.

In the picture above, the core memory itself is just the wire net on the left with the small rings (that look like beads/dots). Enjoyably simple.

The chips on the board are there to easily write and read the hand-built memory and to provide an interface to an Arduino (or other microcontroller).

The kit comes with all the necessary components, including the PCB, magnetic cores, wires, etc., while requiring some soldering to assemble it. Head over to Jussi’s blog to read the complete documentation for the shield and how to get one.