Tracks WIP - Santa Monica Mountains

[Todd K]

This is an experiment. Over the past year or so I've been testing various methods of using photogrammetry to generate roads. My goal is to develop a process suitable for recreating any road, of any distance, with as minimal effort possible. I'm definitely still a looong way off from that, but I am making clear progress. Now that I've landed on a technique that I feel is repeatable enough to begin refining upon, I'm tackling a large scale project to further streamline as I go. The primary test subject for this is something very dear to my heart, which is the network of canyon roads in the Santa Monica Mountains up above Malibu, CA.

I have about 15 miles built and in game already, with about 40 being the ultimate goal. I will release this as a WIP beta once I reach some short term milestones that I'm working on now.

Here's a short video of just one leg, Las Flores Canyon Rd in it's current state.

For the photogrammetry, my method is to mount a GoPro MAX on the end of an 8' pole that I stick through my car's sunroof. The camera shoots 360 degree spherical latlong video with GPS coordinates on each frame as I drive along at about 30 mph. To avoid sun shadows, I have to shoot either at dawn or dusk when the sun is behind the mountains, or wait for the ultra rare overcast Sunday. Here's a full res frame from the video.

A lot of pre-processing is necessary to conform the imagery and GPS data into a format suitable for photogrammety, but once that's done Agisoft Metashape does a pretty great job of generating accurate meshes. After another lengthy process of cleaning up the scan geo, optimizing the meshes and UVs, baking down textures, and smoothing the road surface it can then finally be brought into AC. I'm going a step further though and removing the painted road lines from the textures and laying down my own for better quality. I'm also generating some masks to use in the AC shader for specularity and breakup maps.

Above is a general map of the canyon roads I'm currently working on for an initial release. The green ones have already been generated and are already in-game in some capacity. The yellow ones have been recorded and are currently running through the photogrammetry process. Red ones are some additional connecting canyons I think would be great to add, but have not yet recorded the 360 footage for them and probably won't until after I finish what I have and release it to you.

Las Flores, Stunt, Piuma, and Tuna will make up the core legs of the map. After that the next step will be to create connecting roads between the tips of each of the leg, but for that I'm going to be deviating from reality a little. For example, the northern connecting road will still be Mulholland Hwy, but I'll use the more fun section further to the west and transpose it to connect the bottom of Stunt and Piuma. For the southern connecting route that would be a fairly boring section of Pacific Coast Highway, I'll instead be stealing a more interesting section of the road from much further north up the coast. I also really want to include a short stretch of Old Topanga Canyon Road, as it's another one of my local favorites. By the way, if you aren't familiar with the area, the very center of the map at the intersection of Schueren/Stunt/Saddle Peak is the highest point at 2300 feet, with the outer tip of each leg being closer to sea level or just above.

So what's my ultimate plan? Not really sure haha. At this point the #1 thing I'm looking forward to is getting to drive Tuna Canyon uphill lol! (It's a one-way downill street in real life) But here's my to-do list for the immediate time being:

01) Finish processing Saddle Peak and Tuna Canyon to complete the 4 core legs (the 4 core legs being Tuna, Stunt, Las Flores, and Piuma)
02) Get all road surfaces up to the same level (or better) as seen in the video above.
03) Place basic props such as guard rails, signs, and light poles.
04) Spawn some very basic foliage to mask some of the edges of the photogrammetry geo
05) Release as beta.
06) Create connecting roads, and refine.
07) Optimize (if this is even possible. the file sizes are already gigantic)

Anyway, like I said in the beginning, I want to stress that this really just an experiment. I will definitely be releasing this once I'm happy enough with the 4 core legs I'm working on now, but expect it to live as a beta release for quite some time until I can properly clean all the photogrammetry and optimize everything. I'll definitely do a pass of smoothing out a lot of the biggest offenders, but my plan/hope is to hide most of the nasty stuff with trees, bushes, and really large GrassFX

My normal inclination would be to just wait for an actual release to make a post, but I'm keen on getting the opinions, feedback, and tips from anyone interested in a project like this. Let me know what you think!

-Todd

P.S. If you followed and enjoyed my previous AC track release, Topanga Canyon, just a heads up that I am on the verge of releasing another very similar track, Mulholland Drive. Keep an eye out in the coming week or two! (it's not utilizing this photogrammetry technique)

 

[isamu]

This looks very promising! Good luck and keep us posted

 

[NightEye87]

I wonder what the performance will do in-game, especially texture-wise. Photogrammetry can do weird things with textures, having huge atlasses of pieces. Not a problem at all, but if it makes a new texture for each scan, things might get heavy.

The idea is awesome and I'd love to see more

 

[Moorea]

Very informative thread. Thank you.

 

[Salvatore Sirignano]

A lot of pre-processing is necessary to conform the imagery and GPS data into a format suitable for photogrammety, but once that's done Agisoft Metashape does a pretty great job of generating accurate meshes.

This looks amazing! Could you elaborate on the pre-processing process? I imagine that instead of metashape having to calculate the camera positions from each shot, if that position is known with GPS data, it makes the mesh generation easier?

 

[Todd K]

I wonder what the performance will do in-game, especially texture-wise. Photogrammetry can do weird things with textures, having huge atlasses of pieces. Not a problem at all, but if it makes a new texture for each scan, things might get heavy.

The idea is awesome and I'd love to see more

Yeah this will be one of the biggest hurdles to being able to offer a playable track to the public. I'm doing my absolute best to consolidate as much as possible, but yeah they are already getting heavy. For the 2.5 mile LasFlores stretch in my video above, there are 20x 4k DX5 textures at 16mb each, plus a 1k 1mb mask for each, for a total of 340mb (not including the background terrain, which is separate).

For one of the other roads of similar length, I tried going with 9x 8k textures at 65mb each (+1mb for the mask), which works out to 585mb.

In general, which approach seems better to go with? 20x4k, or 9x8k? I've read here before that fewer larger maps are better than lots of smaller ones, but does that still hold true for maps THIS large?

In the end, as a last ditch effort, I am prepared to create a version with scaled down texture maps if the file size becomes too intense.

Here's sample of how the UVs are laid out, but I had to scale this down to 2k to get it to upload here.

 

[Ryno917]

Looks amazing already. Looking forward to seeing how this continues to develop

 

[Johnr777]

Yeah this will be one of the biggest hurdles to being able to offer a playable track to the public. I'm doing my absolute best to consolidate as much as possible, but yeah they are already getting heavy. For the 2.5 mile LasFlores stretch in my video above, there are 20x 4k DX5 textures at 16mb each, plus a 1k 1mb mask for each, for a total of 340mb (not including the background terrain, which is separate).

For one of the other roads of similar length, I tried going with 9x 8k textures at 65mb each (+1mb for the mask), which works out to 585mb.

In general, which approach seems better to go with? 20x4k, or 9x8k? I've read here before that fewer larger maps are better than lots of smaller ones, but does that still hold true for maps THIS large?

In the end, as a last ditch effort, I am prepared to create a version with scaled down texture maps if the file size becomes too intense.

Here's sample of how the UVs are laid out, but I had to scale this down to 2k to get it to upload here.
View attachment 469867

Really cool project!
Speaking from my own experiences, don’t go 8K. Even MID range GPUs still have a hard time pushing that through the pipeline smoothly

 

[Amir Vodokotlic]

Very cool.

Do you mind sharing how long Metashape takes to process high density point cloud and mesh?

 

[NightEye87]

Yeah this will be one of the biggest hurdles to being able to offer a playable track to the public. I'm doing my absolute best to consolidate as much as possible, but yeah they are already getting heavy. For the 2.5 mile LasFlores stretch in my video above, there are 20x 4k DX5 textures at 16mb each, plus a 1k 1mb mask for each, for a total of 340mb (not including the background terrain, which is separate).

For one of the other roads of similar length, I tried going with 9x 8k textures at 65mb each (+1mb for the mask), which works out to 585mb.

In general, which approach seems better to go with? 20x4k, or 9x8k? I've read here before that fewer larger maps are better than lots of smaller ones, but does that still hold true for maps THIS large?

In the end, as a last ditch effort, I am prepared to create a version with scaled down texture maps if the file size becomes too intense.

Here's sample of how the UVs are laid out, but I had to scale this down to 2k to get it to upload here.
View attachment 469867

Really cool project!
Speaking from my own experiences, don’t go 8K. Even MID range GPUs still have a hard time pushing that through the pipeline smoothly

Yep, 4K is probably the maximum you'd want to go.
Question is if you really need that big diffuse textures at all. You might get away with (the above image at) 1024×1024 as long as your detail textures are applied properly.

And yes, less textures usually means better performance. But with stages this big you might be able to LOD a lot of them to prevent drawing them all at the same time. Or have LODs take one 4K diffuse where non-LOD takes up 4. That requires a bit of UV work (scale 50%, move 50%, times 4, not much really) but might be worth it.

Have fun experimenting

 

[GTAce]

Looks amazing, love your approach!

 

[y0chang]

Yeah this will be one of the biggest hurdles to being able to offer a playable track to the public. I'm doing my absolute best to consolidate as much as possible, but yeah they are already getting heavy. For the 2.5 mile LasFlores stretch in my video above, there are 20x 4k DX5 textures at 16mb each, plus a 1k 1mb mask for each, for a total of 340mb (not including the background terrain, which is separate).

For one of the other roads of similar length, I tried going with 9x 8k textures at 65mb each (+1mb for the mask), which works out to 585mb.

In general, which approach seems better to go with? 20x4k, or 9x8k? I've read here before that fewer larger maps are better than lots of smaller ones, but does that still hold true for maps THIS large?

In the end, as a last ditch effort, I am prepared to create a version with scaled down texture maps if the file size becomes too intense.

Here's sample of how the UVs are laid out, but I had to scale this down to 2k to get it to upload here.
View attachment 469867

Amazing work creating a 3d map from gopro footage. One solution on the terrain diffuse is to use the photometry only for geometry, then regular satellite mapping for most areas and maybe a small amount of more detailed mapping in others. The issue with satellite photometry terran in mountainous areas is the lack of local detail in geometry due to shadows and obstructions.

 

[EttieApple]

I would also use the photogrammetry for reference only, but when accurate this reference would be amazing to work with.
You can probably also make excellent textures from it. I am using screenshots from street view as basis for rock faces, and that already works reasonably. Using 4 different images and repeating them randomly works well, more images are not necessary for me. I assume that you are ending up with every surface with an unique texture, which is absolutely overkill in my opinion

The video already looks amazing I must admit that

 

[Todd K]

This looks amazing! Could you elaborate on the pre-processing process? I imagine that instead of metashape having to calculate the camera positions from each shot, if that position is known with GPS data, it makes the mesh generation easier?

I seemed to find that GPS alleviates a lot of random alignment errors and speeds up the process, as it allows each image to be compared with it's physically closest neighbors (per my understanding of the documentation). Very very rarely does it fail to align a photo, and when it does it's always just due to bad masking (read below). It also seems to help the long string of photos stay on target, which is evident when the GPS data is wrong. Occasionally the GoPro will lose it's signal briefly, resulting in bad coordinates. If I let it align with these, it will indeed try to use those positions for alignment, so I have no choice but to delete the data from those frames.

Admittedly I haven't necessarily done super exhaustive comparisons using the same exact images both with and without it, but once I began achieving near flawless results WITH it, I never looked back.

Here's a quick rundown on what I do to get from the GoPro to photogrammetry.

01) The first step is to get the video and data off the camera. The GoPro MAX records to a proprietary video format, so I first have to convert it to JPG frames which are 5760x2880 latlong images.

02) I use GoPro Telemetry Extractor tool available online to pull the GPS data from the video which gets written out as a text log, one line of text per video frame.

03) I then load the JPG image sequence into Nuke where I run it through a script to add some sharpening to the RGB, and generate an alpha mask for each frame which is critical for a smooth alignment process. I have to mask out my car and the sky from the image so that the algorithm only focuses on trying to align the actual environment. It's probably more important than the GPS is, honestly. When I render out both the RGB and mask files, I only render every third frame which cuts the image count down to about 4,600 in the case of Las Flores, which works out to about 1800 images per mile of road.

04) I wrote a python script that edits the GPS log into a format that can be read by ExifTool, a free program for managing meta data on various types of files. Running another script through ExifTool then injects the GPS coordinates from the log file to the corresponding image as metadata. Agisoft will then be able to read the GPS metadata on each image when it's loaded in.

05) From there I load them into Agisoft as chunks of about 700 images which I guess is about 1/3rd mile, so about 7 chunks for my 2.5 mile stretch of Las Flores. After then attaching the corresponding mask file for each image and making sure to set the Camera Calibration method to Spherical, it's just off to the races. I let them run at night as a batch process.

I suppose it sounds like a lot of work when all written out, but it really just ends up being a series of mostly automated processes. It's a breeze compared to the post-photogrammetry cleanup which involves a lot more manual labor!

Very cool.

Do you mind sharing how long Metashape takes to process high density point cloud and mesh?

Next time I run one I will write down all the exact numbers, but for 1/3 mile long chunks, it takes 3-4 hours to do the alignment, and then another 3-4 hours to generate the high density point cloud. Mesh generation is much quicker.

 

[Ben O'Bro]

Awesome work !
Great to see another LA Canyons on the making
and thanks a lot for your time detailing the whole process

photogrammetry is unbeatable with the continuous uniqueness of the sides of the road

 

[Sincity]

Love your Topanga Canyon track.

 

[seedmole]

Long-time resident of Topanga Canyon here, big fan of the TC Hillclimb track, excited for this and Mulholland, keep it up!

 

[Todd K]

I have a small update to share. I recently learnt that The Snake on Mulholland Hwy is open to pedestrian and bicycle traffic, which I had not previously realized (it has been closed to vehicle traffic for the last ~3 years, so I had not been able to record it using my usual method). Today I strapped the camera to my head and rode down on my bike, capturing what I need for photogrammetry. All the GPS data came through and the camera exposure looks great, so I think this will be an easy one to compile.

I will be incorporating The Snake (and other portions of Mulholland Hwy) into this freeroam map, and will also likely be releasing it as it's own track as a separate download.

 

[Todd K]

I've just released a short section as alpha to gather some initial feedback on this track modding approach. What I learn from this will help inform how I move forward on this larger network of roads. Check it out and give it a go!

The Snake

This is an ALPHA release to test a new road construction method I've been working on. The road surface and ground textures here were generated using photogrammetry. The elevation change, road camber angle, and variable shoulder detail is all...

www.racedepartment.com

 

[Kimmo Kokkonen]

I've just released a short section as alpha to gather some initial feedback on this track modding approach. What I learn from this will help inform how I move forward on this larger network of roads. Check it out and give it a go!

The Snake

This is an ALPHA release to test a new road construction method I've been working on. The road surface and ground textures here were generated using photogrammetry. The elevation change, road camber angle, and variable shoulder detail is all...

www.racedepartment.com

Thanks! What comes to Snake so what about Rattle Snake Point (RSP), cannot find any version for AC. That's one of the best endurance tracks ever. Is there AC RSP somewhere available?