Destinations/Photogrammetry with Regard3D

From Valve Developer Community
Jump to: navigation, search
Dead End - Icon.png
This article has no Wikipedia icon links to other VDC articles. Please help improve this article by adding links Wikipedia icon that are relevant to the context within the existing text.
January 2024

Photogrammetry with Regard3D

Regard3D is a free, open source photogrammetry tool available for Windows, Mac and Linux systems. Both commercial and non-commercial use are permitted, and the software is patent free, to the best of its author's knowledge (but please don't interpret that claim as legal advice).

This page is a brief summary of information available in the developer's own tutorial, plus a few extra details to help smooth out some of the wrinkles you may encounter.

Prerequisites

  • A supported operating system
    • 64-bit only
    • Windows: 7 or newer
    • Mac: OS X 10.7 or newer
  • A graphics card with OpenGL support
  • At least 4 GB of RAM, preferably 8 or more
  • A camera that records its make, model, and current focal length in pictures' EXIF info
    • Alternatively, you'll need knowledge of these things (in particular the actual focal length of each photo, not the "35mm equivalent" length) and the patience to edit XML.
  • A set of photos suitable for photogrammetry, in JPEG format

Installation

Install the Regard3D application

Windows/Mac

Download

  • The Windows installer for Regard3D version 0.9.0 requires UAC elevation, but for some reason doesn't install for all users, so if you start the installer from a standard account you won't get start menu shortcuts. The program's executable can be found in Program Files\Regard3D.
  • Mac users have things easiest! Just run the .dmg like any other installer, then find the .app and Control+Click to open the unsigned bundle.

Linux

  • Unfortunately, there are no binaries available for Linux, so you'll need to compile the program, and its executable and library dependencies, from source. The developer's build instructions leave most of the process as an exercise for the reader, so a degree of familiarity with C++ and Cmake will come in handy.
  • [Todo] Figure out complete build/install procedure

Install the sensor database

Download

Place in the following locations, as necessary:

  • Windows
    • C:\Program Files\Regard3D
      • Administrators already have a copy of the database in %LOCALAPPDATA%\Regard3D, but the aforementioned installer bug means standard users don't, and in fact can't since the registry keys dictating its location don't exist.
      • Remember that as a standard user you'll be unable to save changes to this file directly. You'll need to copy it to a writable folder first, make your edits, then copy it back.
  • Mac
    • /Applications/Regard3D.app/Contents/Resources/
      • As is the case for Windows administrators, this file already exists. You may wish to replace it anyway, as it's out of date compared to the openMVG Github version.
  • Linux
    • Until the compile and install process has been more thoroughly documented, just put the database in the same directory as the Regard3D executable.
    • [Todo] Determine intended sensor database location

Add your camera to the database

Regard3D requires certain information about your camera, namely the horizontal size of its sensor. If your camera is in the database already (open the .csv file in a text editor that supports Unix line endings, e.g. Notepad++, and search for it), you're all set!

If instead you can't find your camera, you'll have to add it yourself. You'll need three pieces of information:

  • Camera make
  • Camera model
  • Sensor width in millimeters (be careful not to confuse this with the diagonal measurement)

The first two can be obtained from your camera's EXIF data, which can be accessed by most image viewing and editing programs.

The sensor width is more involved. Search online for your camera's specifications, looking for either a measurement in millimeters or in fractions of an inch. For example, a full frame SLR would be 36x24mm, while an iPhone 5 would be a 1/3.2" type sensor ('type' since the fractional notation is merely historical convention, not a literal measurement). If you can only find the fractional measurement, use this chart to find the approximate dimensions; a 1/3.2" sensor would be 4.54x3.42mm.

Once you have all the info, you can edit sensor_database.csv. Add a new line at the bottom, using the same format as the others of Make;Model;Sensor width.

  • e.g. Apple;iPhone 5;4.54

The interface

Start Regard3D, and note the basic layout. In the center of the screen is the 3D viewport, with a trackball interface. Click and drag to rotate the view, right click and drag to pan, and zoom with either middle click and drag or just the scroll wheel.

In addition to the trackball, Mac users will see "Regard 3D" displayed as 3D text, where Windows and Linux seem to start blank in version 0.9.0. Mac users can therefore familiarize themselves with all viewport navigation, while everyone else will need to generate some results before seeing zoom and pan operations do anything.

The rest of the interface is just as straightforward. To the left, you'll see a tree view on top and a panel on bottom. The tree view will show you a hierarchy of results generated by each step of the photogrammetry process. Each result is retained until you explicitly delete it, at which point all its descendants are also deleted. The panel below not only provides access to further steps by way of buttons, but also shows the details of whatever result you've highlighted in the tree view. Experimenting with settings is thereby that much easier, since all you have to do is click on a result to see a list of the arguments used to generate it, as well as the running time of that step.

On your right you'll see the 3D viewport's display controls, which should be self explanatory. Of particular note are the "Point size" slider and the "Enable lighting" checkbox. When working with point clouds it's often useful to increase the point size, and when you get to reconstructing meshes with texture info it's nice to turn off lighting.

Finally, the toolbar contains icons for New Project and Open Project, and the View|Console output option lets you see the detailed information displayed by the tools as you go through each stage of processing. Keeping an eye on this isn't as useful for those of us without a background in photogrammetry theory, but it can help diagnose problems, so remember that it's there.

Starting a new project

Time to get started!

  1. Set a default project path
    • Regard3D automatically creates a subdirectory for every project you create. The project path is the top level directory in which that subdirectory is created. You'll have the option of setting this per project, but when you create new projects on a regular basis it's helpful to have the path defined already. Go to Options|Properties, and either type in a path or click Browse to choose one.
  1. Create a new project
    • Click the icon with the plus sign, or go to File|New Project. Set the project path if you don't want to use the default, then give your project a name and click OK.
  1. Create an images directory
    • A folder within your project folder will do nicely, but you can put the files wherever you prefer. The goal here is just to have a dedicated location for your project's input files. Make sure they're copies, while you're at it! If your input requires processing, you don't want to be destroying your originals.
  1. Preprocess images
    • Rotate
      • Regard3D 0.9.0 doesn't account for EXIF orientation tags, so if you have images with different orientations you'll need to rotate them and reset those tags before bringing them in to your project.
      • Windows users can try IrfanView for this. Start the program, hit T to open thumbnail view, select your images, then hit Shift+J to open the "lossless transformations" dialog. Choose "Auto rotate" from the Transformation pane, and click Start to automatically rotate your images based on the EXIF orientation tag. Please note that this will overwrite your input files!
    • Scale
      • For testing purposes, it will help to work with low resolution versions of your images, let's say 640 pixels on the long side.
      • Again, IrfanView makes this easy in Windows.
        • Hit B to open the Batch conversion dialog.
        • Find your files in the browser, select them all, and click Add.
        • Choose JPG from the Output format dropdown. Click Options, drag the Save quality slider all the way to 100, make sure "Keep original EXIF data" is checked, and click OK.
        • Check the "Use advanced options" box, then click the Advanced button. Check the RESIZE options, enter 640 in the "Set long side to" field, check "Preserve aspect ratio" and "Use Resample function", along with "Overwrite existing files", then click OK.
        • Make sure to set the output directory. If you've made sure that the files you're working on are already copies, and have backups elsewhere, you can click "Use current ('look in') directory", which in concert with the overwrite option will simply replace the current files with smaller versions.
        • Click Start Batch to process the images.
  1. Import photos
    • With that out of the way, all that's left to do is bring the files in. Highlight your project in the tree view on the left hand side of the Regard3D interface, then click the Add Picture Set button.
    • Click Add files, find your photos, select them all, and hit Open. The list will begin to populate with EXIF info from your input, and you won't be able to leave this screen until it's finished. With a large set of pictures this can take a while, so in the meantime feel free to come up with a descriptive name for this picture set and type it in the entry field at the bottom.
    • When the table has filled out with all the info, click OK to close the dialog and accept the set. If you get an error about focal length or sensor width, see the installation section above for instructions on adding your camera to the database.
    • If for some reason your photos' properties aren't being detected, because either the EXIF info wasn't parsed properly or isn't there in the first place, you can edit the project file yourself if you know the correct values to use. Close the project in Regard3D, then find the .r3d file on disk that corresponds to it. Open it in a text editor, ideally one that offers syntax highlighting for XML, and search for the filename of the photo you want to edit. You'll see XML tags like <cameraMaker>, <cameraModel>, and <focalLength>, among others. Replace the values, repeat for other photos as necessary, then save the file and reopen the project. Open the picture set from the tree view and look at the different fields to see that they've been filled in correctly.

Basic usage

Compute matches

First we need to determine the common elements in the photos you've provided, so highlight your picture set and click Compute matches. The defaults are a good place to start.

  • If later steps reveal that the defaults here didn't produce enough matches, run this step again with finer settings. Try only one or two notches at a time, and increase the sensitivity slider first. Don't touch the matching ratio until sensitivity has hit its maximum.

Triangulate

Next up is camera registration, in other words the process of determining each camera's position and orientation in the scene. Select your match results and click Triangulation.

  • Ideally you're aiming for X/X calibrations, e.g. for 40 total cameras you'll get the best results if all 40 of them can be calibrated. It's not the end of the world if you can't get them all, but the more the better.
  • If you used Incremental triangulation and only some cameras were calibrated, try running it again with a different starting image pair. If that fails, try using Global triangulation.
  • Once you've got a decent number of cameras calibrated, you may wish to increase the viewport's Point size using the slider on the right, since cameras are displayed as points themselves. They're bright green, but still hard to spot when they're small.

Densify

With a simple, sparse point cloud generated, it's time to flesh that out a bit by "densifying" your triangulation result. Highlight the result of the last step and choose "Create dense pointcloud".

  • Regard3D offers two options, CMVS/PMVS and MVE, the details of which are beyond the scope of this document. One may work better than the other for certain source material, so don't be afraid to experiment! As a basic introduction to their usage:
    • CMVS/PMVS
      • Keep in mind that small and/or few photos may benefit from disabling the CMVS option, where large/many photos will want it enabled for the sake of memory usage.
      • Try increasing the Threshold if you find you have more outlying points than you'd like.
    • MVE
      • The sliders work in opposite directions; to increase the quality of your results, you'll want to decrease Scale but increase Filter width. Be careful to go only one step at a time! These sliders have a dramatic effect on computation time.

Reconstruct

Now that your sparse point cloud has been densified, you can generate the actual mesh for your scene. This step is optional for those who plan to do their surface reconstruction in other software. Everyone else can simply select their Densification, click Create Surface, and choose their options.

  • If using CMVS/PMVS, Poisson reconstruction is the only option. Both are available if you used MVE, but in that case the developer recommends using FSSR.
  • Stick with vertex coloring at first. Tweak your settings until you have a satisfactory mesh (the fewest holes and outliers), then run reconstruction again with the Texture option selected.
  • When generating your texture in Regard3D 0.9.0, you'll probably want Global Seam Leveling disabled, as there's apparently a bug in texrecon that produces lower quality results with it enabled.

Export

At this point you've done all you can do in Regard3D itself, and for the sake of cleaning up imperfections in your mesh, closing any remaining holes, and retopologizing to reduce the polygon count, you'll need to get your results out to another application. You have two options here:

  1. Export the densified point cloud directly
    • Users who skipped Surface generation planning to export their point cloud can simply highlight the appropriate Densification and use "Export point cloud" to do exactly what the name suggests, or "Export scene to MeshLab" to get a MeshLab project complete with geometry, photos, and camera registrations.
  1. Export the generated surface
    • As for the rest of us, we're finished with Regard3D when using version 0.9.0. There's an issue with that release which prevents the "Export surface" button from working properly, and the exported .obj file is malformed in some way. Luckily there's a workaround! Open your project's directory on disk and navigate to the surface reconstruction result, for example:
      • pictureset_0/matching_0/triangulation_0/densification_12/surface_2
    • Replace the numbers as is appropriate for your project. Look in that directory for model_surface_text.obj, model_surface_text.mtl, and any similarly named .png files. These will only exist once you've generated a surface with textures. Copy them to whatever location you find convenient, then load them in your mesh editing software of choice.

And that's it! Once you've refined your mesh with your favorite software and exported it as an .fbx or .obj file, the process picks up here.

You can play around with all of this to get a better feel for the software, replace the downscaled copies of your input photos with full size originals for doing your high quality final work, and check the links below to study the topic in more depth.

Regard3D has its rough edges, and may lack some of the creature comforts of commercial software, but it's certainly capable of getting the job done with a good dataset and a little tweaking. Hopefully this introduction was enough to get you started. Happy reconstruction!

See also

To learn more about photogrammetry, or the particular approach used by Regard3D, review the following:

General knowledge

Regard3D software and algorithms

Wikipedia - Letter.png
This article has not been added to any content Wikipedia icon categories. Please help out by Wikipedia icon adding categories.
January 2024
Retrieved from "https://developer.valvesoftware.com/w/index.php?title=Destinations/Photogrammetry_with_Regard3D&oldid=378979"