Step 4: Build a PhotoMesh Project

This step outlines the workflow for building a PhotoMesh project. The option to run the entire PhotoMesh production with only cloud resources is the main workflow discussed above, and then in "Building a PhotoMesh Project Using the Master Instance" below. It is also possible to apply a hybrid cloud architecture that mixes cloud and on-premises computing power and storage solutions. This option is covered in "Creating an IAM User for Running PhotoMesh Project Using an On-Premises Master" and "Building a PhotoMesh Project Using an On-Premises Master and Storage".

There are a number of variables involved in the selection of fuser instances to process a build:

§  CPU/GPU – The hardware of the host computer used for your instance determines its compute, memory, storage, and graphic capabilities. GPU-based instances provide access to NVIDIA GPUs to accelerate processing. The need for GPUs is generally determined based on build step. See the table below.

§  Price/Performance – Optimized balance between machine price and its computing power. See here for more information: https://aws.amazon.com/ec2/pricing/. See the table below for Skyline recommendations (based on our experience in the US East – N. Virginia region).

§  Availability Zones-Subnets – The availability of the required number of instances and their price should be considered when selecting an availability zone. Since you are also charged for data transfer from one availability zone to another, if you require many fusers, it is recommended to launch the master in an availability zone with good spot availability to avoid significant cross-availability zone charges. A subnet must be created for each availability zone. For more information, consult the Amazon EC2 console as described below.

§  onDemand/ Spot – onDemand instances provide guaranteed compute resources, while spot instances are dependent on the availability of unused instances: spot instances will only be launched if an instance is available and instance use may be interrupted unexpectedly if capacity is no longer available. onDemand instances also have a static price, while spot instance price varies based on demand. For more information see Amazon EC2 Using Spot Instances in Amazon’s AWS documentation.

Analyzing the price history of spot instances in Amazon EC2 console's Pricing History can help you determine availability of different spot instance types in different availability zones. Price is generally determined by the law of supply and demand, so if the price for a particular instance type is low in a specific zone, this usually indicates higher availability of spot instances of that type in the zone. No price is listed for a particular zone if there are no spot instances of that specific machine type in this zone.

See the chart below for our recommendations for each PhotoMesh step.

Recommendations Based on PhotoMesh Step

PhotoMesh Step	I/O (Disk IOPS and Network bandwidth)	CPU	GPU	Recommended # of fusers / 1Gbps	Recommended Machine Types	Preferred Method for Hybrid
Data Preparation	Intensive	Intensive	Low	5	M5dn.4xlarge M5d.4xlarge M5.4xlarge	Local
AT	Intensive	Intensive	No	20	M5dn.4xlarge M5d.4xlarge M5.4xlarge	Local
Point Cloud	Low	High	No	200	M6i.2xlarge M5.2xlarge M5dn.2xlarge	AWS
Model	Low	High	No	200	M6i.2xlarge M5.2xlarge M5dn.2xlarge	AWS
Texture Mesh	High	Low	High	50	G4ad.4xlarge G4dn.4xlarge G3.4xlarge G4dn.2xlarge	AWS
Output	High	Low	High	20	G4ad.4xlarge G4dn.4xlarge G3.4xlarge G4dn.2xlarge	Local

 

Building a PhotoMesh Project Using the Master Instance

To use the configured environment to produce PhotoMesh projects, follow the steps below:

1.     Connect to the PM Master instance.

Note:        If the PM Master instance is not already running, choose Actions, select Instance State, and then choose Start.

2.     Create the directory: "D:\PhotoMesh\AWSConfigurations", and then copy the JSON files ("DataPrep_AT.json", "Reconstruction_CPU.json", and "Reconstruction_GPU.json") from the zip file downloaded in step 6b of "Customizing the (Fuser) Instance" to this directory.

3.     Start PhotoMesh and create a project.

Note:        The project must be created in a location that is accessible to both master and fuser instances. It is recommended to organize all projects and sources in the \\10.0.0.10\d\PMProjects\ directory.

4.     On the Tools tab, in the Fuser Automation group, click AWS Instance Settings.

5.     In the Auto-Scaling Fusers Settings dialog, do the following:

a.     Check the Use AWS Instances check box.

b.     Maximum Price Per Spot Instance: Set the maximum price you want to pay per EC2 spot instance. For more information, see Amazon EC2 Spot Instance Pricing in Amazon’s AWS documentation.

c.     Maximum Instances: Set the number based on the number of EC2 instances you want to run.

d.     Launch Rate: Set the maximum number of fuser instances to launch per minute, e.g. if Maximum Instances was set to 200, and the Launch Rate to 50, the Build Manager will spread the launch of the fusers over 4 minutes. This prevents a potential bottleneck that could be caused by a large number of fusers attempting to read the build data at the same time.

e.     Terminate AWS instances when build finished: Select the check box if you want to terminate AWS fuser instances when the build is complete.

f.      Configuration Name: Select based on the current processing step. Select the GPU_Reconstruction option if you want to run an entire build without needing to reselect the configuration before each build step, since this option can be used for all build steps.

·         GPU_Reconstruction – For texture and output steps.

·         AT_MultiCoreCPU – Multi-core CPU for data preparation and aerotriangulation steps.

·         Reconstruction_CPU – For point cloud and model steps.

Note:    These configuration options are read from the JSON files copied in step 2 above. Each of these JSON files contains an array of instance types that PhotoMesh should use, ordered based on preferability. During production, PhotoMesh iterates over the elements of the array, beginning with the first instance type, until it finds an available instance. PhotoMesh will always attempt first to launch instances from the same availability zone as the master, and only if no instances are available, will it launch an instance from a different zone.
Each instance object in the array has four properties: ami, instanceType, demandType, and fusersCount. The instanceType defines the hardware of the host computer used for your instance. Each instance type offers different compute, memory, and storage capabilities. The demandType determines the pricing arrangement: "onDemand" or "spot." The fusersCount determines the number of PhotoMesh processes to run on each instance. See the overview to step 4 for some of the issues to consider when selecting instance types.

g.     Build Priority: Select either Choose all listed options, to use all machine types listed in the JSON file, with priority based on the order they are listed in the file, or select a specific machine.

h.     Security Group ID: Select PM_SG.

i.      Subnet ID: Click Additional Subnets... and type a list of all the subnet ID's you want to use. These were created in "Configuring a VPC". A subnet must be created for each availability zone. Since you are charged for data transfer from one availability zone to another, it is recommended to launch the master in an availability zone with good spot availability. PhotoMesh will always attempt first to launch instances from the same availability zone as the master, and only if no instances are available, will it launch an instance from a different zone. See the overview to step 4 for some of the issues to consider when selecting a subnet/availability zone.

j.      Key Pair Name: Select PM_KeyPair.

k.     Click Edit (next to Configuration Name), insert in all fields the AMI ID of the PM Linux Fuser Image (created in "Saving a Fuser AMI"), and then click OK.

l.      Select Shutdown inactive fuser instances after and set its value to "1".

m.   Click Edit Startup Script and paste the template script from the zip file downloaded in step 6b of "Customizing the (Fuser) Instance" (Startup_Script_awsQG_Template). Edit the script so it includes the username and the password of your PM Master machine set in step 13 of "Connecting to the (Master) Instance" (In the script, replace all <pass> and <user> placeholders with the required value.

n.     Click OK.

6.     Click Build.

Note:        If a message is displayed that "PhotoMesh’s fuser request was rejected by AWS, since it exceeded the instance limit", you need to request a quota increase for "Spot instance requests". See: Amazon EC2 Service Limits in Amazon’s AWS documentation for more information.

7.     In the PhotoMesh Build Manager dialog, you can monitor the creation and automatic use of the automatically launched fuser instances. You can also monitor the creation of your instances by choosing Instances in the navigation pane of the Amazon EC2 console, and reviewing the custom tags added to PhotoMesh launched instances: Build start time, Owner, Project, Type, and Working folder.

8.     When processing is complete, stop the PM Master instance.

Note:        Do not terminate the master instance; only stop and start as needed.

Note:        It is recommended to verify that the instances are terminated according to the settings defined in the AWS Cloud Settings dialog, particularly the first time they are terminated.

Creating an IAM User for Running PhotoMesh Project Using an On-Premises Master

To allow PhotoMesh Master running on an on-premises computer to launch EC2 instances, create an IAM user and assign permissions.

See: Creating IAM Users (Console) in Amazon’s AWS documentation for more information.

1.     Open the Amazon IAM console at https://console.aws.amazon.com/iam/.

2.     In the navigation pane, choose Users and then choose Add user.

3.     Type the user name for the new user.

4.     Select the Programmatic access check box. Then choose Next: Permissions.

5.     On the Set Permissions page, choose Attach policies directly.

6.     In the list of policies, select the PM_RUNFUSER_POLICY policy. You can use the Filter menu to filter the list of policies.

7.     Choose Next: Review and Create.

8.     Review the role and then choose Create user.

9.     To view the users' access keys (access key IDs and secret access keys), choose Show next to each password and access key that you want to see. To save the access keys, choose Download .csv and then save the file to a safe location.

Note:        This is your only opportunity to view or download the secret access keys. Save the user's new access key ID and secret access key in a safe and secure place. You will not have access to the secret keys again after this step.

Building a PhotoMesh Project Using an On-Premises Master and Storage

Before beginning a build using an on-premises master and storage, ensure that your fuser machines have access to your local storage, e.g., via AWS Direct Connect, Site-to-Site VPN, or Security Group and firewall rules. Sufficient bandwidth for the data transfer between your on-premises storage and the fuser machines significantly affects production performance. See the overview to step 4 for some of the issues to consider regarding the hybrid architecture.

To use the configured environment to produce PhotoMesh projects using on-premises storage and a PhotoMesh that is installed on an on-premises computer while utilizing AWS fuser instances, follow the steps below:

1.     Make sure your on-premises computer has PhotoMesh installed.

Note:        File access can be achieved in different ways, e.g. via AWS Direct Connect, Site-to-Site VPN, or Security Group and firewall rules. Consult AWS documentation, support, your IT person, or Skyline support for more information.

2.     Create the directory: ".\PhotoMesh\AWSConfigurations", and copy the JSON files ("DataPrep_AT.json", "Reconstruction_CPU.json", and "Reconstruction_GPU.json") from the zip file downloaded in step 6b of "Customizing the (Fuser) Instance" to this directory.

3.     Start PhotoMesh and create a project.

Note:        The project must be created in a location that is accessible to all fuser instances.

4.     On the Home tab, in the Process group, click Build, then enter the Build Parameters and click Build.

5.     In the PhotoMesh Build Manger dialog, toggle on Use AWS Instances (Change).

6.     In the AWS Security dialog, enter your AWS credentials created in "Creating an IAM User for Running PhotoMesh Project Using an On-Premises Master".

7.     In the Auto-Scaling Fusers Settings dialog, do the following:

a.     Check the Use AWS Instances check box.

b.     Maximum Price Per Spot Instance: Set the maximum price you want to pay per EC2 spot instance. For more information see Amazon EC2 Spot Instance Pricing in Amazon’s AWS documentation.

c.     Maximum Instances: Set the number based on the number of EC2 instances you want to run.

d.     Launch Rate: Set the maximum number of fuser instances to launch per minute, e.g. if Maximum Instances was set to 200, and the Launch Rate to 50, the Build Manager will spread the launch of the fusers over 4 minutes. This prevents a potential bottleneck that could be caused by a large number of fusers attempting to read the build data at the same time.

e.     Terminate AWS instances when build finished: Select the check box if you want to terminate AWS fuser instances when the build is complete.

f.      Build Priority: Select either Choose all listed options, to use all machine types listed in the JSON file, with priority based on the order they are listed in the file, or select a specific machine.

g.     Security Group ID: Select PM_SG.

h.     Subnet ID: Click Additional Subnets... and type a list of all the subnet ID's you want to use. These were created in "Configuring a VPC". A subnet must be created for each availability zone.  See the overview to step 4 for a comparison of onDemand/spot instances and information on using pricing history to determine instance availability.

i.      Key Pair Name: Select PM_KeyPair.

j.      Configuration Name: Select based on the current processing step. Select the Reconstruction_GPU option if you want to run an entire build without needing to reselect the configuration before each build step, since this option can be used for all build steps.

·         Reconstruction_GPU – For texture and output steps.

·         AT_MultiCoreCPU – Multi-core CPU for data preparation and aerotriangulation steps.

·         Reconstruction_CPU – For point cloud and model steps.

Note:    These configuration options are derived from the JSON files copied in step 2 of "Building a PhotoMesh Project Using the Master Instance". Each of these JSON files contains an array of instance types that PhotoMesh should use, ordered based on preferability. During production, PhotoMesh iterates over the elements of the array, beginning with the first instance type, until it finds an available instance.
Each instance object in the array has four properties: ami, instanceType, demandType, and fusersCount. The instanceType defines the hardware of the host computer used for your instance. Each instance type offers different compute, memory, and storage capabilities. The demandType determines the pricing arrangement: onDemand or spot. The fusersCount determines the number of PhotoMesh processes to run on each instance. See the overview to step 4 for some of the issues to consider when selecting instance types.

k.     Click Edit (next to Configuration Name), insert in all fields the AMI ID of the PM Linux Fuser Image (created in "Saving a Fuser AMI"), and then click OK.

l.      Select Shutdown inactive fuser instances after and set its value to "1".

m.   Click Edit Startup Script and paste the template script from the zip file downloaded in step 6b of "Customizing the (Fuser) Instance (Startup_Script_Template.txt). Edit the script so it includes the username and the password of your PM Master machine set in step 13 of "Connecting to the (Master) Instance" (In the script, replace all <serverip >, <share>, <pass> and <user> placeholders with the required value).

n.     Click OK.

8.     Click Build.

Note:        A message is displayed informing you that no available fusers are currently running and asking if you want to start a local fuser. If you want to start a fuser on the master machine to take advantage of its computing resources, click Yes. If you are running many spot fuser instances, however, it is recommended to click No to free up computing resources for management of the build and file server tasks.

Note:        If a message is displayed that "PhotoMesh’s fuser request was rejected by AWS, since it exceeded the instance limit", you need to request a limit increase for "Spot instance requests". See: Amazon EC2 Service Limits in Amazon’s AWS documentation for more information.

9.     In the PhotoMesh Build Manger dialog, you can monitor the creation and automatic use of the automatically launched fuser instances. You can also monitor the creation of your instances by choosing Instances in the navigation pane of the Amazon EC2 console, and reviewing the custom Tags added to PhotoMesh launched instances: Build start time, Owner, Project, Type, and Working folder.

10.  It is recommended to verify that the instances are terminated according to the settings defined in the AWS Cloud Settings dialog, particularly the first time they are terminated.