EchoPT: A Pretrained Transformer Architecture for Predicting 2D In-Air Sonar Images in Mobile Robotics

<h1>EchoGPT</h1> <p>This folder contains the supplementary data and code for the submission "EchoPT: A Pretrained Transformer Architecture for Predicting 2D In-Air Sonar Images in Mobile Robotics" to the NeurIPS 2024 conference.</p> <h2>Dependencies</h2> <p>Matlab 2024a or higher is needed with the following toolboxes:</p> <ul> <li>Image Processing Toolbox</li> <li>Parallel Computing Toolbox</li> <li>Deep Learning Toolbox</li> <li>Signal Processing Toolbox</li> <li>System Identification Toolbox</li> </ul> <h2>Data</h2> <h3>Simulation</h3> <p>This folder contains the saved sonar images (energyscapes) as well as the motion data for each frame from a particular simulation run (LongRun1). These are saved into batches and saved as compressed .mat files within the <em>\DataCalculated\SimulationData\LongRun1\Raw</em> folder.</p> <h3>Trained model</h3> <p>The model used in the experiments of this submission is saved in <em>\DataCalculated\Networks</em> as a Matlab dlnetwork object. There is a trained (with weights) and an untrained version available.</p> <h2>Code</h2> <h3>Data pre-processing</h3> <p>To extract the data from the batch .mat files into seperate frames a script <strong>preprocessEchoPT.m</strong> is available in the main folder. This will save these individual frames once again to individual .mat files into a folder <em>\DataCalculated\ESSequences</em> by default.</p> <h3>Training</h3> <p>To train the model a Matlab script <strong>trainEchoPT.m</strong> is provided.</p> <h3>Evaluation</h3> <p>To generate the figures of the submission as well as additional outputs such as GIFs three different evaluation scripts are available:</p> <ul> <li><strong>evaluateEchoPT.m</strong>: Evaluate the trained model on the dataset.</li> <li><strong>evaluateEchoPT_ComparisonAccFlow_ARLoop.m</strong>: Evaluate the trained model on the dataset and compare it to acoustic flow with autoregressive prediction.</li> <li><strong>evaluateEchoPT_ComparisonAccFlow_NoAR.m</strong>: Evaluate the trained model on the dataset and compare it to acoustic flow without autoregressive prediction.</li> </ul> <h3>Source</h3> <p>Within the folder <em>\Source</em> all additional Matlab functions and classes can be found.</p> <h2>Open-Source libraries included in this project</h2> <ul> <li>Progress bar by HyunGwang Cho <a href="https://www.mathworks.com/matlabcentral/fileexchange/121363-progress-bar-cli-gui-parfor?s_tid=srchtitle">(link)</a></li> </ul>