Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
7,184
datasets available to search
ShareScore release 0.9.0
Dataset results
7,184 results for “recurrence”
Reproduction package for 'Low-frequency radio observations of recurrent nova RS Ophiuchi with MeerKAT and LOFAR'
<p>This is a basic reproduction package for the paper "Low-frequency radio observations of recurrent nova RS Ophiuchi with MeerKAT and LOFAR".</p><p> </p><p> </p>
Data of "Self-consistency Reinforced minimal Gated Recurrent Unit for surrogate modeling of history-dependent non-linear problems: application to history-dependent homogenized response of heterogeneous materials"
<h1>Development of the Self-Consistency reinforced Minimum Recurrent Unit (SC-MRU)</h1> <p>This directory contains the data and algorithms generated in publication<sup><a href="#fn-1-5292">1</a></sup></p> <h2>Table of Contents</h2> <ol> <li><a href="#dependencies-and-prerequisites">Dependencies and Prerequisites</a></li> <li><a href="#structure-of-repository">Structure of Repository</a></li> <li><a href="#part-1-data-preparation">Part 1: Data preparation</a></li> <li><a href="#part-2-rnn-training">Part 2: RNN training</a></li> <li><a href="#part-3-multiscale-analysis">Part 3: Multiscale analysis</a></li> <li><a href="#part-4-reproduce-paper1-figures">Part 4: Reproduce paper[^1] figures</a></li> </ol> <h2>Dependencies and Prerequisites</h2> <p> </p> <ul> <li> <p>Python, pandas, matplotlib, texttabble and latextable are pre requisites for visualizing and navigating the data.</p> </li> <li> <p>For generating mesh and for vizualization, gmsh (<a href="http://www.gmsh.info" target="_blank" rel="nofollow noreferrer noopener">www.gmsh.info</a>) is required.</p> </li> <li> <p>For running simulations, cm3Libraries (<a href="http://www.ltas-cm3.ulg.ac.be/openSource.htm" target="_blank" rel="nofollow noreferrer noopener">http://www.ltas-cm3.ulg.ac.be/openSource.htm</a>) is required.</p> </li> </ul> <h3>Instructions using apt & pip3 package manager</h3> <p>Instructions for Debian/Ubuntu based workstations are as follows.</p> <h3>python, pandas and dependencies</h3> <div> <pre><code> sudo apt install python3 python3-scipy libpython3-dev python3-numpy python3-pandas</code></pre> </div> <h3>matplotlib, texttabble and latextable</h3> <div> <pre><code> pip3 install matplotlib texttable latextable</code></pre> </div> <h3>Pytorch (only for run with cm3Libraries)</h3> <ul> <li>Without GPU</li> </ul> <div> <pre><code> pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu</code></pre> </div> <ul> <li>With GPU</li> </ul> <div> <pre><code> pip3 install torch torchvision torchaudio</code></pre> </div> <h3>Libtorch (for compiling the cells)</h3> <ul> <li>Without GPU: In a local directory (e.g. <code>~/local</code> with <code>export TORCHDIR=$HOME/local/libtorch</code>)</li> </ul> <div> <pre><code> wget https://download.pytorch.org/libtorch/cpu/libtorch-shared-with-deps-2.1.1%2Bcpu.zip unzip libtorch-shared-with-deps-2.1.1%2Bcpu.zip</code></pre> </div> <ul> <li>With GPU: In a local directory (e.g. <code>~/local</code> with <code>export TORCHDIR=$HOME/local/libtorch</code>)</li> </ul> <div> <pre><code> wget https://download.pytorch.org/libtorch/cu121/libtorch-shared-with-deps-2.1.1%2Bcu121.zip unzip libtorch-shared-with-deps-2.1.1+cu121.zip</code></pre> </div> <h2>Structure of Repository</h2> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/All_Path_Res">All_Path_Res</a>: results of the direct numerical simulations used as training and testing data, see details in <a href="#part-1-data-preparation">Part 1: Data preparation</a>.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/ConstRVE">ConstRVE</a>: script to run direct numerical finite element simulations, see details in <a href="#part-1-data-preparation">Part 1: Data preparation</a>.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale">MultiScale</a>: scripts to run and visualise the multiscale analyses, see details in <a href="#part-3-multiscale-analysis">Part 3: Multiscale analysis</a>.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU">SC_MRU</a>: implementation of the RNN and scripts to train them, see details in <a href="#part-2-rnn-training">Part 2: RNN training</a>.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/TrainingData">TrainingData</a>: scripts to collect, normalise and truncate the RVEs direct simulation results as training and testing data, see details in <a href="#part-1-data-preparation">Part 1: Data preparation</a>. The director also contained the storred processed data used in <sup><a href="#fn-1">1</a></sup>.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/TrainingPaths">TrainingPaths</a>: scripts to generate the different loading paths for the direct numerical simulations used as training and testing data, see details in <a href="#part-1-data-preparation">Part 1: Data preparation</a>.</li> </ul> <h2>Part 1: Data preparation</h2> <h3>Generate the loading paths</h3> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingPaths/testGenerationData.py">TrainingPaths/testGenerationData.py</a> is used to generate random walk paths, with the options <ul> <li><code>Rmax = 0.11</code> # bound on the final Green Lagrange strain</li> <li><code>TimeStep = 1.</code> # in second</li> <li><code>EvalStep = [1e-4,5e-3]</code> #Bounds on the Green Lagrange increments</li> <li><code>Nmax = 2500</code> #maximum length of the sequence</li> <li><code>k = 4000</code> # number of path to generate</li> <li>The path are storred by default in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/ConstRVE/Paths">ConstRVE/Paths/</a>. The path has to be existing before launching the script. You can change the name in line 123 <code>saveDir = '../ConstRVE'+'/Paths/'</code>.</li> <li>Examples of generated paths can be found in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/ConstRVE/PathsExamples">ConstRVE/PathsExamples/</a></li> <li>The command to be run from the directory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/TrainingPaths">TrainingPaths</a> is</li> </ul> </li> </ul> <div> <pre><code>(mkdir ../ConstRVE/Paths) #if needed python3 testGenerationData.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingPaths/generationData_Cyclic.py">TrainingPaths/generationData_Cyclic.py</a> is used to generate random cylic paths, with the options <ul> <li><code>Rmax = [np.random.uniform(0.,0.04),np.random.uniform(0.,0.06),np.random.uniform(0.0,0.09),0.12]</code> # bound on the final Green Lagrange strain is random</li> <li><code>TimeStep = 1.</code> # in second</li> <li><code>EvalStep = [1e-4,5e-3]</code> #Bounds on the Green Lagrange increments</li> <li><code>Nmax = 2500</code> #maximum length of the sequence</li> <li><code>k = 2000</code> # number of path to generate</li> <li>The path are stored by default in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/ConstRVE/Paths">ConstRVE/Paths/</a>. You can change the name in line 123 <code>saveDir = '../ConstRVE'+'/Paths/'</code>.</li> <li>The command to be run from the directory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/TrainingPaths">TrainingPaths</a> is</li> </ul> </li> </ul> <div> <pre><code>(mkdir ../ConstRVE/Paths) #if needed python3 generationData_Cyclic.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingPaths/countPathLength.py">TrainingPaths/countPathLength.py</a> gives average, minimum and maximum lengths of the generated paths and the distribution of the <code>\Delta R</code>. By default the paths are read in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/ConstRVE/Paths">ConstRVE/Paths/</a> but the directory can be given as an argument. The file can be used to read <ul> <li>either the generated loading paths</li> </ul> </li> </ul> <div> <pre><code> python3 countPathLength.py '../ConstRVE/PathsExamples'</code></pre> </div> <ul> <li> <ul> <li>or the results of the <a href="#generate-the-rves-direct-simulation-results">simulations</a></li> </ul> </li> </ul> <div> <pre><code> python3 countPathLength.py '../All_Path_Res/Path_Res9'</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingPaths/graphData.py">TrainingPaths/graphData.py</a> generates illustrations from randomly picked paths in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/ConstRVE/Paths">ConstRVE/Paths/</a> and generate png figures.</li> </ul> <h3>Generate the RVEs direct simulation results</h3> <ul> <li>Uses the loading paths existing in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/ConstRVE/Paths">ConstRVE/Paths/</a>.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/ConstRVE/rve.geo">ConstRVE/rve.geo</a> is the RVE geometry file that can be read by gmsh (<a href="http://www.gmsh.info" target="_blank" rel="nofollow noreferrer noopener">www.gmsh.info</a>).</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/ConstRVE/rve.msh">ConstRVE/rve.msh</a> is the RVE mesh file that can be read by gmsh (<a href="http://www.gmsh.info" target="_blank" rel="nofollow noreferrer noopener">www.gmsh.info</a>).</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/ConstRVE/utilsFunc.py">ConstRVE/utilsFunc.py</a> contains python tools to be used.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/ConstRVE/Rve_withoutInternalVars.py">ConstRVE/Rve_withoutInternalVars.py</a> is used to run all the RVE simulations: <ul> <li>This requires cm3Libraries (<a href="http://www.ltas-cm3.ulg.ac.be/openSource.htm" target="_blank" rel="nofollow noreferrer noopener">http://www.ltas-cm3.ulg.ac.be/openSource.htm</a>).</li> <li>All the ouptus are stored in <code>All_Path_Res/Path_Res12</code>, you can change the name in line 71 <code>Path_Res = '../All_Path_Res/Path_Res12/'</code>. The results of RVE simulations are saved as the sequence (one configuration per line) of the Green-Lagrange strains and Second Piola-Kirchhoff stress (in column). One example can be found in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/All_Path_Res/Path_Res1/data_path1000.csv">All_Path_Res/Path_Res1/data_path1000.csv</a>.</li> <li>The command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/ConstRVE">ConstRVE</a> directory is</li> </ul> </li> </ul> <div> <pre><code> python3 Rve_withoutInternalVars.py</code></pre> </div> <h3>Collect, normalised and truncate the RVEs direct simulation results as training and testing data</h3> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/CheckNanData.py">TrainingData/CheckNanData.py</a> is used to check the integrity of the direct numerical simulations results <ul> <li>DNS results are read from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/All_Path_Res/Path_res1">All_Path_Res/Path_res1</a> to <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/All_Path_Res/Path_res11">All_Path_Res/Path_res11</a> subdirectories.</li> <li>The command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/TrainingData">TrainingData</a> directory is</li> </ul> </li> </ul> <div> <pre><code> python3 CheckNanData.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/CollectData.py">TrainingData/CollectData.py</a> is used to gather all the direct numerical simulations results <ul> <li>DNS results are read from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/All_Path_Res/Path_res1">All_Path_Res/Path_res1</a> to <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/All_Path_Res/Path_res11">All_Path_Res/Path_res11</a> subdirectories. It can be changed in line 31 <code>for ll in range(11):</code>.</li> <li>It saves the bounds and raw data in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Processed_Data/Bounds_GS">TrainingData/Processed_Data/Bounds_GS</a> and <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Processed_Data/Origin_GS">TrainingData/Processed_Data/Origin_GS</a>, respectively.</li> <li>The command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/TrainingData">TrainingData</a> directory is</li> </ul> </li> </ul> <div> <pre><code> python3 CollectData.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Normalization.py">TrainingData/Normalization.py</a> is used to normalise the gathered direct numerical simulations results <ul> <li>Bounds and raw data are read from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Processed_Data/Bounds_GS">TrainingData/Processed_Data/Bounds_GS</a> and <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Processed_Data/Origin_GS">TrainingData/Processed_Data/Origin_GS</a>, respectively.</li> <li>It saves the normalized training (75%) and testing (25%) data in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Processed_Data/Normalized_GS_Train">TrainingData/Processed_Data/Normalized_GS_Train</a> and <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Processed_Data/Normalized_GS_Test">TrainingData/Processed_Data/Normalized_GS_Test</a>, respectively.</li> <li>The command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/TrainingData">TrainingData</a> directory is</li> </ul> </li> </ul> <div> <pre><code> python3 Normalization.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/GaussCollectData.py">TrainingData/GaussCollectData.py</a> is an alternative using Gaussian normaliation and is used to gather all the direct numerical simulations results.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/GaussNormalization.py">TrainingData/GaussNormalization.py</a> is an alternative to normalise following a Gaussian the gathered direct numerical simulations results.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Data_Padding.py">TrainingData/Data_Padding.py</a> is used to pad and trim the normalised data <ul> <li>Normalized training (75%) and testing (25%) data in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Processed_Data/Normalized_GS_Train">TrainingData/Processed_Data/Normalized_GS_Train</a> and <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Processed_Data/Normalized_GS_Test">TrainingData/Processed_Data/Normalized_GS_Test</a>, respectively.</li> <li>The final length of the sequence (including zero padding and trimming) is given by <code>N = 200</code>.</li> <li>It saves the trimmed and padded normalised training and testing data in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Processed_Data">TrainingData/Processed_Data/</a> as <code>'GS_Train'N</code> and <code>'GS_Test'N</code>, respectively.</li> <li>The command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/TrainingData">TrainingData</a> directory is</li> </ul> </li> </ul> <div> <pre><code> python3 Data_Padding.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Tool.py">TrainingData/Tool.py</a> is a list of function used to normalise dat.</li> </ul> <h2>Part 2: RNN training</h2> <h3>Available rnn</h3> <ul> <li>The different cells are in the following directories <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/MGRU/NNW_SMRU">SC_MRU/MGRU/NNW_SMRU</a>: Neural network with SMRU recurrent cell.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/MGRU/NNW_SCMRU_T">SC_MRU/MGRU/NNW_SCMRU_T</a>: Neural network with SC-MRU-T recurrent cell.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/MGRU/NNW_MGRU">SC_MRU/MGRU/NNW_MGRU</a>: Neural network with orginal MGRU recurrent cell.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/MGRU/NNW_MGRU_M">SC_MRU/MGRU/NNW_MGRU_M</a>: Neural network with modified MGRU recurrent cell.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/FNN_LeakyReLU">SC_MRU/FNN_LeakyReLU/NNW_<code>X</code>Fw</a>: Neural network with SC-MRU-I recurrent cell using <code>X</code> Feed Forward non-linear transition layers.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/Quadratic_NLT">SC_MRU/Quandratic_NLT/NNW_<code>X</code>Q, NNW_Q_Fw, NNW_Fw_Q</a>: Neural network with SC-MRU-I recurrent cell using <code>X</code> quadratic or mixed feed-forward-quadratic non-linear transition layers.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/ReferenceRNN">SC_MRU/ReferenceRNN/NNW_<code>X</code>Layers</a>: Neural network with SC-LMSC recurrent cell using <code>X</code> non-linear transition layers.</li> </ul> </li> </ul> <h3>Compile the neural network models</h3> <ul> <li>In the adequate directory, e.g. <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/MGRU/NNW_SMRU">SC_MRU/MGRU/NNW_SMRU</a> for the Neural network with SMRU recurrent cell.</li> </ul> <div> <pre><code> cd build rm -rf * cmake -DCMAKE_PREFIX_PATH=$TORCHDIR .. make</code></pre> </div> <ul> <li>This create the RNN_<code>CELL</code> model in the build directory</li> </ul> <h3>Train the neural network models</h3> <ul> <li>In the adequate directory, e.g. <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/MGRU/NNW_SMRU">SC_MRU/MGRU/NNW_SMRU</a> for the Neural network with SMRU recurrent cell.</li> <li>Requires to have <a href="#compile-the-neural-network-models">compiled</a> the RNN model.</li> <li>The file <code>Train.py</code> <ul> <li>Uses the RNN_<code>CELL</code> model compiled in the <code>build</code> directory.</li> <li>Uses the trimmed and padded normalised training and testing data in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/TrainingData/Processed_Data">TrainingData/Processed_Data/</a> as <code>'GS_Train'N</code> and <code>'GS_Test'N</code>, respectively.</li> <li>Can be modified to use the requested ratios of training and testing sequences of different lengths to prepate the mini-baches, e.g.: <ul> <li><code>ratio = [0.2,0.02,0.6,0.06,0.01,0.9]</code></li> <li><code>PathIn1 = ['../../../TrainingData/Processed_data/GS_Train200','../../../TrainingData/Processed_data/GS_Train2500','../../../TrainingData/Processed_data/GS_Test200','../../../TrainingData/Processed_data/GS_Test2500']</code></li> <li><code>PathIn2 = ['../../../TrainingData/Processed_data/GS_Test200','../../../TrainingData/Processed_data/GS_Test400','../../../TrainingData/Processed_data/GS_Test2500','../../../TrainingData/Processed_data/GS_Test400']</code></li> </ul> </li> <li>Saves the mini-batches in <ul> <li><code>PathOut = "TrainingData.pt"</code></li> </ul> </li> <li>Saves the model <code>module-checkpoint.pt</code> and <code>module-checkpoint-optimizer.pt</code>, and loss evolution <code>Loss.txt</code> in <ul> <li>Module/H<code>n</code> with <code>n</code> the number of hiden variables.</li> <li>Output directory can be changes in NNW_<code>CELL</code>.cpp of the cell name <code>CELL</code> (and recompiling).</li> <li>Warm start can be disabled in by commenting <code>torch::load(net, "./Module/H120/module-checkpoint_CM0.pt");</code> and <code>torch::load(optimizer, "./Module/H120/module-optimizer-checkpoint_CM0.pt");</code> in NNW_<code>CELL</code>.cpp of the cell name <code>CELL</code> (and recompiling).</li> <li>Is executed with</li> </ul> </li> </ul> </li> </ul> <div> <pre><code> python3 Train.py</code></pre> </div> <h3>Convert trained c++ models for pyTorch (in view of multiscale simulations)</h3> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/CheckLoss/Save_RU.py">SC_MRU/CheckLoss/RU.py</a>: functions used to read and use the rnn models.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/CheckLoss/Save_RNN_script.py">SC_MRU/CheckLoss/Save_RNN_script.py</a>: <ul> <li>Converts c++ trained models to pyTorch models</li> <li>Reads the trained models in SC_MRU/CELL_<code>Kind</code>/NNW_<code>CELL</code>/Module/H<code>N</code>/module.pt</li> <li>Save the converted models to <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/CheckLoss">SC_MRU/CheckLoss/</a> and to <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale/Model">MultiScale/Model/</a></li> <li><code>InpType = cell[0]</code> #choose the recurrent cell typeAmong the different available cells <code>cell=["SC_MRU_T","SC_MRU_I","SMRU"]</code>.</li> <li>The command to be run from the directory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/CheckLoss">SC_MRU/CheckLoss/</a> is</li> </ul> </li> </ul> <div> <pre><code>python3 Save_RNN_script.py</code></pre> </div> <h3>Vizualize loss evolution and testing results</h3> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/CheckLoss/Data_prepare.py">SC_MRU/CheckLoss/Data_prepare.py</a> contains fucntion used for testing.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/CheckLoss/LossVS_insertN.py">SC_MRU/CheckLoss/LossVS_insertN.py</a> tests the effect of testing data augmentation: <ul> <li>Requires to have <a href="#compile-the-neural-network-models">compiled</a> the RNN models.</li> <li>Cell type and augmentation kind can be modified: <ul> <li><code>repeat = 10</code> #number of tests</li> <li><code>dataType ='random'</code> #'even' or 'random'</li> <li><code>InpType = cell[0]</code> #choose the recurrent cell typeAmong the different available cells ```cell=["SC_MRU_T","SC_MRU_I","SMRU"]``</li> <li><code>reevaluate= False</code> # <code>False</code> to use the saved loss values and <code>True</code> to evaluate the loss values</li> <li><code>fastshifting=False</code> # <code>False</code> to vizualize before fast shifting (Fig. 11) and <code>True</code> after (Fig. 12). When <code>reevaluate== True</code>, this has no effect: the training with fast-shifting or not has to be done manually, see <a href="#train-the-neural-network-models">details</a>.</li> </ul> </li> <li>Generates new Loss_<code>CELL</code>.txt and TrainingData.pt files in case <code>reevaluate== True</code>, read the existing one if <code>reevaluate== False</code>.</li> <li>Generates Figs. 11 and 12 from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/CheckLoss">SC_MRU/CheckLoss/</a></li> </ul> </li> </ul> <div> <pre><code> python3 LossVS_insertN.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/CheckLoss/Plot_GS.py">SC_MRU/CheckLoss/Plot_GS.py</a> test the different NNW on RVE testing paths: <ul> <li>Requires to have <a href="#compile-the-neural-network-models">compiled</a> the RNN models.</li> <li>Cell type and augmentation kind can be modified: <ul> <li><code>InpType = cell[0]</code> #choose the recurrent cell typeAmong the different available cells ```cell=["SC_MRU_T","SC_MRU_I","SMRU"]``</li> </ul> </li> <li>Generate Figs. 13, 14 and 15 from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/CheckLoss">SC_MRU/CheckLoss/</a></li> <li>Command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/CheckLoss">SC_MRU/CheckLoss/</a> is</li> </ul> </li> </ul> <div> <pre><code> python3 Plot_GS.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/CheckLoss/Plot_GS_step.py">SC_MRU/CheckLoss/Plot_GS_step.py</a> tests the different pyTorch NNWs on the RVE testing paths: <ul> <li>Requires to have <a href="#convert-trained-c-models-for-pytorch-in-view-of-multiscale-simulations">converted</a> the RNN models.</li> <li>Cell type and augmentation kind can be modified: <ul> <li><code>InpType = cell[0]</code> #choose the recurrent cell typeAmong the different available cells ```cell=["SC_MRU_T","SC_MRU_I","SMRU"]``</li> </ul> </li> <li>Command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/CheckLoss">SC_MRU/CheckLoss/</a> is</li> </ul> </li> </ul> <div> <pre><code> python3 Plot_GS_step.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/PlotLoss_HiddenV.py">SC_MRU/PlotLoss_HiddenV.py</a> shows the loss evolution for the different number of hidden variables --<a href="#train-the-neural-network-models">after having trained</a> or using saved loss files-- for the SMRU cell: <ul> <li>Generates Fig. 8.</li> <li>Command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU">SC_MRU/</a> is</li> </ul> </li> </ul> <div> <pre><code> python3 PlotLoss_HiddenV.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/PlotLoss_3MRU.py">SC_MRU/PlotLoss_3MRU.py</a> shows the loss evolution for the different recurrent units --<a href="#train-the-neural-network-models">after having trained</a> or using saved loss files-- for the 120 hidden variables: <ul> <li>Generates Fig. 9.</li> <li>Command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU">SC_MRU/</a> is</li> </ul> </li> </ul> <div> <pre><code> python3 PlotLoss_3MRU.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/PlotLoss_RefNL_H120.py">SC_MRU/PlotLoss_RefNL_H120.py</a> shows the loss evolution for the different SC-LMSC and SC-MRU-I cells --<a href="#train-the-neural-network-models">after having trained</a> or using saved loss files-- for 120 hidden variables: <ul> <li>Generates Fig. 10.</li> <li>Command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU">SC_MRU/</a> is</li> </ul> </li> </ul> <div> <pre><code> python3 PlotLoss_RefNL_H120.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/PlotLoss_MGRU.py">SC_MRU/PlotLoss_MGRU.py</a> shows the loss evolution for the original and modified MGRU --<a href="#train-the-neural-network-models">after having trained</a> or using saved loss files-- and for the 120 hidden variables: <ul> <li>Generates Fig. A.18.</li> <li>Command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU">SC_MRU/</a> is</li> </ul> </li> </ul> <div> <pre><code> python3 PlotLoss_MGRU.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/PlotLoss_NL.py">SC_MRU/PlotLoss_NL.py</a> shows the loss evolution for the different non-liner transition layers (quadratic and Leaky ReLU) of the SC-MRU-I cell --<a href="#train-the-neural-network-models">after having trained</a> or using saved loss files-- and for the 120 hidden variables: <ul> <li>Generates Fig. B.19.</li> <li><code>Case=1</code> # 0 for quadratic transition blocks and 1 for Leaky ReLU transition blocks</li> <li>Command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU">SC_MRU/</a> is</li> </ul> </li> </ul> <div> <pre><code> python3 PlotLoss_NL.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/SC_MRU/PlotLoss_DiffNL_H120.py">SC_MRU/PlotLoss_DiffNL_H120.py</a> shows the loss evolution for the different non-liner transition layers (hybrid) of the SC-MRU-I cell --<a href="#train-the-neural-network-models">after having trained</a> or using saved loss files-- and for the 120 hidden variables: <ul> <li>Generates Fig. B.20.</li> <li>Command to be run from <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU">SC_MRU/</a> is</li> </ul> </li> </ul> <div> <pre><code> python3 PlotLoss_DiffNL_H120.py</code></pre> </div> <h2>Part 3: Multiscale analysis</h2> <h3>Trained surrogate models</h3> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale/Model">MultiScale/Model</a>: contains the different rnn <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/Model/Bounds_GS">MultiScale/Model/Bounds_GS</a>: Bounds.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/Model/DInpFullModel.pt">MultiScale/Model/DInpFullModel.pt</a>: trained rnn with <code>SC-MRU-T</code> recurrent cell.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/Model/IncrementModel.pt">MultiScale/Model/IncrementModel.pt</a>: trained rnn with <code>SC-MRU-I</code> recurrent cell.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/Model/FullModel.pt">MultiScale/Model/FullModel.pt</a>: trained rnn with <code>SMRU</code> recurrent cell.</li> </ul> </li> </ul> <h3>Run mutiscale simulations using the surrogates</h3> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/2D_MultiScale/model.geo">MultiScale/2D_MultiScale/model.geo</a>: geometry of the macro-scale model that can be read by gmsh (<a href="http://www.gmsh.info" target="_blank" rel="nofollow noreferrer noopener">www.gmsh.info</a>).</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/2D_MultiScale/model.msh">MultiScale/2D_MultiScale/model.msh</a>: mesh of the macro-scale model that can be read by gmsh (<a href="http://www.gmsh.info" target="_blank" rel="nofollow noreferrer noopener">www.gmsh.info</a>).</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/2D_MultiScale/model.py">MultiScale/2D_MultiScale/model.py</a>: is used to run all the multiscale simulation: <ul> <li>This requires cm3Libraries (<a href="http://www.ltas-cm3.ulg.ac.be/openSource.htm" target="_blank" rel="nofollow noreferrer noopener">http://www.ltas-cm3.ulg.ac.be/openSource.htm</a>).</li> <li>Uses the bounds and trained models in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale/Model">MultiScale/Model</a>.</li> <li><code>InpType = cell[0]</code> #choose the recurrent cell typeAmong the different available cells <code>cell=["SC_MRU_T","SC_MRU_I","SMRU"]</code>.</li> <li><code>factorStep= 100</code> # is the number of steps x 20 during the reloading stage between points B and C.</li> <li>The command to be run from the directory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale/2D_MultiScale">MultiScale/2D_MultiScale/</a> is</li> </ul> </li> </ul> <div> <pre><code>python3 model.py</code></pre> </div> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale/FE2">MultiScale/FE2</a>: <ul> <li>Contains the reference displacement-force results of the FE2 simulation.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale/FE2/Distributions">MultiScale/FE2/Distributions</a>: contains the macro-scale displacement and stress fields. They can be vizualized with gmsh (<a href="http://www.gmsh.info" target="_blank" rel="nofollow noreferrer noopener">www.gmsh.info</a>) using the mesh file <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/FE2/Distributions/model.msh">model.msh</a>.</li> </ul> </li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale">MultiScale/S<code>CELL</code>_120_<code>STEPS</code></a>: <ul> <li>Contain the reference displacement-force results of the rnn-based multiscale simulations for the different recurrent cells <code>CELL</code> and steps number <code>STEPS</code> during the reloading stage between points B and C.</li> <li>For the cases <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale">MultiScale/S<code>CELL</code>_120_20</a>, the macro-scale displacement and stress fields are also available and can be vizualized with gmsh (<a href="http://www.gmsh.info" target="_blank" rel="nofollow noreferrer noopener">www.gmsh.info</a>) using the mesh file <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/FE2/Distributions/model.msh">model.msh</a>.</li> </ul> </li> </ul> <h3>Vizualize mutiscale simulations results</h3> <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/2D_MultiScale/plot_force.py">MultiScale/2D_MultiScale/plot_force.py</a>: is used to plot the multiscale simulations results: <ul> <li><code>InpType = cell[0]</code> #choose the recurrent cell typeAmong the different available cells <code>cell=["SC_MRU_T","SC_MRU_I","SMRU"]</code>.</li> <li>The multiscale simulations results to be plotted are saved in the directories <code>CELL_120_Step</code>, where <code>Step</code> # is the number of steps during the reloading stage between points B and C.</li> <li>The command to be run from the directory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale/2D_MultiScale">MultiScale/2D_MultiScale/</a> is</li> </ul> </li> </ul> <div> <pre><code>python3 plot_force.py</code></pre> </div> <ul> <li>To vizualize the macro-scale displacement and stress fields distributions: <ul> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale/FE2/Distributions">MultiScale/FE2/Distributions</a>: contains the macro-scale displacement and stress fields. They can be vizualized with gmsh (<a href="http://www.gmsh.info" target="_blank" rel="nofollow noreferrer noopener">www.gmsh.info</a>) using the mesh file <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/FE2/Distributions/model.msh">model.msh</a>.</li> <li><a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale">MultiScale/S<code>CELL</code>_120_20</a>, the macro-scale displacement and stress fields are also available and can be vizualized with gmsh (<a href="http://www.gmsh.info" target="_blank" rel="nofollow noreferrer noopener">www.gmsh.info</a>) using the mesh file <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/blob/main/MultiScale/FE2/Distributions/model.msh">model.msh</a>.</li> </ul> </li> </ul> <h2>Part 4: Reproduce paper<sup><a href="#fn-1">1</a></sup> figures</h2> <ul> <li>Fig. 7: The command to be run from the diretory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/TrainingPaths">TrainingPaths</a> is</li> </ul> <div> <pre><code> python3 countPathLength.py '../ConstRVE/PathsExamples'</code></pre> </div> <ul> <li>Fig. 9: The command to be run from the diretory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU">SC_MRU/</a> is</li> </ul> <div> <pre><code> python3 PlotLoss_HiddenV.py</code></pre> </div> <ul> <li>Fig. 10: The command to be run from the diretory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU">SC_MRU/</a> is</li> </ul> <div> <pre><code> python3 PlotLoss_3MRU.py</code></pre> </div> <ul> <li>Fig. 11: The command to be run from the diretory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU">SC_MRU/</a> is</li> </ul> <div> <pre><code> python3 PlotLoss_DiffNL_H120.py</code></pre> </div> <ul> <li>Figs. 12 and 13: The command to be run from the directory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/CheckLoss">SC_MRU/CheckLoss/</a> is, see <a href="#vizualize-loss-evolution-and-testing-results">details</a></li> </ul> <div> <pre><code> python3 LossVS_insertN.py</code></pre> </div> <ul> <li>Figs. 14, 15 and 16: The command to be run from the directory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/CheckLoss">SC_MRU/CheckLoss/</a></li> </ul> <div> <pre><code> python3 Plot_GS.py</code></pre> </div> <ul> <li>Figs. 17(b)(c)(d): The command to be run from the directory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale/2D_MultiScale">MultiScale/2D_MultiScale/</a> is, see <a href="#vizualize-mutiscale-simulations-results">details</a>:</li> </ul> <div> <pre><code>python3 plot_force.py</code></pre> </div> <ul> <li>Figs. 18-23: Need gmsh to vizualize the results stored in <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale/FE2/Distributions">MultiScale/FE2/Distributions</a> and <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/MultiScale">MultiScale/S<code>CELL</code>_120_20</a>, see <a href="#vizualize-mutiscale-simulations-results">details</a></li> <li>Fig. A.24: The command to be run from the directory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/CheckLoss">SC_MRU/CheckLoss/</a> is, see <a href="#vizualize-loss-evolution-and-testing-results">details</a></li> </ul> <div> <pre><code> python3 PlotLoss_MGRU.py.py</code></pre> </div> <ul> <li>Fig. B.25: The command to be run from the directory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/CheckLoss">SC_MRU/CheckLoss/</a> is, see <a href="#vizualize-loss-evolution-and-testing-results">details</a></li> </ul> <div> <pre><code> python3 PlotLoss_NL.py</code></pre> </div> <ul> <li>Fig. B.26: The command to be run from the directory <a href="../didearot/didearotPublic/publicationsData/2024_scmru/-/tree/main/SC_MRU/CheckLoss">SC_MRU/CheckLoss/</a> is, see <a href="#vizualize-loss-evolution-and-testing-results">details</a></li> </ul> <div> <pre><code> python3 PlotLoss_DiffNL_H120.py</code></pre> </div> <h2>Disclaimer</h2> <p>This project has received funding from the European Union’s Horizon Europe Framework Programme under grant agreement No. 101056682 for the project “DIgital DEsign strategies to certify and mAnufacture Robust cOmposite sTructures (DIDEAROT)”. The contents of this publication are the sole responsibility of ULiege and do not necessarily reflect the opinion of the European Union. Neither the European Union nor the granting authority can be held responsible for them.</p> <ol> <li> <p>The work is described in:<br>"<em>Wu, L. and Noels, L. (2024).</em> <strong>Self-consistency Reinforced minimal Gated Recurrent Unit for surrogate modeling of history-dependent non-linear problems: application to history-dependent homogenized response of heterogeneous materials</strong> 424: 116881, <a href="https://doi.org/10.1016/j.cma.2024.116881" target="_blank" rel="nofollow noreferrer noopener">doi: 10.1016/j.cma.2024.116881</a>" which can be downloaded. We would be grateful if you could cite this publication in case you use the files. <a href="#fnref-1-5292">↩</a> <a href="#fnref-1-2">↩<sup>2</sup></a> <a href="#fnref-1-3">↩<sup>3</sup></a></p> </li> </ol>
Code and partial data used in "Vertically recurrent neural networks for sub-grid parameterization"
<p>This repository contains the RNN training and evaluation code used in the paper<em> Vertically recurrent neural networks for sub-grid parameterization</em></p> <p> </p> <ul> <li> The radiative transfer emulation data can be accessed with through a Climetlab plugin (<a href="https://pypi.org/project/climetlab-maelstrom-radiation/">Climetlab-maelstrom-radiation</a>). <p>Datasets are downloaded and explained in the demo notebook here <a href="https://git.ecmwf.int/projects/MLFET/repos/maelstrom-radiation/browse/notebooks/demo_radiation.ipynb" rel="nofollow">https://git.ecmwf.int/projects/MLFET/repos/maelstrom-radiation/browse/notebooks/demo_radiation.ipynb</a></p> In addition the full training and testing code used in the paper is uploaded here (<em>pu-maelstrom-radiation.tar.gz</em>).</li> <li> </li> </ul> <p>Three parameterization problems from earlier studies are also included (we have modified the code from these papers to incorporate RNNs): </p> <ul> <li>non-orographic gravity wave drag (<a href="https://doi.org/10.1029/2021MS002477">Chantry et al. 2021</a>) <ul> <li>Based on TensorFlow</li> <li>This repository uses the <em>CliMetLab </em>plugin and<strong> downloads the data from the European Weather Cloud</strong></li> </ul> </li> <li>non-local parameterization (<a href="https://doi.org/10.1029/2022MS002984">Wang et al. 2022</a>) <ul> <li>The new code is based on TensorFlow, so you'll need both PyTorch and TensorFlow to run everything</li> <li><strong>See original paper for data access</strong></li> </ul> </li> <li>moist physics (Han et al. <a href="https://doi.org/10.1029/2022MS003508">2023</a>, <a href="https://doi.org/10.1029/2020MS002076">2020</a>) <ul> <li>Based on TensorFlow and PyTorch. This one has the most additions, e.g. code to generate a TensorFlow TFRecord dataset from the raw netCDF data archived in the original paper, autoregressive training and experimental model architectures in PyTorch</li> <li><strong>See original paper for data access</strong></li> </ul> </li> </ul> <p>Each of the code repos (unpack the tars) have an updated README.</p> <p>References:</p> <table> <tbody> <tr> <td> <div>Chantry, M., Hatfield, S., Dueben, P., Polichtchouk, I., & Palmer, T. (2021). Machine learning emulation of gravity wave drag in numerical weather forecasting. <em>Journal of Advances in Modeling Earth Systems</em>, <em>13</em>(7), e2021MS002477</div> <div> </div> <div> <div>Han, Y., Zhang, G. J., Huang, X., & Wang, Y. (2020). A moist physics parameterization based on deep learning. <em>Journal of Advances in Modeling Earth Systems</em>, <em>12</em>(9), e2020MS002076.</div> </div> <div> </div> <div>Han, Y., Zhang, G. J., & Wang, Y. (2023). An ensemble of neural networks for moist physics processes, its generalizability and stable integration. <em>Journal of Advances in Modeling Earth Systems</em>, <em>15</em>(10), e2022MS003508</div> <div> </div> <div>Wang, P., Yuval, J., & O’Gorman, P. A. (2022). Non‐local parameterization of atmospheric subgrid processes with neural networks. <em>Journal of Advances in Modeling Earth Systems</em>, <em>14</em>(10), e2022MS002984.</div> </td> </tr> <tr></tr> </tbody> </table> <div> </div>
Long short-term memory (LSTM) recurrent neural network for muscle activity detection
<p><strong>Background: </strong>The accurate temporal analysis of muscle activation is of great interest in many research areas, spanning<br> from neurorobotic systems to the assessment of altered locomotion patterns in orthopedic and neurological<br> patients and the monitoring of their motor rehabilitation. The performance of the existing muscle activity detectors<br> is strongly affected by both the SNR of the surface electromyography (sEMG) signals and the set of features used to<br> detect the activation intervals. This work aims at introducing and validating a powerful approach to detect muscle<br> activation intervals from sEMG signals, based on long short-term memory (LSTM) recurrent neural networks.<br> </p> <p><strong>Methods: </strong>First, the applicability of the proposed LSTM-based muscle activity detector (LSTM-MAD) is studied<br> through simulated sEMG signals, comparing the LSTM-MAD performance against other two widely used approaches,<br> i.e., the standard approach based on Teager–Kaiser Energy Operator (TKEO) and the traditional approach, used in<br> clinical gait analysis, based on a double-threshold statistical detector (Stat). Second, the effect of the Signal-to-Noise<br> Ratio (SNR) on the performance of the LSTM-MAD is assessed considering simulated signals with nine different SNR<br> values. Finally, the newly introduced approach is validated on real sEMG signals, acquired during both physiological<br> and pathological gait. Electromyography recordings from a total of 20 subjects (8 healthy individuals, 6 orthopedic<br> patients, and 6 neurological patients) were included in the analysis.</p> <p><strong>Results</strong>: The proposed algorithm overcomes the main limitations of the other tested approaches and it works<br> directly on sEMG signals, without the need for background-noise and SNR estimation (as in Stat). Results demonstrate<br> that LSTM-MAD outperforms the other approaches, revealing higher values of F1-score (F1-score > 0.91) and Jaccard<br> similarity index (Jaccard > 0.85), and lower values of onset/offset bias (average absolute bias < 6 ms), both on simulated<br> and real sEMG signals. Moreover, the advantages of using the LSTM-MAD algorithm are particularly evident for<br> signals featuring a low to medium SNR.</p> <p><strong>Conclusions</strong>: The presented approach LSTM-MAD revealed excellent performances against TKEO and Stat. The<br> validation carried out both on simulated and real signals, considering normal as well as pathological motor function<br> during locomotion, demonstrated that it can be considered a powerful tool in the accurate and effective recognition/<br> distinction of muscle activity from background noise in sEMG signals.</p> <p> </p>
Radiation Protection Perspective to Recurrent Medical Radiological Imaging
<p>The topic of recurrent radiological imaging of patients attracted attention due to the recent studies indicating the magnitude of this phenomenon and the associated higher cumulated individual exposure to be more extensive than previously known. Recurrent imaging is used for managing various health conditions and chronic diseases such as malignancies, trauma, end-stage kidney disease, cardiovascular diseases, Crohn’s disease, urolithiasis, cystic pulmonary disease. The published studies, although available from only a part of the world, triggered discussion at international level, including two IAEA technical meetings with representatives of the IAEA Member states and international organization. The conclusions to date were reflected in the published Joint Position Statement and Call for Action by nine international organizations aimed to prompt greater dialogue and engage different stakeholders in developing and implementing strategies and solutions focused upon improved radiation protection of patients with medical conditions which prompt more frequent imaging procedures. Such actions include improved access to dose saving imaging technologies; improved imaging strategies and appropriateness process; specific optimization tailored to the clinical condition and patient habitus; wider utilization of the automatic exposure monitoring systems with an integrated option for individual exposure tracking in standardized patient-specific risk metrics; improved training and communication. These might need strengthening in the radiation protection framework to ensure that patients with medical conditions which prompt more frequent imaging procedures receive needed medical care, without undue exposure to ionizing radiation. Standardized and easily available dose information in patient-specific metrics is needed to improve risk quantification. Consensus is still lacking on the proper utilization of the dose information from the previous procedures, and the concern of misuse and misinterpretation, especially by referring physicians and patients, needs to be addressed. Like any other aspect of medical uses of ionizing radiation, the competence and awareness of users of dose information is paramount, and this is linked to the knowledge, education, training and communication. The integration of the clinical and exposure history data will support research studies and improved knowledge about radiation risks from low doses and individual radiosensitivity. The radiation protection framework will need to respond to the challenge of recurrent imaging and high individual doses. The radiation protection perspective complements the clinical perspective, and the risk to benefit analysis must account in holistic for all incidental and long-term benefits and risks for patients, their clinical history and specific needs. This is a step toward the patient-centric healthcare.</p>
A complete list of filtered variants and gene lists of frequently and recurrently mutated genes in DLBCL, MCL, T-NHL, and BL, as well as pre-assembled CNV gene list
<p>A complete list of variants which passed filtering described in supplemental methods, that were found in both PDX model sample and patient’s sample from which it was derived (S1A) and variants which were gained (S1B) or lost (S1C) during PDX model derivation. Gene lists for filtration of variants in genes of special interest are included in the table (S1D). Gene list for filtration of CNV changes in genes of special interest is included in the table (S1E). Chr - Chromosome, REF - Reference allele, ALT - Alternative allele, AA change - Amino acid change, Patient AF - Allele frequency in the patient’s sample, Patient Depth – Read depth in patient’s sample, PDX AF- Allele frequency in PDX model sample, PDX Depth - Read depth in PDX sample, DLBCL - Diffuse Large B-cell lymphoma, MCL - Mantle Cell Lymphoma, TCL - T-cell lymphoma, BL – Burkitt Lymphoma, and CNV - Copy Number Variation.</p>
Raw Data for the article: Clinical and Molecular-Based Approach in the Evaluation of Hepatocellular Carcinoma Recurrence after Radical Liver Resection
<p><strong>Background: </strong>Hepatic resection remains the treatment of choice for patients with early-stage HCC with preserved liver function. Unfortunately, however, the majority of patients develop tumor recurrence. While several clinical factors were found to be associated with tumor recurrence, HCC pathogenesis is a complex process of accumulation of somatic genomic alterations, which leads to a huge molecular heterogeneity that has not been completely understood. The aim of this study is to complement potentially predictive clinical and pathological factors with next-generation sequencing genomic profiling and loss of heterozygosity analysis.</p> <p><strong>Methods: </strong>124 HCC patients, who underwent a primary hepatic resection from January 2016 to December 2019, were recruited for this study. Next-generation sequencing (NGS) analysis and allelic imbalance assessment in a case-control subgroup analysis were performed. A time-to-recurrence analysis was performed as well by means of Kaplan-Meier estimators.</p> <p><strong>Results: </strong>Cumulative number of HCC recurrences were 26 (21%) and 32 (26%), respectively, one and two years after surgery. Kaplan-Meier estimates for the probability of recurrence amounted to 37% (95% C.I.: 24-47) and to 51% (95% C.I.: 35-62), after one and two years, respectively. Multivariable analysis identified as independent predictors of HCC recurrence: hepatitis C virus (HCV) infection (HR: 1.96, 95%C.I.: 0.91-4.24, <em>p</em> = 0.085), serum bilirubin levels (HR: 5.32, 95%C.I.: 2.07-13.69, <em>p</em> = 0.001), number of nodules (HR: 1.63, 95%C.I.: 1.12-2.38, <em>p</em> = 0.011) and size of the larger nodule (HR: 1.11, 95%C.I.: 1.03-1.18, <em>p</em> = 0.004). Time-to-recurrence analysis showed that loss of heterozygosity in the <em>PTEN</em> loci (involved in the PI3K/AKT/mTOR signaling pathway) was significantly associated with a lower risk of HCC recurrence (HR: 0.35, 95%C.I.: 0.13-0.93, <em>p</em> = 0.036).</p> <p><strong>Conclusions: </strong>multiple alterations of cancer genes are associated with HCC progression. In particular, the evidence of a specific AI mutation presented in 20 patients seemed to have a protective effect on the risk of HCC recurrence.</p>
Recurrence and metastasis detection of breast cancer
<p>This dataset includes 88 H&E stained whole slide images (WSI) of breast cancer downloaded from TCGA (<a href="https://portal.gdc.cancer.gov/repository/">https://portal.gdc.cancer.gov/repository/</a>) with the type of Formalin-Fixed Paraffin-Embedded (FFPE), of which 5 cases have recurrence or metastasis.</p>
Gut metabolites predict Clostridioides difficile recurrence
<p>This repository contains:</p> <p>1. Data used for analyses from:</p> <p>A) 16S rRNA gene amplicon analysis after paired-end Fastq files were truncated, filtered, denoised, and merged ["1-ASV_Counts.xlsx"]</p> <p>B) Untargeted metabolomics as provided by Metabolon ["2-UntargetedMetabolomics.xlsx"]</p> <p>C) Precision SCFA analysis ["3-PrecisionSCFA.xlsx"]</p> <p>D) Clincal demographic data of the study cohort ["Clinical Demographics.csv"]</p> <p> </p> <p>2. Detailed results of univariate analyses in sub-folder "Univariate analysis results" for clinical data, untargeted metabolomics data, SCFAs, and ASVs</p> <p> </p> <p>3. Detailed results of predictive analysis in subfolder "Predictive Analyses results", including per-fold performance results ("9-PredictiveResults.xlsx") and analysis of predictive features for ASVs ("10-ASVsPredictiveAnalysis.xlsx"), untargeted metabolites ("11-MetabsPredictiveAnalysis.xlsx"), targeted SCFAs ("12-SCFAPredictive Analysis.xlsx"), clinical data ("13-DemoPredictiveAnalysis.xlsx") and all data ("14-AllDataPredictiveAnalysis.xlsx")</p> <p>4. Data for generating all figures and analyses in the publication ("GenFigure&AnalysesData")</p>
Data for "Predicting aggregate morphology of sequence-defined macromolecules with Recurrent Neural Networks"
<p>These are the data associated with the paper, "Predicting aggregate morphology of sequence-defined macromolecules with Recurrent Neural Networks" (DOI 10.1039/D2SM00452F). Three of the directories contains subdirectories with `GSD` files dumped from HOOMD. The other contains pretrained RNN models as TorchScript binaries exported from PyTorch.</p>
Dataset - Generalization of deep recurrent optical flow estimation for particle-image velocimetry data
<p>This is the official test datasets of "Generalization of deep recurrent optical flow estimation for particle-image velocimetry data" published in Measurement Science and Technology. Particle-Image Velocimetry (PIV) is one of the key techniques in modern experimental fluid mechanics to determine the velocity components of flow fields in a wide range of complex engineering problems. Current PIV processing tools are mainly handcrafted models based on cross-correlations computed across interrogation windows. Although widely used, these existing tools have a number of well-known shortcomings, including limited spatial output resolution and peak-locking biases. Recently, new approaches for PIV processing leveraging a novel neural network architecture for optical flow estimation called Recurrent All-Pairs Field Transforms (RAFT) have been developed. These have matched or exceeded the performance of classical, handcrafted models. While the RAFT-PIV method is a promising approach, it is important for the broader fluids community to more completely understand its empirical behavior and performance. To this end, in this study, we thoroughly investigate the performance of RAFT-PIV under varying image and lighting conditions. IWe consider applications spanning synthetic and experimental data, with a breadth and depth going far beyond currently available empirical results. The results for the wide variation of experiments included in this dataset shed new light on the capabilities of deep learning for PIV processing. This dataset is given as binary TFRECORD format.</p>
Data archive for "Seamless lightning nowcasting with recurrent-convolutional deep learning"
<p>This dataset contains the machine learning training data files, pretrained model weights and precomputed results for the paper "Seamless lightning nowcasting with recurrent-convolutional deep learning" published in:<br> Leinonen, J., Hamann, U., & Germann, U. (2022). Seamless Lightning Nowcasting with Recurrent-Convolutional Deep Learning, <em>Artificial Intelligence for the Earth Systems</em>, <em>1</em>(4), e220043, doi:<a href="https://doi.org/10.1175/AIES-D-22-0043.1">10.1175/AIES-D-22-0043.1</a>.<br> A preprint of the paper can be found at <a href="https://arxiv.org/abs/2203.10114">https://arxiv.org/abs/2203.10114</a>.</p> <p>The ML code can be found at <a href="https://github.com/MeteoSwiss/c4dl-lightningdl">https://github.com/MeteoSwiss/c4dl-lightningdl</a>. Download all the files here and extract the contents to the following subdirectories in the ML code directory:</p> <ul> <li>Training data (c4dl-patches-*.zip) -> data/2020/</li> <li>Results (<a href="https://zenodo.org/api/files/d4829f50-55fd-4d86-b875-7f2b91dba74f/c4dl-results-lightningdl.zip?versionId=54046830-4c7e-48c6-af42-d6d5606af86b">c4dl-results-lightningdl.zip</a>) -> results/</li> <li>Pretrained models (<a href="https://zenodo.org/api/files/d4829f50-55fd-4d86-b875-7f2b91dba74f/c4dl-models-lightningdl.zip?versionId=364bca7c-e6ad-4ed9-9264-57c759ea0ac6">c4dl-models-lightningdl.zip</a>) -> models/</li> </ul> <p>Additionally, the file <a href="https://zenodo.org/api/files/d4829f50-55fd-4d86-b875-7f2b91dba74f/c4dl-randomexamples-lightningdl.zip?versionId=426ca113-950f-4a50-8ef0-8e5d12afe697">c4dl-randomexamples-lightningdl.zip</a> contains the randomly selected examples complementing Figs. 7–9 of the paper, and the file <a href="https://zenodo.org/api/files/939609f2-6699-4f56-9428-391ebe78e010/c4dl-inputsamples-lightningdl.zip">c4dl-inputsamples-lightningdl.zip</a> contains figures showing samples of all the input variables for the three cases shown in Figs. 7–9.</p>
Dataset from: "Uncertainty-Aware Interpretable Prognosis for Wave Energy Converters with Recurrent Expansion"
<p>This dataset comprises run-to-failure sensor data derived from a mathematical model simulating wave energy converter behavior, particularly for Oscillating Water Column Turbines (OWCTs). The dataset includes vibration, temperature, pressure, acceleration, strain, flow, torque, rotation, and remaining useful life (RUL) readings for one life cycle. Through normalization, the data is scaled uniformly for robust analysis and interpretation. Researchers can leverage this dataset to develop and validate their prognostic models for OWCTs.</p> <p>To cite this dataset, please refer to:</p> <p>Berghout Tarek and Benbouzid Mohamed. (2024). Uncertainty-Aware Interpretable Prognosis for Wave Energy Converters with Recurrent Expansion. SSRN, 1–22. <span><a href="https://dx.doi.org/10.2139/ssrn.4825408" target="_blank" rel="noopener"><span>http://dx.doi.org/10.2139/ssrn.4825408</span></a> </span></p>
MATLAB Codes for: Fault Diagnosis in Drones via Multiverse Augmented Extreme Recurrent Expansion of Acoustic Emissions with Uncertainty Bayesian Optimisation
<p>The following MATLAB codes belong to the paper following paper which has been publication in MDPI Machines. This repository includes all the necessary MATLAB scripts and functions used in the research for diagnosing faults in drones using advanced acoustic emission analysis and optimization techniques. The dataset used in this paper is referenced in the article. Please check the publication for the dataset reference. Download the dataset, decompress it, and place it in the same repository as these codes to ensure proper functionality. For any queries or further information, please refer to this paper.</p> <p>Berghout, Tarek, and Mohamed Benbouzid. 2024. "Fault Diagnosis in Drones via Multiverse Augmented Extreme Recurrent Expansion of Acoustic Emissions with Uncertainty Bayesian Optimisation" <em>Machines</em> 12, no. 8: 504. https://doi.org/10.3390/machines12080504 </p> <p> </p>
Medical Concept Normalization in Social Media Posts with Recurrent Neural Networks
<p>Text mining of scientific libraries and social media has already proven itself as a reliable tool for<br> drug repurposing and hypothesis generation. The task of mapping a disease mention to a concept<br> in a controlled vocabulary, typically to the standard thesaurus in the Unified Medical Language<br> System (UMLS), is known as medical concept normalization. This task is challenging due to the<br> differences in medical terminology between health care professionals and social media texts coming<br> from the lay public. To bridge this gap, we use sequence learning with recurrent neural networks<br> and semantic representation of one- or multi-word expressions: we develop end-to-end architectures<br> directly tailored to the task, including bidirectional Long Short-Term Memory and Gated Recurrent<br> Units with an attention mechanism and additional semantic similarity features based on UMLS.<br> Our evaluation over a standard benchmark shows that recurrent neural networks improve results<br> over an effective baseline for classification based on convolutional neural networks. A qualitative<br> examination of mentions discovered in a dataset of user reviews collected from popular online health<br> information platforms as well as quantitative evaluation both show improvements in the semantic<br> representation of health-related expressions in social media.</p>
Data from: Billeci et al. "Patient-specific seizure prediction based on heart rate variability and recurrence quantification analysis"
<p>Dataset of electrocardiogram and electroencephalogram signals (.edf) acquired in epileptic patients (N=15).</p> <p>All the patients were long-term monitored with a Video-EEG, with electrodes arranged on the basis of the international 10-20 system, and with ECG. ECG was measured simultaneously with a sampling rate of 512 Hz.</p> <p>Each data include a descriptor file (.txt) containing all the information related to the acquisition: data, registration start (time), registration end (time), seizure/s start, seizure/s end and the electrodes involved at the seizure onset.</p> <p> </p>
Hydrogen Burning on Accreting White Dwarfs: Stability, Recurrent Novae, and the Post-nova Supersoft Phase
<p>MESA inlists associated with <a href="https://ui.adsabs.harvard.edu/?#abs/2013ApJ...777..136W">Hydrogen Burning on Accreting White Dwarfs: Stability, Recurrent Novae, and the Post-nova Supersoft Phase</a></p>
Fig. 10 in Recurrent constructional pattern of the crustacean burrow Sinusichnus sinuosus from the Paleogene and Neogene of Spain
Fig. 10. Stratigraphic distribution of known fossil occurrences of the ichnogenus Sinusichnus.
Enhancement of green ghosts due to recurrence of sprite element
<p><span>These files contain the dataset supporting the manuscript titled "Enhancement of green ghosts due to recurrence of sprite element". The dataset includes the original images and videos of events.</span></p>
Data from: Recurrent sublethal warming reduces embryonic survival, inhibits juvenile growth, and alters species distribution projections under climate change
The capacity to tolerate climate change often varies across ontogeny in organisms with complex life cycles. Recently developed species distribution models incorporate traits across life stages; however, these life-cycle models primarily evaluate effects of lethal change. Here, we examine impacts of recurrent sublethal warming on development and survival in ecological projections of climate change. We reared lizard embryos in the laboratory under temperature cycles that simulated contemporary conditions and warming scenarios. We also artificially warmed natural nests to mimic laboratory treatments. In both cases, recurrent sublethal warming decreased embryonic survival and hatchling sizes. Incorporating survivorship results into a mechanistic species distribution model reduced annual survival by up to 24% compared to models that did not incorporate sublethal warming. Contrary to models without sublethal effects, our model suggests that modest increases in developmental temperatures influence species ranges due to effects on survivorship.
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.