Find research datasets worth reusing

Surrogate Modeling Benchmark - Damped Oscillator

This dataset is related to the damped oscillator benchmark case. A detailed description of the benchmark case can be found on the public online community website UQWorld: <a href="https://uqworld.org/t/benchmark-case-damped-oscillator/" target="_blank" rel="noopener">https://uqworld.org/t/benchmark-case-damped-oscillator/</a>. The experimental designs include datasets with 400, 800, 1200, 1600, and 2000 samples, each generated using optimized maximin distance Latin Hypercube Sampling (LHS) with 1000 iterations. Each dataset is replicated 20 times. The validation set contains 100,000 samples generated by Monte Carlo simulation. Each dataset contains input samples and the corresponding computational model responses. <h2>Description of the dataset file</h2> The dataset file includes two variables: <ul> <li>ExpDesigns, and</li> <li>ValidationSet.</li> </ul> Both variables are Matlab structures with fields X, Y, and nSamples. Variable ExpDesigns is a non-scalar structure sized according to the number of experimental design groups. Each field of X for the i-th element of the struct array contains replicated datasets, forming a matrix of size [number of samples] x [dimensionality] x [number of replications]. Similarly, each field of Y for the i-th element contains replicated computational model responses that correspond to the experimental design of the same replication, sized [number of samples] x [number of model outputs] x [number of replications]. The same structure logic applies to the ValidationSet variable, except it contains only one dataset per benchmark case. The structure can be summarized as follows: <ul> <li>ExpDesigns(i).X(j,k,l) <ul> <li>i: dataset group,</li> <li>j: sample index,</li> <li>k: variable index, and</li> <li>l: replication index.</li> </ul> </li> </ul> <ul> <li>ExpDesigns(i).Y(j,m,l) <ul> <li>i, j, l: same as above,</li> <li>m: computational model output index.</li> </ul> </li> </ul> <ul> <li>ValidationSet.X(j,k) <ul> <li>j, k: same as above.</li> </ul> </li> </ul> <ul> <li>ValidationSet.Y(j,m) <ul> <li>j, m: same as above.</li> </ul> </li> </ul> <h2>Description of benchmarked metamodel competitors</h2> The selection of competitors was based on our experience with meta-modeling and includes various metamodel types: Polynomial Chaos Expansions (PCE), Polynomial Chaos Kriging (PCK), and Kriging. Given that each metamodel has many hyperparameters, we chose the most general settings to address different benchmark case difficulties, including dimensionality, nonlinearity, and non-monotonicity. For Polynomial Chaos Expansions (PCE), we used a polynomial degree and q-norm adaptivity approach. This approach adaptively increases the maximum polynomial degree and truncation q-norm until the estimated leave-one-out error starts increasing. Maximum polynomial interaction terms were limited to 2 due to the memory requirements for large model dimensionality and large experimental designs. We tested three different solvers to calculate the PCE coefficients: Least Angle Regression (LARS), Orthogonal Matching Pursuit (OMP), and Subspace Pursuit (SP). Polynomial Chaos Kriging (PCK) employs a sequential combination strategy of PCE and Kriging. PCE uses degree adaptivity with a fixed q-norm. The maximum number of interactions is again set to 2 with the LARS solver. Ordinary Kriging is applied using the Matérn-5/2 correlation family, ellipsoidal, and anisotropic correlation function. We used a hybrid genetic algorithm to optimize the hyperparameters. We benchmarked both linear and ordinary Kriging, including Matérn-5/2 and Gaussian correlation families and separable and ellipsoidal correlation, resulting in eight different Kriging competitors. The hyperparameters were calculated using a hybrid covariance matrix adaptation-evolution strategy optimization. For further details on the settings, please refer to the competitors.m file and UQLab user manuals: <ul> <li>S. Marelli, N. Luethen, B. Sudret, <a href="https://www.uqlab.com/pce-user-manual">UQLab User Manual – Polynomial Chaos Expansions</a>, Report UQLab-V2.1-104, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.</li> <li>C. Lataniotis, D. Wicaksono, S. Marelli, B. Sudret, <a href="https://www.uqlab.com/kriging-user-manual">UQLab User Manual – Kriging (Gaussian Process Modeling)</a>, Report UQLab-V2.1-105, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.</li> <li>R. Schoebi, S. Marelli, B. Sudret, <a href="https://www.uqlab.com/pck-user-manual">UQLab User Manual – Polynomial Chaos Kriging</a>, Report UQLab-V2.0-109, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2022.</li> </ul> <h2>Description of the results file</h2> The results file contains one variable: Metrics. It is a Matlab structure with fields corresponding to each competitor (currently 12). Each competitor field contains data of type non-scalar struct array. The performance metrics included are RelMSE, RelRMSE, RelMAE, MAPE, Q2, and RelCVErr. Each field of Metrics.(CompetitorName) for the i-th element of the struct array contains metrics corresponding to the replicated dataset and the competitor, structured as follows: <ul> <li>Metrics.(CompetitorName)(i).(MetricName)(l) <ul> <li>i: dataset group,</li> <li>l: replication index.</li> </ul> </li> </ul> The description of the performance measures (metrics) can be found here: <a href="https://uqworld.org/t/metamodel-performance-measures/" target="_blank" rel="noopener">https://uqworld.org/t/metamodel-performance-measures/</a>. <h2>Additional files</h2> We provide files in three languages (MATLAB, Python, and Julia) to showcase how to work with datasets, results, and their visualization. The files are called working_with_datafiles.* (the extension depends on the selected language). <h2>Acknowledgment</h2> This project was supported by the Open Research Data Program of the ETH Board under Grant number EPFL SCR0902285. The calculations were run on the Euler cluster of ETH Zürich using the MATLAB-based UQLab software developed at the Chair of Risk, Safety and Uncertainty Quantification of ETH Zürich.

Surrogate Modeling Benchmark - Truss model

This dataset is related to the truss model benchmark case. A detailed description of the benchmark case can be found on the public online community website UQWorld: <a href="https://uqworld.org/t/benchmark-case-truss-model/" target="_blank" rel="noopener">https://uqworld.org/t/benchmark-case-truss-model/</a>. The experimental designs include datasets with 100, 200, 300, 400, and 500 samples, each generated using optimized maximin distance Latin Hypercube Sampling (LHS) with 1000 iterations. Each dataset is replicated 20 times. The validation set contains 100,000 samples generated by Monte Carlo simulation. Each dataset contains input samples and the corresponding computational model responses. <h2>Description of the dataset file</h2> The dataset file includes two variables: <ul> <li>ExpDesigns, and</li> <li>ValidationSet.</li> </ul> Both variables are Matlab structures with fields X, Y, and nSamples. Variable ExpDesigns is a non-scalar structure sized according to the number of experimental design groups. Each field of X for the i-th element of the struct array contains replicated datasets, forming a matrix of size [number of samples] x [dimensionality] x [number of replications]. Similarly, each field of Y for the i-th element contains replicated computational model responses that correspond to the experimental design of the same replication, sized [number of samples] x [number of model outputs] x [number of replications]. The same structure logic applies to the ValidationSet variable, except it contains only one dataset per benchmark case. The structure can be summarized as follows: <ul> <li>ExpDesigns(i).X(j,k,l) <ul> <li>i: dataset group,</li> <li>j: sample index,</li> <li>k: variable index, and</li> <li>l: replication index.</li> </ul> </li> </ul> <ul> <li>ExpDesigns(i).Y(j,m,l) <ul> <li>i, j, l: same as above,</li> <li>m: computational model output index.</li> </ul> </li> </ul> <ul> <li>ValidationSet.X(j,k) <ul> <li>j, k: same as above.</li> </ul> </li> </ul> <ul> <li>ValidationSet.Y(j,m) <ul> <li>j, m: same as above.</li> </ul> </li> </ul> <h2>Description of benchmarked metamodel competitors</h2> The selection of competitors was based on our experience with meta-modeling and includes various metamodel types: Polynomial Chaos Expansions (PCE), Polynomial Chaos Kriging (PCK), and Kriging. Given that each metamodel has many hyperparameters, we chose the most general settings to address different benchmark case difficulties, including dimensionality, nonlinearity, and non-monotonicity. For Polynomial Chaos Expansions (PCE), we used a polynomial degree and q-norm adaptivity approach. This approach adaptively increases the maximum polynomial degree and truncation q-norm until the estimated leave-one-out error starts increasing. Maximum polynomial interaction terms were limited to 2 due to the memory requirements for large model dimensionality and large experimental designs. We tested three different solvers to calculate the PCE coefficients: Least Angle Regression (LARS), Orthogonal Matching Pursuit (OMP), and Subspace Pursuit (SP). Polynomial Chaos Kriging (PCK) employs a sequential combination strategy of PCE and Kriging. PCE uses degree adaptivity with a fixed q-norm. The maximum number of interactions is again set to 2 with the LARS solver. Ordinary Kriging is applied using the Matérn-5/2 correlation family, ellipsoidal, and anisotropic correlation function. We used a hybrid genetic algorithm to optimize the hyperparameters. We benchmarked both linear and ordinary Kriging, including Matérn-5/2 and Gaussian correlation families and separable and ellipsoidal correlation, resulting in eight different Kriging competitors. The hyperparameters were calculated using a hybrid covariance matrix adaptation-evolution strategy optimization. For further details on the settings, please refer to the competitors.m file and UQLab user manuals: <ul> <li>S. Marelli, N. Luethen, B. Sudret, <a href="https://www.uqlab.com/pce-user-manual">UQLab User Manual – Polynomial Chaos Expansions</a>, Report UQLab-V2.1-104, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.</li> <li>C. Lataniotis, D. Wicaksono, S. Marelli, B. Sudret, <a href="https://www.uqlab.com/kriging-user-manual">UQLab User Manual – Kriging (Gaussian Process Modeling)</a>, Report UQLab-V2.1-105, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.</li> <li>R. Schoebi, S. Marelli, B. Sudret, <a href="https://www.uqlab.com/pck-user-manual">UQLab User Manual – Polynomial Chaos Kriging</a>, Report UQLab-V2.0-109, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2022.</li> </ul> <h2>Description of the results file</h2> The results file contains one variable: Metrics. It is a Matlab structure with fields corresponding to each competitor (currently 12). Each competitor field contains data of type non-scalar struct array. The performance metrics included are RelMSE, RelRMSE, RelMAE, MAPE, Q2, and RelCVErr. Each field of Metrics.(CompetitorName) for the i-th element of the struct array contains metrics corresponding to the replicated dataset and the competitor, structured as follows: <ul> <li>Metrics.(CompetitorName)(i).(MetricName)(l) <ul> <li>i: dataset group,</li> <li>l: replication index.</li> </ul> </li> </ul> The description of the performance measures (metrics) can be found here: <a href="https://uqworld.org/t/metamodel-performance-measures/" target="_blank" rel="noopener">https://uqworld.org/t/metamodel-performance-measures/</a>. <h2>Additional files</h2> We provide files in three languages (MATLAB, Python, and Julia) to showcase how to work with datasets, results, and their visualization. The files are called working_with_datafiles.* (the extension depends on the selected language). <h2>Acknowledgment</h2> This project was supported by the Open Research Data Program of the ETH Board under Grant number EPFL SCR0902285. The calculations were run on the Euler cluster of ETH Zürich using the MATLAB-based UQLab software developed at the Chair of Risk, Safety and Uncertainty Quantification of ETH Zürich.

Surrogate Modeling Benchmark - Wing weight function

This dataset is related to the wing weight function benchmark case. A detailed description of the benchmark case can be found on the public online community website UQWorld: <a href="https://uqworld.org/t/benchmark-case-wing-weight-function/" target="_blank" rel="noopener">https://uqworld.org/t/benchmark-case-wing-weight-function/</a>. The experimental designs include datasets with 100, 200, 300, 400, and 500 samples, each generated using optimized maximin distance Latin Hypercube Sampling (LHS) with 1000 iterations. Each dataset is replicated 20 times. The validation set contains 100,000 samples generated by Monte Carlo simulation. Each dataset contains input samples and the corresponding computational model responses. <h2>Description of the dataset file</h2> The dataset file includes two variables: <ul> <li>ExpDesigns, and</li> <li>ValidationSet.</li> </ul> Both variables are Matlab structures with fields X, Y, and nSamples. Variable ExpDesigns is a non-scalar structure sized according to the number of experimental design groups. Each field of X for the i-th element of the struct array contains replicated datasets, forming a matrix of size [number of samples] x [dimensionality] x [number of replications]. Similarly, each field of Y for the i-th element contains replicated computational model responses that correspond to the experimental design of the same replication, sized [number of samples] x [number of model outputs] x [number of replications]. The same structure logic applies to the ValidationSet variable, except it contains only one dataset per benchmark case. The structure can be summarized as follows: <ul> <li>ExpDesigns(i).X(j,k,l) <ul> <li>i: dataset group,</li> <li>j: sample index,</li> <li>k: variable index, and</li> <li>l: replication index.</li> </ul> </li> </ul> <ul> <li>ExpDesigns(i).Y(j,m,l) <ul> <li>i, j, l: same as above,</li> <li>m: computational model output index.</li> </ul> </li> </ul> <ul> <li>ValidationSet.X(j,k) <ul> <li>j, k: same as above.</li> </ul> </li> </ul> <ul> <li>ValidationSet.Y(j,m) <ul> <li>j, m: same as above.</li> </ul> </li> </ul> <h2>Description of benchmarked metamodel competitors</h2> The selection of competitors was based on our experience with meta-modeling and includes various metamodel types: Polynomial Chaos Expansions (PCE), Polynomial Chaos Kriging (PCK), and Kriging. Given that each metamodel has many hyperparameters, we chose the most general settings to address different benchmark case difficulties, including dimensionality, nonlinearity, and non-monotonicity. For Polynomial Chaos Expansions (PCE), we used a polynomial degree and q-norm adaptivity approach. This approach adaptively increases the maximum polynomial degree and truncation q-norm until the estimated leave-one-out error starts increasing. Maximum polynomial interaction terms were limited to 2 due to the memory requirements for large model dimensionality and large experimental designs. We tested three different solvers to calculate the PCE coefficients: Least Angle Regression (LARS), Orthogonal Matching Pursuit (OMP), and Subspace Pursuit (SP). Polynomial Chaos Kriging (PCK) employs a sequential combination strategy of PCE and Kriging. PCE uses degree adaptivity with a fixed q-norm. The maximum number of interactions is again set to 2 with the LARS solver. Ordinary Kriging is applied using the Matérn-5/2 correlation family, ellipsoidal, and anisotropic correlation function. We used a hybrid genetic algorithm to optimize the hyperparameters. We benchmarked both linear and ordinary Kriging, including Matérn-5/2 and Gaussian correlation families and separable and ellipsoidal correlation, resulting in eight different Kriging competitors. The hyperparameters were calculated using a hybrid covariance matrix adaptation-evolution strategy optimization. For further details on the settings, please refer to the competitors.m file and UQLab user manuals: <ul> <li>S. Marelli, N. Luethen, B. Sudret, <a href="https://www.uqlab.com/pce-user-manual">UQLab User Manual – Polynomial Chaos Expansions</a>, Report UQLab-V2.1-104, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.</li> <li>C. Lataniotis, D. Wicaksono, S. Marelli, B. Sudret, <a href="https://www.uqlab.com/kriging-user-manual">UQLab User Manual – Kriging (Gaussian Process Modeling)</a>, Report UQLab-V2.1-105, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.</li> <li>R. Schoebi, S. Marelli, B. Sudret, <a href="https://www.uqlab.com/pck-user-manual">UQLab User Manual – Polynomial Chaos Kriging</a>, Report UQLab-V2.0-109, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2022.</li> </ul> <h2>Description of the results file</h2> The results file contains one variable: Metrics. It is a Matlab structure with fields corresponding to each competitor (currently 12). Each competitor field contains data of type non-scalar struct array. The performance metrics included are RelMSE, RelRMSE, RelMAE, MAPE, Q2, and RelCVErr. Each field of Metrics.(CompetitorName) for the i-th element of the struct array contains metrics corresponding to the replicated dataset and the competitor, structured as follows: <ul> <li>Metrics.(CompetitorName)(i).(MetricName)(l) <ul> <li>i: dataset group,</li> <li>l: replication index.</li> </ul> </li> </ul> The description of the performance measures (metrics) can be found here: <a href="https://uqworld.org/t/metamodel-performance-measures/" target="_blank" rel="noopener">https://uqworld.org/t/metamodel-performance-measures/</a>. <h2>Additional files</h2> We provide files in three languages (MATLAB, Python, and Julia) to showcase how to work with datasets, results, and their visualization. The files are called working_with_datafiles.* (the extension depends on the selected language). <h2>Acknowledgment</h2> This project was supported by the Open Research Data Program of the ETH Board under Grant number EPFL SCR0902285. The calculations were run on the Euler cluster of ETH Zürich using the MATLAB-based UQLab software developed at the Chair of Risk, Safety and Uncertainty Quantification of ETH Zürich.

Surrogate Modeling Benchmark - Borehole function

This dataset is related to the Borehole function benchmark case. A detailed description of the benchmark case can be found on the public online community website UQWorld: <a href="https://uqworld.org/t/benchmark-case-borehole-function/">https://uqworld.org/t/benchmark-case-borehole-function/</a>. The experimental designs include datasets with 40, 80, 120, 160, and 200 samples, each generated using optimized maximin distance Latin Hypercube Sampling (LHS) with 1000 iterations. Each dataset is replicated 20 times. The validation set contains 100,000 samples generated by Monte Carlo simulation. Each dataset contains input samples and the corresponding computational model responses. <h2>Description of the dataset file</h2> The dataset file includes two variables: <ul> <li>ExpDesigns, and</li> <li>ValidationSet.</li> </ul> Both variables are Matlab structures with fields X, Y, and nSamples. Variable ExpDesigns is a non-scalar structure sized according to the number of experimental design groups. Each field of X for the i-th element of the struct array contains replicated datasets, forming a matrix of size [number of samples] x [dimensionality] x [number of replications]. Similarly, each field of Y for the i-th element contains replicated computational model responses that correspond to the experimental design of the same replication, sized [number of samples] x [number of model outputs] x [number of replications]. The same structure logic applies to the ValidationSet variable, except it contains only one dataset per benchmark case. The structure can be summarized as follows: <ul> <li>ExpDesigns(i).X(j,k,l) <ul> <li>i: dataset group,</li> <li>j: sample index,</li> <li>k: variable index, and</li> <li>l: replication index.</li> </ul> </li> </ul> <ul> <li>ExpDesigns(i).Y(j,m,l) <ul> <li>i, j, l: same as above,</li> <li>m: computational model output index.</li> </ul> </li> </ul> <ul> <li>ValidationSet.X(j,k) <ul> <li>j, k: same as above.</li> </ul> </li> </ul> <ul> <li>ValidationSet.Y(j,m) <ul> <li>j, m: same as above.</li> </ul> </li> </ul> <h2>Description of benchmarked metamodel competitors</h2> The selection of competitors was based on our experience with meta-modeling and includes various metamodel types: Polynomial Chaos Expansions (PCE), Polynomial Chaos Kriging (PCK), and Kriging. Given that each metamodel has many hyperparameters, we chose the most general settings to address different benchmark case difficulties, including dimensionality, nonlinearity, and non-monotonicity. For Polynomial Chaos Expansions (PCE), we used a polynomial degree and q-norm adaptivity approach. This approach adaptively increases the maximum polynomial degree and truncation q-norm until the estimated leave-one-out error starts increasing. Maximum polynomial interaction terms were limited to 2 due to the memory requirements for large model dimensionality and large experimental designs. We tested three different solvers to calculate the PCE coefficients: Least Angle Regression (LARS), Orthogonal Matching Pursuit (OMP), and Subspace Pursuit (SP). Polynomial Chaos Kriging (PCK) employs a sequential combination strategy of PCE and Kriging. PCE uses degree adaptivity with a fixed q-norm. The maximum number of interactions is again set to 2 with the LARS solver. Ordinary Kriging is applied using the Matérn-5/2 correlation family, ellipsoidal, and anisotropic correlation function. We used a hybrid genetic algorithm to optimize the hyperparameters. We benchmarked both linear and ordinary Kriging, including Matérn-5/2 and Gaussian correlation families and separable and ellipsoidal correlation, resulting in eight different Kriging competitors. The hyperparameters were calculated using a hybrid covariance matrix adaptation-evolution strategy optimization. For further details on the settings, please refer to the competitors.m file and UQLab user manuals: <ul> <li>S. Marelli, N. Luethen, B. Sudret, <a href="https://www.uqlab.com/pce-user-manual">UQLab User Manual – Polynomial Chaos Expansions</a>, Report UQLab-V2.1-104, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.</li> <li>C. Lataniotis, D. Wicaksono, S. Marelli, B. Sudret, <a href="https://www.uqlab.com/kriging-user-manual">UQLab User Manual – Kriging (Gaussian Process Modeling)</a>, Report UQLab-V2.1-105, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.</li> <li>R. Schoebi, S. Marelli, B. Sudret, <a href="https://www.uqlab.com/pck-user-manual">UQLab User Manual – Polynomial Chaos Kriging</a>, Report UQLab-V2.0-109, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2022.</li> </ul> <h2>Description of the results file</h2> The results file contains one variable: Metrics. It is a Matlab structure with fields corresponding to each competitor (currently 12). Each competitor field contains data of type non-scalar struct array. The performance metrics included are RelMSE, RelRMSE, RelMAE, MAPE, Q2, and RelCVErr. Each field of Metrics.(CompetitorName) for the i-th element of the struct array contains metrics corresponding to the replicated dataset and the competitor, structured as follows: <ul> <li>Metrics.(CompetitorName)(i).(MetricName)(l) <ul> <li>i: dataset group,</li> <li>l: replication index.</li> </ul> </li> </ul> The description of the performance measures (metrics) can be found here: <a href="https://uqworld.org/t/metamodel-performance-measures/" target="_blank" rel="noopener">https://uqworld.org/t/metamodel-performance-measures/</a>. <h2>Additional files</h2> We provide files in three languages (MATLAB, Python, and Julia) to showcase how to work with datasets, results, and their visualization. The files are called working_with_datafiles.* (the extension depends on the selected language). <h2>Acknowledgment</h2> This project was supported by the Open Research Data Program of the ETH Board under Grant number EPFL SCR0902285. The calculations were run on the Euler cluster of ETH Zürich using the MATLAB-based UQLab software developed at the Chair of Risk, Safety and Uncertainty Quantification of ETH Zürich.

Surrogate Modeling Benchmark - Undamped Oscillator

This dataset is related to the undamped oscillator benchmark case. A detailed description of the benchmark case can be found on the public online community website UQWorld: <a href="https://uqworld.org/t/benchmark-case-undamped-oscillator/" target="_blank" rel="noopener">https://uqworld.org/t/benchmark-case-undamped-oscillator/</a>. The experimental designs include datasets with 40, 80, 120, 160, and 200 samples, each generated using optimized maximin distance Latin Hypercube Sampling (LHS) with 1000 iterations. Each dataset is replicated 20 times. The validation set contains 100,000 samples generated by Monte Carlo simulation. Each dataset contains input samples and the corresponding computational model responses. <h2>Description of the dataset file</h2> The dataset file includes two variables: <ul> <li>ExpDesigns, and</li> <li>ValidationSet.</li> </ul> Both variables are Matlab structures with fields X, Y, and nSamples. Variable ExpDesigns is a non-scalar structure sized according to the number of experimental design groups. Each field of X for the i-th element of the struct array contains replicated datasets, forming a matrix of size [number of samples] x [dimensionality] x [number of replications]. Similarly, each field of Y for the i-th element contains replicated computational model responses that correspond to the experimental design of the same replication, sized [number of samples] x [number of model outputs] x [number of replications]. The same structure logic applies to the ValidationSet variable, except it contains only one dataset per benchmark case. The structure can be summarized as follows: <ul> <li>ExpDesigns(i).X(j,k,l) <ul> <li>i: dataset group,</li> <li>j: sample index,</li> <li>k: variable index, and</li> <li>l: replication index.</li> </ul> </li> </ul> <ul> <li>ExpDesigns(i).Y(j,m,l) <ul> <li>i, j, l: same as above,</li> <li>m: computational model output index.</li> </ul> </li> </ul> <ul> <li>ValidationSet.X(j,k) <ul> <li>j, k: same as above.</li> </ul> </li> </ul> <ul> <li>ValidationSet.Y(j,m) <ul> <li>j, m: same as above.</li> </ul> </li> </ul> <h2>Description of benchmarked metamodel competitors</h2> The selection of competitors was based on our experience with meta-modeling and includes various metamodel types: Polynomial Chaos Expansions (PCE), Polynomial Chaos Kriging (PCK), and Kriging. Given that each metamodel has many hyperparameters, we chose the most general settings to address different benchmark case difficulties, including dimensionality, nonlinearity, and non-monotonicity. For Polynomial Chaos Expansions (PCE), we used a polynomial degree and q-norm adaptivity approach. This approach adaptively increases the maximum polynomial degree and truncation q-norm until the estimated leave-one-out error starts increasing. Maximum polynomial interaction terms were limited to 2 due to the memory requirements for large model dimensionality and large experimental designs. We tested three different solvers to calculate the PCE coefficients: Least Angle Regression (LARS), Orthogonal Matching Pursuit (OMP), and Subspace Pursuit (SP). Polynomial Chaos Kriging (PCK) employs a sequential combination strategy of PCE and Kriging. PCE uses degree adaptivity with a fixed q-norm. The maximum number of interactions is again set to 2 with the LARS solver. Ordinary Kriging is applied using the Matérn-5/2 correlation family, ellipsoidal, and anisotropic correlation function. We used a hybrid genetic algorithm to optimize the hyperparameters. We benchmarked both linear and ordinary Kriging, including Matérn-5/2 and Gaussian correlation families and separable and ellipsoidal correlation, resulting in eight different Kriging competitors. The hyperparameters were calculated using a hybrid covariance matrix adaptation-evolution strategy optimization. For further details on the settings, please refer to the competitors.m file and UQLab user manuals: <ul> <li>S. Marelli, N. Luethen, B. Sudret, <a href="https://www.uqlab.com/pce-user-manual">UQLab User Manual – Polynomial Chaos Expansions</a>, Report UQLab-V2.1-104, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.</li> <li>C. Lataniotis, D. Wicaksono, S. Marelli, B. Sudret, <a href="https://www.uqlab.com/kriging-user-manual">UQLab User Manual – Kriging (Gaussian Process Modeling)</a>, Report UQLab-V2.1-105, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2024.</li> <li>R. Schoebi, S. Marelli, B. Sudret, <a href="https://www.uqlab.com/pck-user-manual">UQLab User Manual – Polynomial Chaos Kriging</a>, Report UQLab-V2.0-109, Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich, Switzerland, 2022.</li> </ul> <h2>Description of the results file</h2> The results file contains one variable: Metrics. It is a Matlab structure with fields corresponding to each competitor (currently 12). Each competitor field contains data of type non-scalar struct array. The performance metrics included are RelMSE, RelRMSE, RelMAE, MAPE, Q2, and RelCVErr. Each field of Metrics.(CompetitorName) for the i-th element of the struct array contains metrics corresponding to the replicated dataset and the competitor, structured as follows: <ul> <li>Metrics.(CompetitorName)(i).(MetricName)(l) <ul> <li>i: dataset group,</li> <li>l: replication index.</li> </ul> </li> </ul> The description of the performance measures (metrics) can be found here: <a href="https://uqworld.org/t/metamodel-performance-measures/" target="_blank" rel="noopener">https://uqworld.org/t/metamodel-performance-measures/</a>. <h2>Additional files</h2> We provide files in three languages (MATLAB, Python, and Julia) to showcase how to work with datasets, results, and their visualization. The files are called working_with_datafiles.* (the extension depends on the selected language). <h2>Acknowledgment</h2> This project was supported by the Open Research Data Program of the ETH Board under Grant number EPFL SCR0902285. The calculations were run on the Euler cluster of ETH Zürich using the MATLAB-based UQLab software developed at the Chair of Risk, Safety and Uncertainty Quantification of ETH Zürich.

Residual stress in 316L stainless steel benchmark additively manufactured arches determined by neutron diffraction and snychtron X-ray diffraction

Residual stress data recorded as part of the EASI-STRESS project.  The data presented is the residual stress in three orthogonal directions determined by neutron diffraction (SALSA at ILL) and synchrotron X-ray diffraction (P07 and P61A at Desy operated by Hereon and ID15A at ESRF). The data is for residual stress in a benchmark 316L stainless steel arch manufactured by laser powder bed fusion using a MetalFAB1 additive manufacturing machine. The arch is square topped with dimensions of nominally 20 mm in both the x and y direction (in plane). The overhang which creates the arch shape runs parallel to the y direction. The height of the arch is nominally 10 mm with the ligament above the over hang being of 2 mm nominal thickness.  Data for two lines is presented: line 1 runs from the centre of the top surface (defined at the origin) down into the arch (defined as positive z direction). Line 2 runs along to the x axis at a depth of 1 mm into the arch.  The stress-free reference used was a reference comb cut from an identically made arch. Different gauge volume sizes and shapes were defined at each institution, all measurement locations indicate the centre of the gauge volume during each measurement. gauge volumes were: P07 200 x 200 µm2, ~1.9 mm, P61A 150 × 150 µm2 ~2.9 mm, ID15A 200 ´ 50 µm2 ~1.7 mm, SALSA 0.6 x 0.6 x2 mm3.

As built and post heat treatment residual stresses in 316L stainless steel additively manufactured benchmark arches

The data presented is the residual stress in three orthogonal directions determined by neutron diffraction (SALSA at ILL) and synchrotron X-ray diffraction (P07 and P61A at Desy operated by Hereon). The data is for residual stress in a benchmark 316L stainless steel arches manufactured by laser powder bed fusion using MetalFAB1 and EOS M290 additive manufacturing machines. These arches are square topped with dimensions of nominally 20 mm in both the x and y direction (in plane). The overhang which creates the arch shape runs parallel to the y direction. The height of the arch is nominally 10 mm with the ligament above the over hang being of 2 mm nominal thickness.  Data for part in both an as built and after a 700 °C 2 hour heat treatment are presented. The measurement  line runs from the centre of the top surface (defined at the origin) down into the arch (defined as positive z direction). The stress-free reference used was a reference comb cut from an identically made arches. For heat treated samples, heat treated combs were used. Different gauge volume sizes and shapes were defined at each institution, all measurement locations indicate the centre of the gauge volume during each measurement. gauge volumes were: P07 200 x 200 µm2, ~1.9 mm, P61A 150 × 150 µm2 ~2.9 mm, ID15A 200 x 50 µm2 ~1.7 mm, SALSA 0.6 x 0.6 x2 mm3.

opencc-by-4.0Nov 2023View details →

Aberrant gene expression prediction benchmark based on GTEx v8

This repository contains the aberrant gene expression prediction benchmark data as well as the necessary expected gene expression across tissues and tissue-specific isoform contribution scores for AbExp prediction.   The aberrant gene expression prediction benchmark data (aberrant_expression_prediction_benchmark.parquet) contains the following columns: <ul> <li>individual: GTEx individual</li> <li>gene: Ensembl gene identifier</li> <li>tissue: GTEx tissue</li> <li>tissue_type: GTEx tissue type</li> <li>mu: OUTRIDER-estimated expected gene expression</li> <li>theta: OUTRIDER-estimated gene dispersion</li> <li>counts: Raw gene expression count</li> <li>normalized_counts: OUTRIDER-normalized gene expression count</li> <li>l2fc: log2 fold change between observed and expected gene expression count</li> <li>zscore: z-score of gene expression, obtained by quantile-mapping the OUTRIDER-estimated distribution to the standard normal distribution</li> <li>nominal_pvalue: OUTRIDER-estimated p-value of being an expression outlier</li> <li>FDR: FDR-adjusted p-value of being an expression outlier</li> <li>is_in_benchmark: Whether this observation is part of the aberrant gene expression prediction benchmark</li> <li>is_underexpressed_outlier: Whether this observation is an underexpression outlier at FDR < 5%. This is the benchmark prediction label.</li> </ul> The isoform proportions table (gtex_v8_isoform_proportions.tsv) contains the following columns: <ul> <li>gene: Ensembl gene identifier</li> <li>tissue_type: GTEx tissue type</li> <li>tissue: GTEx tissue</li> <li>transcript: Ensembl transcript identifier</li> <li>mean_transcript_proportions: mean transcript proportions across individuals in GTEx v8</li> <li>median_transcript_proportions: median transcript proportions across individuals in GTEx v8</li> <li>sd_transcript_proportions: standard deviation of transcript proportions across individuals in GTEx v8</li> </ul> The expected gene expression table (gtex_v8_expected_expression.tsv) contains the following columns: <ul> <li>gene: Ensembl gene identifier</li> <li>tissue_type: GTEx tissue type</li> <li>tissue: GTEx tissue</li> <li>gene_is_expressed: Whether the gene is expressed in the tissue</li> <li>median_expression: median OUTRIDER-estimated expected gene expression (mu) across individuals</li> <li>expression_dispersion: OUTRIDER-estimated gene dispersion (theta)</li> </ul>

opencc-by-4.0Nov 2023View details →

Real-world Optimization Benchmark from Vehicle Dynamics - Data and Code

Data and Code of five 2D single-objective optimization problems from vehicle dynamics design for benchmarking Conference Paper at ECTA Real-world Optimization Benchmark from Vehicle Dynamics: Specification of Problems in 2D and Methodology for Transferring (Meta-)Optimized Algorithm Parameters

RSYD-BASIC results for AMR benchmarking dataset subset (MiSeq data)

Input data: <ul> <li>20240905_test_config.yaml: original config file used to run the pipeline</li> <li>20241004_rsyd_largeset_reads.zip: renamed Illumina MiSeq reads</li> <li>20241011-sample-overview.xlsx: overview of SRR accession numbers to internal sample numbers</li> <li>ILM_Run0001_Y20240904_kts_new.xlsx: runsheet </li> <li>input_en.yaml: column name configuration for the run</li> <li>lis_data.zip: LIS report and bacteria list used for LIS-specific results</li> </ul> Expected results: <ul> <li>20240910_test_illumina_largeset.zip: Results of the RSYD-BASIC pipeline, version 1.15.1, with the reads used</li> </ul>

opencc-by-4.0May 2024View details →

Loading Stability Benchmark for Pallet Loading Problems

<h3>Description</h3> The dataset contains information about approx. 32.000 pallet loading problem (PLP) cargo layouts derived from a physical simulation with the MSC ADAMS software. The question of the study is to compare and benchmark multiple static stability algorithms: full base support, partial base support, static mechanical equilibrium, and physical simulation. As there are no real-world cargo loading datasets available, we decided to approximate real-world loading. We used a multibody simulator (MSC ADAMS) and simulated cargo loadings for a set of layouts. The benchmark simulation was adjusted in an iterative process, in which we visually tested simulations and incorporated insights from literature about physical cargo properties until we were satisfied with the plausibility of the results. However, the benchmark simulation depends on many parameters that need to be adjusted, and our adjustments might be unprecise. The dataset contains 2 sub-datasets: Dataset 1 is based on the <a href="https://github.com/fbrandt/ACLPP" target="_blank" rel="noopener">ACLPP instances</a> generated by Brandt & Nickel (2019). Dataset 2 is based on the <a href="https://www.sciencedirect.com/science/article/pii/S1568494623011869" target="_blank" rel="noopener">instances</a> from Ali et al. (2024). For every sub-dataset, we assembled three complexity scenarios with four stages. Scenario 1 is the easiest scenario, which assumes all items (boxes) have uniform density and no displaced center of mass (in relation to their geometric center). Scenario 2a moves the CoM now to a random position in the items' dimensions according to a Gaussian distribution around the item's geometric center. Scenario 2b now applies the same procedure as Scenario 2a but employs a uniform distribution. <h3>Stages</h3> Stage 0 ("0_input") are the raw data from both datasets. We did not include the raw data but provided the link to the dataset in the meta-information here. Stage 1 ("1_AeULDs") are the input cargo items with itemLabel, weight, (box)-shape with width, height, depth, loading coordinate (x,y,z), sequence, and center of mass (x, y, z). We imposed a cap on the number of items of 20. If a ULD exceeds this threshold, we include only the first 20 items. Stage 2 ("2_AeJobs") transfers the ULDs to an executable format and includes assessment information (i.e., stability approaches). Stage 3 ("3_AeResults") are the results from the different static stability algorithms with a uniform static stability score between 0 (first item unstable) and 1 (all items stable) and a runtime.  Stage 4 ("4_Benchmark") are the benchmark data from our multibody simulation. The first folder ("done_raw") is raw output of our ADAMS simulation, which tracks relevant physical characteristics such as angular momentum, angular velocity, acceleration, position, velocity (about the center of mass), translation, rotation, and contact forces with other items. We measured multiple observations per loading sequence, which all have an assigned time step. The total simulation length is 0.3 s. The second folder ("done_intermediate") now filters the raw data, such that we track the largest translation and rotation in x, y, and z- directions per loading step per item. We also calculate the maximal translation and rotation within the simulation. The third folder ("done") transforms the intermediate steps into a final quantified stability outcome, in case any item exceeds the threshold values for translation and rotation. The final outcome is normalized, such that 1 represents a stable cargo layout and 0 represents a layout in which the first item is unstable. Divide the number of stable loading steps by the total number of items in the cargo layout. The fourth folder ("done_sensitivity_analysis") computes the benchmark results for different epsilon_translations and epsilon_rotation values for sensitivity analyses. We filtered the data for the analysis, such that only ULDs with a minimal width, depth, and height of every item are included for the final analysis. Further, we filtered out ULDs that contained less than two items. Stage 5 ("5_Final_results") contains aggregated results, such as the aggregated number of correct predictions, underestimations, overestimations, and sensitivity analyses results. <h3>Further references</h3> The paper (preprint) describing the study can be found <a href="https://papers.ssrn.com/abstract=4778113" target="_blank" rel="noopener">here</a>. The code for data generation, simulation, and analysis is in the <a href="https://github.com/philippmaz/palletizing_stability_benchmark/" target="_blank" rel="noopener">GitHub Repository</a> (also linked below). <h3>Changelog:</h3> <table> <tbody> <tr> <td>Version</td> <td>Change</td> </tr> <tr> <td>0.0.3</td> <td>This version fixes a bug during item dimension mapping in dataset 2 that made a re-simulation necessary. We imposed a cap on the number of items of 20. If a ULD exceeds this threshold, we include only the first 20 items. For the analysis, we included a minimal item level per ULD of 2. We set the minimal item dimensions to 10 (previously: 15). We removed macOS-specific files from the archive.</td> </tr> </tbody> </table>

opencc-by-nc-4.0Oct 2024View details →

Birdsong NOIZEUS: Bioacoustics noise reduction benchmark dataset

------------------------------------------------------------------------ Birdsong noizeus dataset ------------------------------------------------------------------------ Authors: Tim Sainburg & Asaf Zorea Year: 2024 ------------------------------------------------------------------------ General information ------------------------------------------------------------------------ - There are 5 recordings from each of 14 individuals (European starlings) recorded in an acoustically isolated chamber.  - For each song, we apply noise at 5 SNR levels (0dB, 5dB, 10dB, 15dB).  - There are 8 noise types, each taken from a single noise clip from the "Soundscapes from around the world" dataset. - They are "rain", "town", "wind", "waterfall", "insect", "swamp" "frogscape", "forest" - Each soundscape contains multiple noise sources.  - Noise levels were estimated using the pyloudnorm software (Steinmetz et al., 2021) - Audio is provided as waveforms at 44100 samplerate ------------------------------------------------------------------------ Data format ------------------------------------------------------------------------ - clean     - {bird_name}_{timestamp}.wav - noisy     - {snr}dB         - {bird_name}_{timestamp}_{noise_category}_{snr}.wav - noise_sample     - {snr}dB         - {bird_name}_{timestamp}_{noise_category}_{snr}.wav `clean` contains the original clean audio. `noisy` contains the song+noise `noise_sample` contains a 1-second sample of noise only.  Timestamp is in the format YYYY-MM-DD_HH-MM-SS-MILLISECONDS and refers to the time that the song was recorded.  ------------------------------------------------------------------------ Data sources ------------------------------------------------------------------------ Birdsong --------- Birdsong are acoustically isolated songs from 14 European Starlings https://zenodo.org/records/3237218 Citation:  Arneodo, Z., Sainburg, T., Jeanne, J., & Gentner, T. (2019). An acoustically isolated European starling song library (Version v1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.3237218 This dataset is available under the following license: -  Creative Commons Attribution 4.0 International (https://creativecommons.org/licenses/by/4.0/legalcode) Noise ----- Noise are taken from the Xeno-canto - "Soundscapes from around the world" dataset https://www.gbif.org/dataset/ff571aeb-46bf-45c4-ad2c-af4d68315765 Citation: Vellinga W (2024). Xeno-canto - Soundscapes from around the world. Xeno-canto Foundation for Nature Sounds. Occurrence dataset https://doi.org/10.15468/9u3zaq accessed via GBIF.org on 2024-10-17. We sampled 8 soundscapes from this dataset:     1. Rain https://www.gbif.org/occurrence/4523646364         - Virginia         - 457s         - rain, Recording of the feeders in the backyard with a light rain falling on the leaf litter, while a freight train passes by ~1/2 mile away.     2. Town https://xeno-canto.org/696263         - 2:22         -  the closer habitat are greater trees and coniferes ..smaller bushes ...some other Krautgärten ... traffic-noise is to hear..as airplanes,too.         - insects, birds, bells, traffic?, airplane         - Germany     3. Wind https://xeno-canto.org/911773         - 5:29         - Very windy day, grassland to shrubland habitat         - Wisconsin     4. Waterfall https://xeno-canto.org/406993         - 24:02         - sound scenes captured in a river forest with zarzas, hiedras and other matorrales at the edge of a small waterfall.         - Spain     5. Insect https://xeno-canto.org/454914         - 5:45         - Australia         - cicadias, birds,     6. Swanp https://xeno-canto.org/909875         - 2:24         - Swampy area along dirt road/trail. Species Include: American Bullfrog, Cricket Frogs, American Crow, Prothonotary Warbler, Red-winged Blackbird, Indigo Bunting, Northern Cardinal     7. Frogscape https://xeno-canto.org/718213         - 4:42          - European Tree Frog Hyla arborea Green Frog Pelophlyax sp.         - Austrial      8. Forest https://xeno-canto.org/900744          - 3:04         - Sweden         - A beautiful choir with frogs, toads, geese and ducks to enjoy in the darkness a foggy night.                            All noise recordings have one of the following licensesL - Creative Commons Attribution-NonCommercial-ShareAlike 4.0 (https://creativecommons.org/licenses/by-nc-sa/4.0/) - Creative Commons Attribution-ShareAlike 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Additional information about the noise dataset ------------------------------------------------------------------------ citations ------------------------------------------------------------------------ Steinmetz, C. J., & Reiss, J. (2021, May). pyloudnorm: A simple yet flexible loudness meter in python. In Audio Engineering Society Convention 150. Audio Engineering Society. Arneodo, Z., Sainburg, T., Jeanne, J., & Gentner, T. (2019). An acoustically isolated European starling song library (Version v1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.3237218 Vellinga W (2024). Xeno-canto - Soundscapes from around the world. Xeno-canto Foundation for Nature Sounds. Occurrence dataset https://doi.org/10.15468/9u3zaq accessed via GBIF.org on 2024-10-17.

opencc-by-4.0Aug 2023View details →

Code and data for molecular dating benchmark based on real and simulated Primates gene trees

Benchmark of molecular clock dating applied to single gene trees separately, whose results are described in “Factors influencing the accuracy and precision in dating single gene trees” by Guillaume Louvel and Hugues Roest Crollius. <ol> <li>Real Primates gene trees are analyzed to identify what characteristics of a gene tree are related to the precision of dating; </li> <li>alignments are also simulated on the tree of Primates to measure the accuracy of dating under controlled parameters such as the degree of rate variation and the length of the alignment.</li> </ol> Content <code>Louvel_Accuracy-dating_results_2024.tar.gz</code>:     - <code>notebook/</code>: statistical analyses in Python;     - <code>outputs/</code>: html reports with figures/tables resulting from the analysis;     - <code>lib/</code>: required libraries.     - <code>data/</code>: intermediate data needed for the final analysis (dates, gene tree features); <code>Louvel_Accuracy-dating_dating-source_2024.tar.gz</code>:     - <code>dating-source/</code>: input data and config files necessary to reproduce <code>data</code>; <code>Louvel_Accuracy-dating_raw-data-preparation_2024.tar.gz</code>:     - <code>raw-data-preparation/</code>: raw data and steps to produce <code>dating-source</code>. Requirements This code requires the libraries developed in the lab for this project, available at <a href="https://github.com/DyogenIBENS/">github.com/DyogenIBENS/</a>, but also included here in <code>lib/</code>. - <a href="https://github.com/DyogenIBENS/Phylorgs">Phylorgs</a> - <a href="https://github.com/DyogenIBENS/LibsDyogen_py3">LibsDyogen_py3</a> - <a href="https://github.com/DyogenIBENS/ToolsDyogen_py3">ToolsDyogen_py3</a>.

opencc-by-4.0Dec 2022View details →

Benchmark instances and models for the Oven Scheduling Problem

This dataset contains benchmark instances for the Oven Scheduling Problem that was introduced by the authors in Lackner, M.-L., Mrkvicka, C., Musliu, N., Walkiewicz, D., Winter, F., Minimizing Cumulative Batch Processing Time for an Industrial Oven Scheduling Problem. In: 27th International Conference on Principles and Practice of Constraint Programming (CP 2021). Moreover, MiniZinc models and CP Optimizer models as well as the solutions found for the benchmark set are provided.

Ice Anatomy: A Benchmark Dataset and Methodology for Automatic Ice Boundary Extraction from Radio-Echo Sounding Data

The measurement of ice thickness is of great importance for the accurate estimation of glacier volume and the delineation of their bedrock topography. In particular, this is a crucial factor in forecasting the future evolution of glaciers in the context of a changing climate. In order to derive the ice thickness, the travel time of electromagnetic waves in radargrams acquired by radio-echo sounding (RES) systems is analyzed. This can only be achieved by identifying the ice surface and underlying ice bottom in corresponding radargrams. Manually identifying these two reflection horizons in RES data is a laborious and time-consuming process. Consequently, scientists are attempting to automate this task through the use of techniques such as deep learning. Such automation can significantly reduce the time between a field campaign and the calculation of the glacier's ice thickness distribution. In this paper, we present the first benchmark dataset for delineating the ice surface and bottom boundaries in RES data, to facilitate straightforward comparisons of deep learning models in the future. The ``IceAnatomy'' dataset comprises radargrams and the corresponding manual picks, amounting to a total of over 45,000km of observations. The RES data originates from three sources: FAU, CReSIS, and AWI. The dataset comprises different RES systems as well as different pre-processing methods. In addition, the data was acquired over a large range of geographical and glaciological settings, featuring different thermal regimes present in Antarctica and the Southern Patagonian Icefield. This diversity ensures that the models' behaviors can be analyzed in different scenarios. We define a standardized train-test split for each source in the dataset. This allows us to introduce not only a baseline model trained on the entire training set (the ``omni'' model), but also three source-specific baseline models. The source-specific models are trained exclusively on the subset of the training data acquired by the specified source. The baseline models provide an initial benchmark against which subsequent models can be compared. The source-specific models demonstrate more accurate results than the omni model. For the FAU, CReSIS, and AWI test sets, the source-specific models achieve  Mean Meter Errors of 2.1m, 23.1m, and 4.9m for the ice surface and 9.1m, 78.2m, and 29.3m for the ice bottom. In relation to the mean measured ice thickness, these errors equate to 1.2%, 3.1%, and 0.3% for the ice surface and  4.9%, 10.4%, and 1.5% for the ice bottom.    For more information, please read the following paper: [Coming soon. Currently under review.] Please also cite this paper if you plan on using the dataset.   For the implementation of a baseline model please visit: [Coming soon]

opencc-by-4.0Nov 2024View details →

LiDAR Segmentation Benchmark Dataset: La Palma

The purpose of this benchmark dataset is primarily to allow the assessment of individual tree segmentation algorithms on LiDAR data. The high density ULS point cloud data were captured by the Zenmuse L1 sensor on a DJI Matrice 300RTK over a coniferous plot on La Palma (Canary Islands, Spain). The dataset comprises the CHM and the normalized 3D cloud, as well as the fully annotated data to serve as a benchmark for segmentation. The publication associated with this database is: Marcello, J.; Spínola, M.; Albors, L.; Marqués, F.; Rodríguez-Esparragón, D.; Eugenio, F. Performance of Individual Tree Segmentation Algorithms in Forest Ecosystems Using UAV LiDAR Data. Drones 2024, 8, 772. https://doi.org/10.3390/drones8120772

opencc-by-4.0Jun 2024View details →

WONDERBREAD: A Benchmark + Dataset for Business Process Management (BPM) Tasks

Paper: <a href="https://arxiv.org/abs/2406.13264">WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Tasks</a> <h2>Background</h2> The WONDERBREAD dataset contains 2,928 human demonstrations of 598 web navigation workflows across 6 types of BPM tasks. These tasks measure the ability of a model to generate accurate documentation, assist in knowledge transfer, and improve the efficiency of workflows. Please see our website for more details: <a href="https://wonderbread.stanford.edu/">https://wonderbread.stanford.edu/</a> <h2>Quick Start</h2> To start, download debug_demos.zip (1 GB). It contains a subset of 24 demonstrations which can give you a sense of how the dataset is structured. To reproduce the paper, download gold_demos.zip (33 GB). It contains 724 demonstrations corresponding to the 162 "Gold" tasks which were used for all the evaluations in the original paper. To obtain the full dataset, download demos.zip (133 GB). This contains all 2,928 demonstrations and can be used for training, fine-tuning, and evaluating models. <h2>Dataset Structure</h2> The dataset contains several files, defined below. <ol> <li>Raw Data (useful for training/fine-tuning/evaluation) <ol> <li>debug_demos.zip (1 GB) -- a subset of only 24 demonstrations taken from the full dataset. Useful to get a sense of the dataset and for debugging.</li> <li>gold_demos.zip (22 GB) -- a subset of only 724 demonstrations corresopnding to the 162 "Gold" tasks. This is the dataset that was used for all evaluations in the original WONDERBREAD paper.</li> <li>demos.zip (133 GB) -- all 2,928 demonstrations across 598 tasks. Useful for training your own models.</li> </ol> </li> <li>Modality-Specific Subsets of Raw Data (useful for specific types of training/fine-tuning/evaluation) <ol> <li>All Demos <ol> <li>demos_sop_only.zip (4 MB)-- only the SOP <code>.txt</code> files for all 2,928 demonstrations</li> <li>demos_sop_and_trace_only.zip (770 MB)-- only the SOP <code>.txt</code> files and action trace <code>.json</code> files for all 2,928 demonstrations</li> <li>demos_sop_and_trace_and_screenshots_only.zip (22 GB)-- only the SOP <code>.txt</code> files and action trace <code>.json</code> files and screenshot images for all 2,928 demonstrations</li> </ol> </li> <li>"Gold" Demos <ol> <li>gold_demos_sop_only.zip (1 MB)-- only the SOP <code>.txt</code> files for the 724 demonstrations in the "Gold" tasks.</li> <li>gold_demos_sop_and_trace_only.zip (190 MB) -- only the SOP <code>.txt</code> files and action trace <code>.json</code> files for the 724 demonstrations in the "Gold" tasks.</li> <li>gold_demos_sop_and_trace_and_screenshots_only.zip (6 GB) -- only the SOP <code>.txt</code> files and action trace <code>.json</code> files and screenshot images for the 724 demonstrations in the "Gold" tasks</li> </ol> </li> </ol> </li> <li>Evaluation (useful for evaluation) <ol> <li>qa_dataset.csv -- contains all 120 questions and ground truth answers used in the "Knowlege Transfer" evaluation. </li> <li>df_rankings.csv -- contains the rankings of all "Gold" tasks used in the "SOP Ranking" evaluation. </li> </ol> </li> <li>Metadata (can be safely ignored) <ol> <li>Process Mining Task Demonstrations.xlsx -- maps human annotators to specific demonstrations; also contains "Gold" task rankings used in the "SOP Ranking" evaluation.</li> <li>metadata.json -- maps Google Drive URLs to Google Drive Folder IDs to demonstration names</li> <li>df_valid.csv -- tracks assets associated with each demonstration</li> </ol> </li> </ol>

openmit-licenseApr 2021View details →

PSB2: The Second Program Synthesis Benchmark Suite

PSB2: The Second Program Synthesis Benchmark Suite Datasets Version 1.0.1 (see version history at bottom) This repository contains datasets for the 25 problems described in the paper PSB2: The Second Program Synthesis Benchmark Suite. These problems come from a variety of sources, and require a range of programming constructs and datatypes to solve. These datasets are designed to be usable for any method of performing general program synthesis, including and not limited to inductive program synthesis and evolutionary methods such as genetic programming. For more information, see the associated website: <a href="https://cs.hamilton.edu/~thelmuth/PSB2/PSB2.html">https://cs.hamilton.edu/~thelmuth/PSB2/PSB2.html</a> Use Each problem in the benchmark suite is located in a separate directory in the `datasets` directory. For each problem, we provide a set of `edge` cases and a set of `random` cases. The `edge` cases are hand-chosen cases representing the limits of the problem. The `random` cases are all generated based on problem-specific distributions. For each problem, we included exactly 1 million `random` cases. A typical use of these datasets for a set of runs of program synthesis would be: <ul> <li>For each run, use every `edge` case in the training set</li> <li>For each run, use a different, randomly-sampled set of `random` cases in the training set.</li> <li>Use a larger set of `random` cases as an unseen test set.</li> </ul> Sampling Libraries We provide the following libraries to make the downloading and sampling of these datasets easier. Using these libraries, you do not need to download the entire dataset from Zenodo; the individual problem datasets are downloaded and stored once when first sampling them. <ul> <li>Python: <a href="https://github.com/thelmuth/psb2-python">https://github.com/thelmuth/psb2-python</a></li> <li>Clojure: <a href="https://github.com/thelmuth/psb2-clojure">https://github.com/thelmuth/psb2-clojure</a></li> </ul> Dataset format Each edge and random dataset is provided in three formats: CSV, JSON, and EDN, with all three formats containing identical data. The CSV files are formatted as follows: <ul> <li>The first row of the file is the column names.</li> <li>Each following row corresponds to one set of program inputs and expected outputs.</li> <li>Input columns are labeled `input1`, `input2`, etc., and output columns are labeled `output1`, `output2`, etc.</li> <li>In CSVs, string inputs and outputs are double quoted when necessary, but not if not necessary. Newlines within strings are escaped.</li> <li>Columns in CSV files are comma-separated.</li> </ul> The JSON and EDN files are formatted using the <a href="https://jsonlines.org/">JSON Lines</a> standard (adapted for EDN). Each case is put on its own line of the data file. The files should be read line-by-line and each parsed into an object/map using a JSON/EDN parser. Citation If you use these datasets in a publication, please cite the paper PSB2: The Second Program Synthesis Benchmark Suite and include a link to this repository. BibTeX entry for paper: <pre><code>@InProceedings{Helmuth:2021:GECCO, author = "Thomas Helmuth and Peter Kelly", title = "{PSB2}: The Second Program Synthesis Benchmark Suite", booktitle = "2021 Genetic and Evolutionary Computation Conference", series = {GECCO '21}, year = "2021", isbn13 = {978-1-4503-8350-9}, address = {Lille, France}, size = {10 pages}, doi = {10.1145/3449639.3459285}, publisher = {ACM}, publisher_address = {New York, NY, USA}, month = {10-14} # jul, doi-url = {https://doi.org/10.1145/3449639.3459285}, URL = {https://dl.acm.org/doi/10.1145/3449639.3459285}, }</code></pre> Version History 1.0.0 - 2021/4/10 - Initial publication of PSB2 datasets on Zenodo. 1.0.1 - 2021/7/9 - Changes to CSVs to quote all strings that could be read as integers. No changes in actual data, just formatting.

opencc-by-4.0Jul 2021View details →

The Oberwolfach Steel-Profile benchmark revisited

This archive features the matrix data for the reimplemented Steel Profile benchmark from the Oberwolfach Collection. In contrast to the original, it uses FENICS, rather than ALBERTA, for the finite element semi-discretization in space. <ol> <li>Download and unzip all ZIP files for the dimensions you need (numbers in the names are dimensions of the system)</li> <li>add getrail.m to the same folder</li> <li>download mmread.m from https://math.nist.gov/MatrixMarket/mmio/matlab/mmiomatlab.html</li> </ol> See `help getrail` for usage instructions.