Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
15
datasets available to search
ShareScore release 0.9.0
Dataset results
15 results for “Supercomputing”
Performance results of LeanMD on the Joliot-Curie supercomputer using different load balancing algorithms
<p>This dataset contains the raw output files generated from the execution of LeanMD on the Joliot-Curie supercomputer (20 SKL Irene nodes), the scripts used to generate them, and the scripts used to parse these results for statistical analysis and plotting.</p> <p><strong>Software information</strong>:</p> <ul> <li>OS: Red Hat Enterprise Linux 7.6</li> <li>OpenMPI: version 2.0.4</li> <li>Compilers: C/C++ Intel 17.0.6.256</li> <li>Charm++ version: v6.9.0-rc3, build mpi-linux-x86_64 --with-production</li> <li>LeanMD source: <a href="https://charm.cs.illinois.edu/gerrit/gitweb?p=benchmarks/leanmd.git">https://charm.cs.illinois.edu/gerrit/gitweb?p=benchmarks/leanmd.git</a></li> <li>Additional load balancers source: <a href="https://github.com/viniciusmctf/packing-schemes/tree/packs_2019-v1">https://github.com/viniciusmctf/packing-schemes/tree/packs_2019-v1</a></li> <li>Charm++, LeanMD, and the load balancers were compiled with -O3</li> </ul> <p><strong>File information</strong>:</p> <p>The raw result files are organized in four directories (oct18, oct23, oct24, and oct24_2).<br> Each directory contains the results of one batched execution in the supercomputer.<br> Each batch is composed of 10 repetitions of a set of experiments.<br> Each set of experiments includes different load balancing algorithms and different problem sizes.<br> Each set is randomly ordered to avoid interference coming from a specific order of execution.<br> Each raw file contains the appended output of the application and its load balancer for all 10 repetitions.<br> The name of the files indicate the load balancer and size of the problem.<br> For instance, `PackStealLB.240` means that the application was run with PackStealLB and the problem size parameter is 240.</p> <p><strong>Problem sizes</strong>:</p> <ul> <li>80: 80×11×5 cells of dimensions 15×15×30</li> <li>120: 120×11×5 cells of dimensions 15×15×30</li> <li>160: 160×11×5 cells of dimensions 15×15×30</li> <li>240: 240×11×5 cells of dimensions 15×15×30</li> <li>320: 320×11×5 cells of dimensions 15×15×30</li> </ul> <p>Each execution of LeanMD ran for 301 iterations with load balancing calls at iterations 40, 140, and 240.</p> <p><strong>Raw output files</strong>:</p> <p>Each raw output file starts with a Charm++ header providing information on the execution.</p> <p>For each step of the application, its execution time in ms is provided.</p> <p>Load balancing calls usually provide information about their start time, end time, and duration. Depending on the load balancer, more information is provided.</p> <p>Each raw file contains ten executions of the application with a given load balancer and input size.</p> <p><strong>Generating plots</strong>:</p> <p>The analysis of the results can be done by running the Jupyter notebook named "Analysis of load balancing results.ipynb"</p>
Communication Characterization and Optimization of Applications Using Topology-Aware Task Mapping on Large Supercomputers (Data)
<p>Auxiliary materials for ICPE 2016 paper titled:<br /> "Communication Characterization and Optimization of Applications Using Topology-Aware Task Mapping on Large Supercomputers".</p> <p> </p>
Wrapper Impact Workloads and BSC Slurm Simulator Output of Static Traces based on Data from LUMI Supercomputer
<p>This dataset contains the workloads, with the workflow added to them, and the results of the simulations of the static trace utilizing LUMI fitted data carried out using <a href="https://ieeexplore.ieee.org/abstract/document/8641556">BSC's SLURM Simulator</a>.</p> <p>It is organized in two folders: workloads and results. In the first, we find a folder per experiment, which is a different randomly generated workload file. Within each experiment we find a folder per fair share inidicating the target platform, the workflow it was based on, and the characteristics of the tracked job: number of cores and runtime. The results folder follows the same scheme but with a file extension of ".trace".</p>
April 2019 Darshan counters from the Cori supercomputer
<p><em>Copyright (c) 2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy). All rights reserved.</em></p> <p> </p> <p><strong>GENERAL INFORMATION</strong></p> <p> </p> <p>1. <em>Title of Dataset</em>: <strong>April 2019 Darshan counters from the Cori supercomputer</strong></p> <p> </p> <p>2. <em>Author Information</em>:</p> <p> </p> <p> A. Principal Investigator Contact Information</p> <p> - Name: Suren Byna</p> <p> - Institution: Lawrence Berkeley National Laboratory</p> <p> - Address: 1 Cyclotron Rd, Berkeley, CA 94720</p> <p> - Email: sbyna@lbl.gov</p> <p> </p> <p> B. Associate or Co-investigator Contact</p> <p> - Name: Jean Luca Bez</p> <p> - Institution: Lawrence Berkeley National Laboratory</p> <p> - Address: 1 Cyclotron Rd, Berkeley, CA 94720</p> <p> - Email: jlbez@lbl.gov</p> <p> </p> <p>3. <em>Date of data collection</em>: 2019-04-01</p> <p> </p> <p>4. <em>Geographic location of data collection</em>: Berkeley, California, USA</p> <p> </p> <p>5. <em>Information about funding sources that supported the collection of the data</em>:</p> <p> </p> <p> This research used resources of the National Energy Research Scientific Computing Center (NERSC), a U.S. Department of Energy Office of Science User Facility located at Lawrence Berkeley National Laboratory, operated under Contract No. DE-AC02-05CH11231.</p> <p> </p> <p><strong>SHARING/ACCESS INFORMATION</strong></p> <p> </p> <p>1. <em>Licenses/restrictions placed on the data</em>:</p> <p> </p> <p> - Creative Commons - Attribution-NonCommercial-NoDerivatives 4.0 International (<a href="https://creativecommons.org/licenses/by-nc-nd/4.0">CC BY-NC-ND 4.0</a>)</p> <p> </p> <p>2. <em>Links to publications that cite or use the data</em>: N/A</p> <p> </p> <p>3. <em>Links to other publicly accessible locations of the data</em>: N/A</p> <p> </p> <p>4. <em>Links/relationships to ancillary data sets</em>: N/A</p> <p> </p> <p>5. <em>Was data derived from another source?</em> YES</p> <p> </p> <p> This dataset is derived from raw Darshan logs collected in Cori during April 2020.</p> <p> </p> <p>6. <em>Recommended citation for this dataset</em>:</p> <p> </p> <p> Jean Luca Bez, Ahmad Maroof Karimi, Arnab K. Paul, Bing Xie, Suren Byna, Philip Carns, Sarp Oral, Feiyi Wang, and Jesse Hanley. 2022. Access Patterns and Performance Behaviors of Multi-layer Supercomputer I/O Subsystems under Production Load. In Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing (HPDC '22). Association for Computing Machinery, New York, NY, USA.</p> <p> </p> <p><strong>DATA & FILE OVERVIEW</strong></p> <p> </p> <p>1. <em>File List</em>:</p> <p> </p> <p> 1. `darshan`: Dataset derived from raw Darshan logs collected in Cori during April 2020. Months are organized as directories with a `Darshan.csv` file in each day containing all the counters collected and relevant to our investigation.</p> <p> </p> <p> 2. `domains`: Dataset maps job identification to their primary associated science domain as reported by NERSC upon project registration. A single CSV comprises all jobs in April 2020 covered or not by Darshan.</p> <p> </p> <p>2. <em>Additional related data collected that was not included in the current data package</em>:</p> <p> </p> <p> - The original raw Darshan logs are not included in this package.</p> <p> </p> <p>4. <em>Are there multiple versions of the dataset?</em></p> <p> </p> <p> - No, this is the first version of this dataset.</p> <p> </p> <p><strong>DESCRIPTIONS FOR VARIABLES USED IN THE FILES</strong></p> <p> </p> <p>1. <em>Darshan counter dataset</em></p> <p> </p> <p>* `uid`: User identifier that ran this job</p> <p>* `jobid`: Job identifier from the SLURM scheduler</p> <p>* `rank`: Rank identifier</p> <p>* `recordid`: File record identifier</p> <p>* `fstype`: File system type</p> <p>* `POSIX_OPENS`: Number of open calls in POSIX</p> <p>* `POSIX_FILENOS`: Number of POSIX fileno operations</p> <p>* `POSIX_DUPS`: Number of POSIX dup operations</p> <p>* `POSIX_READS`: Number of POSIX read operations</p> <p>* `POSIX_WRITES`: Number of POSIX write operations</p> <p>* `POSIX_SEEKS`: Number of POSIX seek operations</p> <p>* `POSIX_STATS`: Number of POSIX stat operations</p> <p>* `POSIX_MMAPS`: Number of POSIX mmap operations</p> <p>* `POSIX_FSYNCS`: Number of POSIX fsync operations</p> <p>* `POSIX_FDSYNCS`: Number of POSIX fdatasync operations</p> <p>* `POSIX_BYTES_READ`: Total number of bytes that were read from the file</p> <p>* `POSIX_BYTES_WRITTEN`: Total number of bytes written to the file</p> <p>* `POSIX_CONSEC_READS`: Number of consecutive reads (that were immediately adjacent to the previous access)</p> <p>* `POSIX_CONSEC_WRITES`: Number of consecutive writes (that were immediately adjacent to the previous access)</p> <p>* `POSIX_SEQ_READS`: Number of sequential reads (at a higher offset than where the previous access left off)</p> <p>* `POSIX_SEQ_WRITES`: Number of sequential writes (at a higher offset than where the previous access left off)</p> <p>* `POSIX_SIZE_READ_0_100`: Number of read access between 0 and 100 bytes at POSIX level</p> <p>* `POSIX_SIZE_READ_100_1K`: Number of read access between 100 bytes and 1KB at POSIX level</p> <p>* `POSIX_SIZE_READ_1K_10K`: Number of read access between 1KB and 10KB at POSIX level</p> <p>* `POSIX_SIZE_READ_10K_100K`: Number of read access between 10KB and 100KB at POSIX level</p> <p>* `POSIX_SIZE_READ_100K_1M`: Number of read access between 100KB and 1MB at POSIX level</p> <p>* `POSIX_SIZE_READ_1M_4M`: Number of read access between 1MB and 4MB at POSIX level</p> <p>* `POSIX_SIZE_READ_4M_10M`: Number of read access between 4MB and 10MB at POSIX level</p> <p>* `POSIX_SIZE_READ_10M_100M`: Number of read access between 10MB and 100MB at POSIX level</p> <p>* `POSIX_SIZE_READ_100M_1G`: Number of read access between 100MB and 1GB at POSIX level</p> <p>* `POSIX_SIZE_READ_1G_PLUS`: Number of read access greater than 1GB at POSIX level</p> <p>* `POSIX_SIZE_WRITE_0_100`: Number of write access between 0 and 100 bytes at POSIX level</p> <p>* `POSIX_SIZE_WRITE_100_1K`: Number of write access between 100 bytes and 1KB at POSIX level</p> <p>* `POSIX_SIZE_WRITE_1K_10K`: Number of write access between 1KB and 10KB at POSIX level</p> <p>* `POSIX_SIZE_WRITE_10K_100K`: Number of write access between 10KB and 100KB at POSIX level</p> <p>* `POSIX_SIZE_WRITE_100K_1M`: Number of write access between 100KB and 1MB at POSIX level</p> <p>* `POSIX_SIZE_WRITE_1M_4M`: Number of write access between 1MB and 4MB at POSIX level</p> <p>* `POSIX_SIZE_WRITE_4M_10M`: Number of write access between 4MB and 10MB at POSIX level</p> <p>* `POSIX_SIZE_WRITE_10M_100M`: Number of write access between 10MB and 100MB at POSIX level</p> <p>* `POSIX_SIZE_WRITE_100M_1G`: Number of write access between 100MB and 1GB at POSIX level</p> <p>* `POSIX_SIZE_WRITE_1G_PLUS`: Number of write access greater than 1GB at POSIX level</p> <p>* `POSIX_F_OPEN_START_TIMESTAMP`: Timestamp that the first POSIX file open operation began</p> <p>* `POSIX_F_READ_START_TIMESTAMP`: Timestamp that the first POSIX file read operation began</p> <p>* `POSIX_F_WRITE_START_TIMESTAMP`: Timestamp that the first POSIX file write operation began</p> <p>* `POSIX_F_CLOSE_START_TIMESTAMP`: Timestamp that the first POSIX file close operation began</p> <p>* `POSIX_F_OPEN_END_TIMESTAMP`: Timestamp that the last POSIX file open operation ended</p> <p>* `POSIX_F_READ_END_TIMESTAMP`: Timestamp that the last POSIX file read operation ended</p> <p>* `POSIX_F_WRITE_END_TIMESTAMP`: Timestamp that the last POSIX file write operation ended</p> <p>* `POSIX_F_CLOSE_END_TIMESTAMP`: Timestamp that the last POSIX file close operation ended</p> <p>* `POSIX_F_READ_TIME`: Cumulative time spent reading at the POSIX level</p> <p>* `POSIX_F_WRITE_TIME`: Cumulative time spent in write, fsync, and fdatasync at the POSIX level</p> <p>* `POSIX_F_META_TIME`: Cumulative time spent in open, close, stat, and seek at the POSIX level</p> <p>* `MPIIO_INDEP_OPENS`: Number of non-collective MPI opens</p> <p>* `MPIIO_COLL_OPENS`: Number of collective MPI opens</p> <p>* `MPIIO_INDEP_READS`: Number of non-collective MPI reads</p> <p>* `MPIIO_INDEP_WRITES`: Number of non-collective MPI writes</p> <p>* `MPIIO_COLL_READS`: Number of collective MPI reads</p> <p>* `MPIIO_COLL_WRITES`: Number of collective MPI writes</p> <p>* `MPIIO_SPLIT_READS`: Number of MPI split collective reads</p> <p>* `MPIIO_SPLIT_WRITES`: Number of MPI split collective writes</p> <p>* `MPIIO_NB_READS`: Number of MPI non-blocking reads</p> <p>* `MPIIO_NB_WRITES`: Number of MPI non-blocking writes</p> <p>* `MPIIO_SYNCS`: Number of MPI file syncs</p> <p>* `MPIIO_MODE`: MPI mode that the file was last opened in</p> <p>* `MPIIO_BYTES_READ`: Total number of bytes that were read from the file at MPI level</p> <p>* `MPIIO_BYTES_WRITTEN`: Total number of bytes written to the file at MPI level</p> <p>* `MPIIO_RW_SWITCHES`: Number of times that access toggled between read and write in consecutive MPI operations</p> <p>* `MPIIO_SIZE_READ_0_100`: Number of read access between 0 and 100 bytes at MPIIO level</p> <p>* `MPIIO_SIZE_READ_100_1K`: Number of read access between 100 bytes and 1KB at MPIIO level</p> <p>* `MPIIO_SIZE_READ_1K_10K`: Number of read access between 1KB and 10KB at MPIIO level</p> <p>* `MPIIO_SIZE_READ_10K_100K`: Number of read access between 10KB and 100KB at MPIIO level</p> <p>* `MPIIO_SIZE_READ_100K_1M`: Number of read access between 100KB and 1MB at MPIIO level</p> <p>* `MPIIO_SIZE_READ_1M_4M`: Number of read access between 1MB and 4MB at MPIIO level</p> <p>* `MPIIO_SIZE_READ_4M_10M`: Number of read access between 4MB and 10MB at MPIIO level</p> <p>* `MPIIO_SIZE_READ_10M_100M`: Number of read access between 10MB and 100MB at MPIIO level</p> <p>* `MPIIO_SIZE_READ_100M_1G`: Number of read access between 100MB and 1GB at MPIIO level</p> <p>* `MPIIO_SIZE_READ_1G_PLUS`: Number of read access greater than 1GB at MPIIO level</p> <p>* `MPIIO_SIZE_WRITE_0_100`: Number of write access between 0 and 100 bytes at MPIIO level</p> <p>* `MPIIO_SIZE_WRITE_100_1K`: Number of write access between 100 bytes and 1KB at MPIIO level</p> <p>* `MPIIO_SIZE_WRITE_1K_10K`: Number of write access between 1KB and 10KB at MPIIO level</p> <p>* `MPIIO_SIZE_WRITE_10K_100K`: Number of write access between 10KB and 100KB at MPIIO level</p> <p>* `MPIIO_SIZE_WRITE_100K_1M`: Number of write access between 100KB and 1MB at MPIIO level</p> <p>* `MPIIO_SIZE_WRITE_1M_4M`: Number of write access between 1MB and 4MB at MPIIO level</p> <p>* `MPIIO_SIZE_WRITE_4M_10M`: Number of write access between 4MB and 10MB at MPIIO level</p> <p>* `MPIIO_SIZE_WRITE_10M_100M`: Number of write access between 10MB and 100MB at MPIIO level</p> <p>* `MPIIO_SIZE_WRITE_100M_1G`: Number of write access between 100MB and 1GB at MPIIO level</p> <p>* `MPIIO_SIZE_WRITE_1G_PLUS`: Number of write access greater than 1GB at MPIIO level</p> <p>* `MPIIO_F_OPEN_START_TIMESTAMP`: Timestamp that the first MPIIO file open operation began</p> <p>* `MPIIO_F_READ_START_TIMESTAMP`: Timestamp that the first MPIIO file read operation began</p> <p>* `MPIIO_F_WRITE_START_TIMESTAMP`: Timestamp that the first MPIIO file write operation began</p> <p>* `MPIIO_F_CLOSE_START_TIMESTAMP`: Timestamp that the first MPIIO file close operation began</p> <p>* `MPIIO_F_OPEN_END_TIMESTAMP`: Timestamp that the last MPIIO file open operation ended</p> <p>* `MPIIO_F_READ_END_TIMESTAMP`: Timestamp that the last MPIIO file read operation ended</p> <p>* `MPIIO_F_WRITE_END_TIMESTAMP`: Timestamp that the last MPIIO file write operation ended</p> <p>* `MPIIO_F_CLOSE_END_TIMESTAMP`: Timestamp that the last MPIIO file close operation ended</p> <p>* `MPIIO_F_READ_TIME`: Cumulative time spent reading at MPI level</p> <p>* `MPIIO_F_WRITE_TIME`: Cumulative time spent write and sync at MPI level</p> <p>* `MPIIO_F_META_TIME`: Cumulative time spent in open and close at MPI level</p> <p>* `MPIIO_F_VARIANCE_RANK_TIME`: The population variance for MPI I/O time of all the ranks</p> <p>* `MPIIO_F_VARIANCE_RANK_BYTES`: The population variance for bytes transferred of all the ranks at MPI level</p> <p>* `STDIO_OPENS`: Number of stdio file open operations</p> <p>* `STDIO_FDOPENS`: Number of stdio fdopen operations</p> <p>* `STDIO_READS`: Number of stdio read operations</p> <p>* `STDIO_WRITES`: Number of stdio write operations</p> <p>* `STDIO_SEEKS`: Number of stdio seek operations</p> <p>* `STDIO_STATS`: Number of stdio stats operations</p> <p>* `STDIO_FLUSHES`: Number of stdio flush operations</p> <p>* `STDIO_BYTES_READ`: Total number of bytes read from the file using stdio operations</p> <p>* `STDIO_BYTES_WRITTEN`: Total number of bytes written to the file using stdio operations</p> <p>* `STDIO_META_TIME`: Cumulative time spent in stdio open/close/seek operations</p> <p>* `STDIO_READ_TIME`: Cumulative time spent in stdio read operations</p> <p>* `STDIO_WRITE_TIM`: Cumulative time spent in stdio write operations</p> <p> </p> <p>2. <em>Science domain dataset</em></p> <p> </p> <p>* `job_id`: Unique job identifier automatically generated by SLURM upon submission</p> <p>* `date`: Date the job started to run as provided by SLURM</p> <p>* `science_domain`: Science domain of the job based on the information provided upon project allocation</p> <p> </p> <p><strong>RESOURCES</strong></p> <p> </p> <p>Helpful resources in understanding and utilizing this dataset:</p> <p> </p> <p>1. <em>Data collection methodology, understanding of Darshan logs and counters</em></p> <p> </p> <p> - <a href="https://www.mcs.anl.gov/research/projects/darshan/documentation">Darshan</a>: Darshan HPC I/O Characterization Tool Documentation</p> <p> </p> <p>2. <em>General information about the NERSC Cori Supercomputer</em></p> <p> </p> <p> - <a href="https://docs.nersc.gov/systems/cori">CORI</a>: General information about the Cori supercomputer</p> <p> - <a href="https://docs.nersc.gov/tools/performance/darshan">Darshan</a>: Darshan I/O profiler documentation in Cori</p>
MARCONI KLN supercomputer HPCG benchmark
<p>This dataset collects High Performance Conjugate Gradient (HPCG) benchmark results of Marconi KNL (Intel Xeon PHI partition) supercomputer, carried out in May 2017.</p>
Marconi supercomputer KNL partition benchmark (HPL and single node stream)
<p>This dataset collect the data stored during the procedure of evaluation of the Marconi KNL machine at CINECA in order to classify it for Top500 list.</p> <p>Dataset also include a report summarizing the results of benchmarks (STREAM for single node memory assessment and HPL for HPC parallel performance) carried out in Nov. 2016.</p>
Predicting the distribution of serotonergic axons: A supercomputing simulation of reflected fractional Brownian motion in a 3D-mouse brain model
Open the record for dataset details and reuse information.
Code and data for Porting the Meso-NH Atmospheric Model on Different GPU Architectures for the Next Generation of Supercomputers (version MESONH-v55-OpenACC)
<p>GeometricMG.pdf (source: https://bitbucket.org/em459/tensorproductmultigrid/src/master/Documentation/)<br>MESONH_Bench_HECTOR_ADASTRA_LEONARDO.tar.gz: code and data for Meso-NH bench<br>Performance.zip: code and data for figures related to performance<br>WeatherApplications.zip: namelists for running weather applications<br>OASIS3_WW3.tar.gz: OASIS and WW3 codes for running the Meso-NH WWW3 coupled simulation</p>
Tuning VASP (ver. 6.2.1) performance on AMD supercomputers
<p>Here are provided several test results for the VASP code version 6.2.1 on Karolina supercomputer (karolina.it4i.cz) at IT4Innovations, VSB - Technical University of Ostrava, Czech Republic (it4i.cz) compiled with different compilers, compiler options, and numerical libraries. Tests have been done for two small systems (folders SMALL-1 and SMALL-2), and two<br> average systems (folders AVERAGE-1 and AVERAGE-2).<br> </p>
Maps of the Top500 Supercomputers over Time
<p>A collection of maps plotting the Top500 Supercomputers from 1993-2022.</p>
The organization of serotonergic fibers in the Pacific angelshark brain: Neuroanatomical and supercomputing analyses
Open the record for dataset details and reuse information.
MOVES-Matrix 3.0: On-Road Energy and Emission Modeling with High-Performance Supercomputing
<p><span>This is the dataset for the NCST project "</span><span>MOVES-Matrix 3.0: On-Road Energy and Emission Modeling with High-Performance Supercomputing</span><span>"</span></p> <p> </p> <p><span>The Georgia Tech research team developed MOVES-Matrix 3.0 based on the EPA's MOVES3 (version 3.1.0) energy use and emission rate model by running MOVES3 thousands of times on the PACE supercomputing cluster across all combinations of input variables and storing the output as lookup tables.<span> </span>MOVES-Matrix 3.0 allows on-road energy consumption and emissions modeling to be conducted more than 800 times faster than running MOVES, while it generates the exact same results, as verified in this report. <span> </span>MOVES-Matrix 3.0 was designed similarly to its predecessor, MOVES-Matrix 2014, but required extensive code modifications to accommodate changes in the MOVES3 environment (including a shift from MySQL to MariaDB and incorporation of new vehicle source sub-types and operating parameters). <span> </span>The review of the fuel and I/M scenarios indicated that MOVES3 now defines 122 modeling regions, as compared with 109 regions in MOVES 2014b (different matrices need to be developed each modeling region).<span> </span>The development of matrices for each modeling region takes approximately 15-20 days on the PACE supercomputing cluster given our assigned resources (compared with only 5-7 days to develop matrices for MOVES 2014). <span> </span>A case study of 3,000 roadway links using Atlanta's matrices confirmed that MOVES-Matrix 3.0 produces the exact same energy consumption and emissions results as MOVES3, but execution modules operate 800 times faster using MOVES-Matrix lookups than running MOVES for any single run.</span></p>
Dataset for the paper: Linking the Dynamic PicoProbe Analytical Electron-Optical Beam Line / Microscope to Supercomputers
<p><strong>Dataset</strong></p> <p>This dataset contains the input and output examples used in the publication described below. The hyperspectral folder contains the first example and use case. The spatiotemporal folder contains the second example use case.</p> <p><strong>Contents</strong></p> <p>data/<br> ├── hyperspectral<br> │ ├── VeloxTest-Membranes.emd<br> │ └── example_output<br> │ ├── simulator-0-VeloxTest-Membranes.emd<br> │ ├── simulator-0-VeloxTest-Membranes.json<br> │ └── simulator-0-VeloxTest-Membranes.png<br> └── spatiotemporal<br> ├── 2023081040-700kx MultiFrams 991ms 600F Counting & FFI mode Falcon 700 kx 1819.emd<br> ├── best-yolo.pt<br> └── example_output<br> ├── prediction-simulator-17-2023081040-700kx MultiFrams 991ms 600F Counting & FFI mode Falcon 700 kx 1819.mp4<br> ├── simulator-17-2023081040-700kx MultiFrams 991ms 600F Counting & FFI mode Falcon 700 kx 1819.emd<br> ├── simulator-17-2023081040-700kx MultiFrams 991ms 600F Counting & FFI mode Falcon 700 kx 1819.json<br> └── simulator-17-2023081040-700kx MultiFrams 991ms 600F Counting & FFI mode Falcon 700 kx 1819.mp4<br> </p> <p><strong>Source Code</strong><br> Our code is hosted on GitHub at the following link: <a href="https://github.com/ramanathanlab/PicoProbeDataFlow/tree/main">https://github.com/ramanathanlab/PicoProbeDataFlow/tree/main</a><br> <br> <strong>Paper Name</strong><br> Linking the Dynamic PicoProbe Analytical Electron-Optical Beam Line / Microscope to Supercomputers<br> <br> <strong>Paper Abstract</strong><br> The Dynamic PicoProbe at Argonne National Laboratory is undergoing upgrades that will enable it to produce up to 100s of GB of data per day. While this data is highly important for both fundamental science and industrial applications, there is currently limited on-site infrastructure to handle these high-volume data streams. We address this problem by providing a software architecture capable of supporting large-scale data transfers to the neighboring supercomputers at the Argonne Leadership Computing Facility. To prepare for future scientific workflows, we implement two instructive use cases for hyperspectral and spatiotemporal datasets, which include: (i) off-site data transfer, (ii) machine learning/artificial intelligence and traditional data analysis approaches, and (iii) automatic metadata extraction and cataloging of experimental results. This infrastructure supports expected workloads and also provides domain scientists the ability to reinterrogate data from past experiments to yield additional scientific value and derive new insights.</p>
Inside-Out: Supercomputers
<p>A really cool short movie about the power of supercomputing.</p> <p>It was created about the time of the birth of K-Computer in Japan - at the time the most powerful supercomputer in the world.</p> <p>A*STAR Computational Resource Centre (A*CRC) which I was a director of at the time had a very powerful Fujitsu supercomputer.</p> <p>Marek Michalewicz speaks at 19:36</p>
UniChem: Molecular Chemistry in a Supercomputing Environment
<p>Movie presenting UNICHEM chemical modelling software form Cary, Inc.</p> <p>The first part presents the software, GUI, components and methods; the second part presents the three cases od research done at CSIRO using this software: hydrogeneted C60H32, surface plasmons on C60 molecule, and neurominidase research Later to be known as Relenza - a flu medicine).</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.