Find research datasets worth reusing
Search datasets from major research repositories and use ShareScore to quickly assess how well each record supports discovery, access, and reuse.
298
datasets available to search
ShareScore release 0.9.0
Dataset results
298 results for “Multi-modal”
A multi-modal human neuroimaging dataset for data integration: simultaneous EEG and fMRI acquisition during a motor imagery neurofeedback task: XP1
Open the record for dataset details and reuse information.
Drone onboard multi-modal sensor dataset for complex outdoor scenarios
<p>The Data acquisition missions were designed and executed using DJI Pilot 2’s flight route planning feature. The missions encompassed five distinct geometric patterns: 1. triangular, 2. circular, 3. rectangular, 4. linear, and 5. multi-dimensional. Each mission was configured as a waypoint flight path, allowing precise customization of parameters such as altitude, speed, and turning angle for each waypoint. The dataset consists of 3D space flight data such as take-off, landing and varying altitude to introduce the z-axis changes. It must be noted that data was logged at a frequency of 10 Hz.</p> <p>To ensure consistency within the data, identical parameters were maintained across all data acquisition missions. The dataset comprises 20 distinct flights, with each flight path repeated multiple times, resulting in approximately 30 minutes of flight time per mission. The dataset is structured as time-series data, with each flight uniquely identified by a flight number and corresponding timestamp. The drone's spatial position is represented by the variables <strong>position_x, position_y, position_z </strong>while its orientation is captured by the variables <strong>orientation_x, orientation_y, orientation_z, orientation_w</strong>. Additionally, the drone's velocity and angular velocity are represented by the variables <strong>velocity_x, velocity_y, velocity_z, angular_x, angular_y, angular_z </strong>respectively. The linear acceleration is described by the variables <strong>linear_acceleration_x, linear_acceleration_y, linear_acceleration_z</strong>. The dataset also includes environmental data such as <strong>wind_speed, wind_angle </strong>using the TriSonica Mini Wind and Weather Sensor as well as information regarding the drone's battery status, including <strong>battery_voltage, battery_current.</strong></p> <p><strong>Data Acquisition Paths: <a href="https://ucy-my.sharepoint.com/:i:/g/personal/ygrigo01_ucy_ac_cy/EYAgdcLGCWxPloO1NMnsF-8Btf390Kmx854IuDe9R3E1ig?e=3Trbuk">Data acquisition paths</a></strong></p> <p>The dataset includes labels for various operational states of the drone, such as IDLE_HOVER, ASCEND, TURN, HMSL and DESCEND. These labels can be utilized to classify the drone's current activity. Moreover, the annotated dataset can be applied in multi-task learning to predict the drone's trajectory.</p> <p>The DJI Matrice 300 RTK is utilized as the primary platform for data acquisition, leveraging its compatibility with onboard development kits to facilitate the extraction of data from its integrated sensors and flight controller. To execute the developed software the NVIDIA Jetson Xavier NX serves as the embedded computing device. Utilizing the Onboard software development kit the Jetson Xavier NX enables real-time access and processing of data from the drone's sensors and flight controller.</p>
Drone onboard multi-modal sensor dataset
<p><strong>Drone onboard multi-modal sensor dataset :</strong></p> <p>This dataset contains timeseries data from numerous drone flights. Each flight record has a unique identifier (uid) and a timestamp indicating when the flight occurred. The drone's position is represented by the coordinates (position_x, position_y, position_z) and altitude. The orientation of the drone is represented by the quaternion (orientation_x, orientation_y, orientation_z, orientation_w). The drone's velocity and angular velocity are represented by (velocity_x, velocity_y, velocity_z) and (angular_x, angular_y, angular_z) respectively. The linear acceleration of the drone is represented by (linear_acceleration_x, linear_acceleration_y, linear_acceleration_z).</p> <p>In addition to the above, the dataset also contains information about the battery voltage (battery_voltage) and current (battery_current) and the payload attached. The payload information indicates if the drone operated with an embdded device attached (nvidia jetson), various sensors, and a solid-state weather station (trisonica).</p> <p>The dataset also includes annotations for the current state of the drone, including IDLE_HOVER, ASCEND, TURN, HMSL and DESCEND. These states can be used for classification to identify the current state of the drone. Furthermore, the labeled dataset can be used for predicting the trajectory of the drone using multi-task learning.</p> <p>For the annotation, we look at the change in position_x, position_y, position_z and yaw. Specifically, if the position_x,<br> position_y changes, it means that the drone moves in a horizontal straight line, if the position_z changes, it means that the drone performs ascending or descending (depends on whether it increases or decreases), if the yaw changes, it means that the drone performs a turn and finally if any of the above features do not change, it means the drone is in idle or hover mode.</p> <p>In addition to the features already mentioned, this dataset also includes data from various sensors including a weather station and an Inertial Measurement Unit (IMU). The weather station provides information about the weather conditions during the flight. This information includes, wind speed, and wind angle. These weather variables could be important factors that could influence the flight of the drone and battery consumption. The IMU is a sensor that measures the drone's acceleration, angular velocity, and magnetic field. The accelerometer provides information about the drone's linear acceleration, while the gyroscope provides information about the drone's angular velocity. The magnetometer measures the Earth's magnetic field, which can be used to determine the drone's orientation.</p> <p>Field deployments were performed in order to collect empirical data using a specific type of drone, specifically a DJI Matrice 300 (M300). The M300 is equipped with advanced sensors and flight control systems, which can provide high-precision flight data. The flights were designed to cover a range of flight patterns, which include triangular flight patterns, square flight patterns, polygonal flight pattern, and random flight patterns. These flight patterns were chosen to represent a variety of different flight scenarios that could be encountered in real-world applications. The triangular flight pattern consists of the drone flying in a triangular path with a fixed altitude. The square flight pattern involves the drone flying in a square path with a fixed altitude. The polygonal flight pattern consists of the drone flying in a polygonal path with a fixed altitude, and the random flight pattern involves the drone flying in a random path with a fixed altitude. Overall, this dataset contains a rich set of flight data that can be used for various research purposes, including developing and testing algorithms for drone control, trajectory planning, and machine learning.</p> <p> </p>
THÖR-Magni (Demo Subset): a new multi-modal context-rich dataset of human-robot motion
<p>The Magni Human Motion Dataset provides high-quality tracking information from motion capture, eye-gaze trackers, and on-board robot sensors in a semantically rich environment. To induce natural behavior of recorded participants, we utilized loosely scripted task assignment, which induced participants to navigate through a dynamic laboratory environment in a natural and purposeful way. The dataset sets a high-quality standard as realistic and accurate data is enhanced with semantic information, enabling development of new algorithms that rely not only on tracking information but also on contextual cues of moving agents, static and dynamic environments.</p> <p> </p> <p>Link to dashboard that uses the data: https://magni-dash.streamlit.app/</p> <p><br> Here we publish a subset of the final dataset, to accompany the presentation at the 2023 IEEE International Conference on Robotics and Automation (ICRA)</p>
The open D1NAMO dataset: A multi-modal dataset for research on non-invasive type 1 diabetes management
<p>The description of the dataset is available at <a href="https://doi.org/10.1016/j.imu.2018.09.003">https://doi.org/10.1016/j.imu.2018.09.003</a></p> <p>The usage of wearable devices has gained popularity in the latest years, especially for health-care and well being. Recently there has been an increasing interest in using these devices to improve the management of chronic diseases such as diabetes. The quality of data acquired through <a href="https://www.sciencedirect.com/topics/medicine-and-dentistry/wearable-sensor">wearable sensors</a> is generally lower than what medical-grade devices provide, and existing datasets have mainly been acquired in highly controlled clinical conditions. In the context of the <em>D1NAMO</em> project — aiming to detect <a href="https://www.sciencedirect.com/topics/medicine-and-dentistry/glycemic">glycemic</a> events through non-invasive <a href="https://www.sciencedirect.com/topics/medicine-and-dentistry/ecg-abnormality">ECG pattern</a> analysis — we elaborated a dataset that can be used to help developing health-care systems based on wearable devices in non-clinical conditions. This paper describes this dataset, which was acquired on 20 healthy subjects and 9 patients with type-1 diabetes. The acquisition has been made in real-life conditions with the <em>Zephyr BioHarness 3</em> wearable device. The dataset consists of <em>ECG</em>, <em>breathing</em>, and <em><a href="https://www.sciencedirect.com/topics/medicine-and-dentistry/accelerometer">accelerometer</a></em> signals, as well as <em>glucose</em> measurements and annotated <em>food pictures</em>. We open this dataset to the scientific community in order to allow the development and evaluation of diabetes management algorithms.</p>
Sample Multi-Modal BIDS dataset (v2.1)
<p>This a sample BIDS dataset created for continous integration of the Connectome Mapper 3.</p> <p>This dataset was acquired at the Centre Hospitalier Universitaire Vaudois (CHUV), Lausanne, Switzerland, using a 3T Siemens Prisma MRI scanner.</p> <p>It adopts the sub-/ses- structure and contains one T1w anatomical MRI (MPRAGE), one diffusion MRI (DSI) , and one resting-state functional MRI as well as additional Freesurfer derivatives.</p> <p>It is distributed under the Creative Commons Attribution 4.0 International (CC BY 4.0) license. (See https://creativecommons.org/licenses/by/4.0/ for more details)</p> <p><strong><em>Changes</em></strong></p> <p><em>Version 2.1</em></p> <ul> <li>Fix issues with the resampling of the DWI and rfMRI scans with Slicer. They were regenerated in version 2.1 with `mri_convert` to better handle the 4th dimension.</li> <li>For the sake of the size of the dataset, only 100 frames in the fMRI recording has been kept and the <em>sourcedata/</em> folder has been dropped but can be easily be retrieved in the previous 2.0 version (https://zenodo.org/record/5788803#.Yb2-giYo8bV).</li> </ul> <p>Version 2.0</p> <ul> <li>For testing purposes, scans found in the root <em>sub-01</em> directory have been downsampled to 2x2x2 mm3 (MPRAGE), and to 3x3x3 mm3 (DSI and rfMRI) with the ResampleScalarVolume module of Slicer 4.6.2. A copy of the output produced in the terminal by Slicer has been created in the `code/` directory.</li> <li>Original data have been placed in <em>sourcedata/</em> in concordance to BIDS.</li> </ul> <p> </p> <p> </p> <p> </p> <p> </p>
LifeSnaps: a 4-month multi-modal dataset capturing unobtrusive snapshots of our lives in the wild
<p><strong>LifeSnaps Dataset Documentation</strong></p> <blockquote> <p>Ubiquitous self-tracking technologies have penetrated various aspects of our lives, from physical and mental health monitoring to fitness and entertainment. Yet, limited data exist on the association between in the wild large-scale physical activity patterns, sleep, stress, and overall health, and behavioral patterns and psychological measurements due to challenges in collecting and releasing such datasets, such as waning user engagement, privacy considerations, and diversity in data modalities. In this paper, we present the <strong>LifeSnaps dataset</strong>, a multi-modal, longitudinal, and geographically-distributed dataset, containing a plethora of anthropological data, collected unobtrusively for the total course of more than 4 months by n=71 participants, under the <a href="https://rais-itn.eu/">European H2020 RAIS project</a>. LifeSnaps contains more than 35 different data types from second to daily granularity, totaling more than 71M rows of data. The participants contributed their data through numerous validated surveys, real-time ecological momentary assessments, and a Fitbit Sense smartwatch, and consented to make these data available openly to empower future research. We envision that releasing this large-scale dataset of multi-modal real-world data, will open novel research opportunities and potential applications in the fields of medical digital innovations, data privacy and valorization, mental and physical well-being, psychology and behavioral sciences, machine learning, and human-computer interaction.</p> </blockquote> <p> </p> <p>The following instructions will get you started with the LifeSnaps dataset and are complementary to the original publication.</p> <p><strong>Data Import: Reading CSV</strong></p> <p>For ease of use, we provide CSV files containing Fitbit, SEMA, and survey data at daily and/or hourly granularity. You can read the files via any programming language. For example, in Python, you can read the files into a Pandas DataFrame with the <a href="https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html">pandas.read_csv()</a> command.</p> <p><strong>Data Import: Setting up a MongoDB (Recommended)</strong></p> <p>To take full advantage of the LifeSnaps dataset, we recommend that you use the raw, complete data via importing the LifeSnaps MongoDB database.</p> <p>To do so, open the terminal/command prompt and run the following command for each collection in the DB. Ensure you have MongoDB Database Tools installed from <a href="https://www.mongodb.com/try/download/database-tools">here</a>.</p> <p>For the Fitbit data, run the following:</p> <pre><code>mongorestore --host localhost:27017 -d rais_anonymized -c fitbit <path to file fitbit.bson></code></pre> <p>For the SEMA data, run the following:</p> <pre><code>mongorestore --host localhost:27017 -d rais_anonymized -c sema <path to file sema.bson></code></pre> <p>For surveys data, run the following:</p> <pre><code>mongorestore --host localhost:27017 -d rais_anonymized -c surveys <path to file surveys.bson></code></pre> <p>If you have access control enabled, then you will need to add the --username and --password parameters to the above commands.</p> <p><strong>Data Availability</strong></p> <p>The MongoDB database contains three collections, fitbit, sema, and surveys, containing the Fitbit, SEMA3, and survey data, respectively. Similarly, the CSV files contain related information to these collections. Each document in any collection follows the format shown below:</p> <pre><code>{ _id: <ObjectId> id (or user_id): <ObjectId> type: <String> data: <Object> }</code></pre> <p>Each document consists of four fields: id (also found as user_id in sema and survey collections), type, and data. The _id field is the MongoDB-defined primary key and can be ignored. The id field refers to a user-specific ID used to uniquely identify each user across all collections. The type field refers to the specific data type within the collection, e.g., steps, heart rate, calories, etc. The data field contains the actual information about the document e.g., steps count for a specific timestamp for the steps type, in the form of an embedded object. The contents of the data object are type-dependent, meaning that the fields within the data object are different between different types of data. As mentioned previously, all times are stored in local time, and user IDs are common across different collections. For more information on the available data types, see the related publication.</p> <p><strong>Surveys Encoding</strong></p> <p><strong>BREQ2</strong></p> <p><em>Why do you engage in exercise?</em></p> <table> <tbody> <tr> <td>Code</td> <td>Text</td> </tr> <tr> <td>engage[SQ001]</td> <td>I exercise because other people say I should</td> </tr> <tr> <td>engage[SQ002]</td> <td>I feel guilty when I don’t exercise</td> </tr> <tr> <td>engage[SQ003]</td> <td>I value the benefits of exercise</td> </tr> <tr> <td>engage[SQ004]</td> <td>I exercise because it’s fun</td> </tr> <tr> <td>engage[SQ005]</td> <td>I don’t see why I should have to exercise</td> </tr> <tr> <td>engage[SQ006]</td> <td>I take part in exercise because my friends/family/partner say I should</td> </tr> <tr> <td>engage[SQ007]</td> <td>I feel ashamed when I miss an exercise session</td> </tr> <tr> <td>engage[SQ008]</td> <td>It’s important to me to exercise regularly</td> </tr> <tr> <td>engage[SQ009]</td> <td>I can’t see why I should bother exercising</td> </tr> <tr> <td>engage[SQ010]</td> <td>I enjoy my exercise sessions</td> </tr> <tr> <td>engage[SQ011]</td> <td>I exercise because others will not be pleased with me if I don’t</td> </tr> <tr> <td>engage[SQ012]</td> <td>I don’t see the point in exercising</td> </tr> <tr> <td>engage[SQ013]</td> <td>I feel like a failure when I haven’t exercised in a while</td> </tr> <tr> <td>engage[SQ014]</td> <td>I think it is important to make the effort to exercise regularly</td> </tr> <tr> <td>engage[SQ015]</td> <td>I find exercise a pleasurable activity</td> </tr> <tr> <td>engage[SQ016]</td> <td>I feel under pressure from my friends/family to exercise</td> </tr> <tr> <td>engage[SQ017]</td> <td>I get restless if I don’t exercise regularly</td> </tr> <tr> <td>engage[SQ018]</td> <td>I get pleasure and satisfaction from participating in exercise</td> </tr> <tr> <td>engage[SQ019]</td> <td>I think exercising is a waste of time</td> </tr> </tbody> </table> <p><strong>PANAS</strong></p> <p><em>Indicate the extent you have felt this way over the past week </em></p> <table> <tbody> <tr> <td>P1[SQ001]</td> <td>Interested</td> </tr> <tr> <td>P1[SQ002]</td> <td>Distressed</td> </tr> <tr> <td>P1[SQ003]</td> <td>Excited</td> </tr> <tr> <td>P1[SQ004]</td> <td>Upset</td> </tr> <tr> <td>P1[SQ005]</td> <td>Strong</td> </tr> <tr> <td>P1[SQ006]</td> <td>Guilty</td> </tr> <tr> <td>P1[SQ007]</td> <td>Scared</td> </tr> <tr> <td>P1[SQ008]</td> <td>Hostile</td> </tr> <tr> <td>P1[SQ009]</td> <td>Enthusiastic</td> </tr> <tr> <td>P1[SQ010]</td> <td>Proud</td> </tr> <tr> <td>P1[SQ011]</td> <td>Irritable</td> </tr> <tr> <td>P1[SQ012]</td> <td>Alert</td> </tr> <tr> <td>P1[SQ013]</td> <td>Ashamed</td> </tr> <tr> <td>P1[SQ014]</td> <td>Inspired</td> </tr> <tr> <td>P1[SQ015]</td> <td>Nervous</td> </tr> <tr> <td>P1[SQ016]</td> <td>Determined</td> </tr> <tr> <td>P1[SQ017]</td> <td>Attentive</td> </tr> <tr> <td>P1[SQ018]</td> <td>Jittery</td> </tr> <tr> <td>P1[SQ019]</td> <td>Active</td> </tr> <tr> <td>P1[SQ020]</td> <td>Afraid</td> </tr> </tbody> </table> <p><strong>Personality</strong></p> <p><em>How Accurately Can You Describe Yourself?</em></p> <table> <tbody> <tr> <td>Code</td> <td>Text</td> </tr> <tr> <td>ipip[SQ001]</td> <td>Am the life of the party.</td> </tr> <tr> <td>ipip[SQ002]</td> <td>Feel little concern for others.</td> </tr> <tr> <td>ipip[SQ003]</td> <td>Am always prepared.</td> </tr> <tr> <td>ipip[SQ004]</td> <td>Get stressed out easily.</td> </tr> <tr> <td>ipip[SQ005]</td> <td>Have a rich vocabulary.</td> </tr> <tr> <td>ipip[SQ006]</td> <td>Don't talk a lot.</td> </tr> <tr> <td>ipip[SQ007]</td> <td>Am interested in people.</td> </tr> <tr> <td>ipip[SQ008]</td> <td>Leave my belongings around.</td> </tr> <tr> <td>ipip[SQ009]</td> <td>Am relaxed most of the time.</td> </tr> <tr> <td>ipip[SQ010]</td> <td>Have difficulty understanding abstract ideas.</td> </tr> <tr> <td>ipip[SQ011]</td> <td>Feel comfortable around people.</td> </tr> <tr> <td>ipip[SQ012]</td> <td>Insult people.</td> </tr> <tr> <td>ipip[SQ013]</td> <td>Pay attention to details.</td> </tr> <tr> <td>ipip[SQ014]</td> <td>Worry about things.</td> </tr> <tr> <td>ipip[SQ015]</td> <td>Have a vivid imagination.</td> </tr> <tr> <td>ipip[SQ016]</td> <td>Keep in the background.</td> </tr> <tr> <td>ipip[SQ017]</td> <td>Sympathize with others' feelings.</td> </tr> <tr> <td>ipip[SQ018]</td> <td>Make a mess of things.</td> </tr> <tr> <td>ipip[SQ019]</td> <td>Seldom feel blue.</td> </tr> <tr> <td>ipip[SQ020]</td> <td>Am not interested in abstract ideas.</td> </tr> <tr> <td>ipip[SQ021]</td> <td>Start conversations.</td> </tr> <tr> <td>ipip[SQ022]</td> <td>Am not interested in other people's problems.</td> </tr> <tr> <td>ipip[SQ023]</td> <td>Get chores done right away.</td> </tr> <tr> <td>ipip[SQ024]</td> <td>Am easily disturbed.</td> </tr> <tr> <td>ipip[SQ025]</td> <td>Have excellent ideas.</td> </tr> <tr> <td>ipip[SQ026]</td> <td>Have little to say.</td> </tr> <tr> <td>ipip[SQ027]</td> <td>Have a soft heart.</td> </tr> <tr> <td>ipip[SQ028]</td> <td>Often forget to put things back in their proper place.</td> </tr> <tr> <td>ipip[SQ029]</td> <td>Get upset easily.</td> </tr> <tr> <td>ipip[SQ030]</td> <td>Do not have a good imagination.</td> </tr> <tr> <td>ipip[SQ031]</td> <td>Talk to a lot of different people at parties.</td> </tr> <tr> <td>ipip[SQ032]</td> <td>Am not really interested in others.</td> </tr> <tr> <td>ipip[SQ033]</td> <td>Like order.</td> </tr> <tr> <td>ipip[SQ034]</td> <td>Change my mood a lot.</td> </tr> <tr> <td>ipip[SQ035]</td> <td>Am quick to understand things.</td> </tr> <tr> <td>ipip[SQ036]</td> <td>Don't like to draw attention to myself.</td> </tr> <tr> <td>ipip[SQ037]</td> <td>Take time out for others.</td> </tr> <tr> <td>ipip[SQ038]</td> <td>Shirk my duties.</td> </tr> <tr> <td>ipip[SQ039]</td> <td>Have frequent mood swings.</td> </tr> <tr> <td>ipip[SQ040]</td> <td>Use difficult words.</td> </tr> <tr> <td>ipip[SQ041]</td> <td>Don't mind being the centre of attention.</td> </tr> <tr> <td>ipip[SQ042]</td> <td>Feel others' emotions.</td> </tr> <tr> <td>ipip[SQ043]</td> <td>Follow a schedule.</td> </tr> <tr> <td>ipip[SQ044]</td> <td>Get irritated easily.</td> </tr> <tr> <td>ipip[SQ045]</td> <td>Spend time reflecting on things.</td> </tr> <tr> <td>ipip[SQ046]</td> <td>Am quiet around strangers.</td> </tr> <tr> <td>ipip[SQ047]</td> <td>Make people feel at ease.</td> </tr> <tr> <td>ipip[SQ048]</td> <td>Am exacting in my work.</td> </tr> <tr> <td>ipip[SQ049]</td> <td>Often feel blue.</td> </tr> <tr> <td>ipip[SQ050]</td> <td>Am full of ideas.</td> </tr> </tbody> </table> <p><strong>STAI</strong></p> <p><em>Indicate how you feel right now</em></p> <table> <tbody> <tr> <td>Code</td> <td>Text</td> </tr> <tr> <td>STAI[SQ001]</td> <td>I feel calm</td> </tr> <tr> <td>STAI[SQ002]</td> <td>I feel secure</td> </tr> <tr> <td>STAI[SQ003]</td> <td>I am tense</td> </tr> <tr> <td>STAI[SQ004]</td> <td>I feel strained</td> </tr> <tr> <td>STAI[SQ005]</td> <td>I feel at ease</td> </tr> <tr> <td>STAI[SQ006]</td> <td>I feel upset</td> </tr> <tr> <td>STAI[SQ007]</td> <td>I am presently worrying over possible misfortunes</td> </tr> <tr> <td>STAI[SQ008]</td> <td>I feel satisfied</td> </tr> <tr> <td>STAI[SQ009]</td> <td>I feel frightened</td> </tr> <tr> <td>STAI[SQ010]</td> <td>I feel comfortable</td> </tr> <tr> <td>STAI[SQ011]</td> <td>I feel self-confident</td> </tr> <tr> <td>STAI[SQ012]</td> <td>I feel nervous</td> </tr> <tr> <td>STAI[SQ013]</td> <td>I am jittery</td> </tr> <tr> <td>STAI[SQ014]</td> <td>I feel indecisive</td> </tr> <tr> <td>STAI[SQ015]</td> <td>I am relaxed</td> </tr> <tr> <td>STAI[SQ016]</td> <td>I feel content</td> </tr> <tr> <td>STAI[SQ017]</td> <td>I am worried</td> </tr> <tr> <td>STAI[SQ018]</td> <td>I feel confused</td> </tr> <tr> <td>STAI[SQ019]</td> <td>I feel steady</td> </tr> <tr> <td>STAI[SQ020]</td> <td>I feel pleasant</td> </tr> </tbody> </table> <p><strong>TTM</strong></p> <p><em>Do you engage in regular physical activity according to the definition above? How frequently did each event or experience occur in the past month?</em></p> <table> <tbody> <tr> <td>Code</td> <td>Text</td> </tr> <tr> <td>processes[SQ002]</td> <td>I read articles to learn more about physical activity.</td> </tr> <tr> <td>processes[SQ003]</td> <td>I get upset when I see people who would benefit from physical activity but choose not to do physical activity.</td> </tr> <tr> <td>processes[SQ004]</td> <td>I realize that if I don't do physical activity regularly, I may get ill and be a burden to others.</td> </tr> <tr> <td>processes[SQ005]</td> <td>I feel more confident when I do physical activity regularly.</td> </tr> <tr> <td>processes[SQ006]</td> <td>I have noticed that many people know that physical activity is good for them.</td> </tr> <tr> <td>processes[SQ007]</td> <td>When I feel tired, I make myself do physical activity anyway because I know I will feel better afterwards.</td> </tr> <tr> <td>processes[SQ008]</td> <td>I have a friend who encourages me to do physical activity when I don't feel up to it.</td> </tr> <tr> <td>processes[SQ009]</td> <td>One of the rewards of regular physical activity is that it improves my mood.</td> </tr> <tr> <td>processes[SQ010]</td> <td>I tell myself that I can keep doing physically activity if I try hard enough.</td> </tr> <tr> <td>processes[SQ011]</td> <td>I keep a set of physical activity clothes with me so I can do physical activity whenever I get the time.</td> </tr> <tr> <td>processes[SQ012]</td> <td>I look for information related to physical activity.</td> </tr> <tr> <td>processes[SQ013]</td> <td>I am afraid of the results to my health if I do not do physical activity.</td> </tr> <tr> <td>processes[SQ014]</td> <td>I think that by doing regular physical activity I will not be a burden to the healthcare system.</td> </tr> <tr> <td>processes[SQ015]</td> <td>I believe that regular physical activity will make me a healthier, happier person.</td> </tr> <tr> <td>processes[SQ016]</td> <td>I am aware of more and more people who are making physical activity a part of their lives.</td> </tr> <tr> <td>processes[SQ017]</td> <td>Instead of taking a nap after work, I do physical activity.</td> </tr> <tr> <td>processes[SQ018]</td> <td>I have someone who encourages me to do physical activity. </td> </tr> <tr> <td>processes[SQ019]</td> <td>I try to think of physical activity as a time to clear my mind as well as a workout for my body.</td> </tr> <tr> <td>processes[SQ020]</td> <td>I make commitments to do physical activity.</td> </tr> <tr> <td>processes[SQ021]</td> <td>I use my calendar to schedule my physical activity time.</td> </tr> <tr> <td>processes[SQ022]</td> <td>I find out about new methods of being physically active.</td> </tr> <tr> <td>processes[SQ023]</td> <td>I get upset when I realize that people I love would have better health if they were physically active.</td> </tr> <tr> <td>processes[SQ024]</td> <td>I think that regular physical activity plays a role in reducing health care costs.</td> </tr> <tr> <td>processes[SQ025]</td> <td>I feel better about myself when I do physical activity.</td> </tr> <tr> <td>processes[SQ026]</td> <td>I notice that famous people often say that they do physical activity regularly.</td> </tr> <tr> <td>processes[SQ027]</td> <td>Instead of relaxing by watching TV or eating, I take a walk or am physically active.</td> </tr> <tr> <td>processes[SQ028]</td> <td>My friends encourage me to do physical activity.</td> </tr> <tr> <td>processes[SQ029]</td> <td>If I engage in regular physical activity, I find that I get the benefit of having more energy.</td> </tr> <tr> <td>processes[SQ030]</td> <td>I believe that I can do physical activity regularly. </td> </tr> <tr> <td>processes[SQ031]</td> <td>I make sure I always have a clean set of physical activity clothes.</td> </tr> </tbody> </table> <p><strong>Files Description</strong></p> <p><em>Scored Surveys:</em> CSV files containing scored versions of BREQ-2, IPIP, PANAS, STAI, and TTM surveys in scored_surveys folder</p> <p><em>Fitbit & EMA Data (daily granularity):</em> csv_rais_anonymized/daily_fitbit_sema_df_unprocessed.csv</p> <p><em>Fitbit & EMA Data (hourly granularity):</em> csv_rais_anonymized/hourly_fitbit_sema_df_unprocessed.csv</p> <p><em>MongoDB Dumps (compressed for practicality):</em> mongo_rais_anonymized/fitbit.bson for Fitbit data, mongo_rais_anonymized/sema for EMA data, and mongo_rais_anonymized/surveys for raw surveys data. </p> <p><strong>Code Availability</strong></p> <p><em>Data Anonymization:</em> <a href="https://github.com/syfantid/RAIS-Anonymization">https://github.com/syfantid/RAIS-Anonymization</a></p> <p><em>Exploratory Data Analysis:</em> <a href="https://github.com/kcristinaa/LifeSnaps-EDA">https://github.com/kcristinaa/LifeSnaps-EDA</a></p> <p><strong>Related Publications</strong></p> <p>Sofia Yfantidou, Christina Karagianni, Stefanos Efstathiou, Athena Vakali, Joao Palotti, Dimitrios Panteleimon Giakatos, Thomas Marchioro, Andrei Kazlouski, Elena Ferrari, and Sarunas Girdzijauskas, 2022, LifeSnaps: a 4-month multi-modal dataset capturing unobtrusive snapshots of our lives in the wild (Submitted for peer-review).</p> <p><strong>Acknowledgments</strong></p> <p>This project has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grant agreement No 813162. The content of this paper reflects only the authors' view and the Agency and the Commission are not responsible for any use that may be made of the information it contains. First and foremost, the authors would like to thank the participants of the LifeSnaps study who agreed to share their data for scientific advancement. The authors would like to further thank the web developers, T. Valk and S. Karamanidis, for their contribution to the project, all past and present RAIS fellows for their help with participants' recruitment, G. Pallis and M. Christodoulaki for their support with the ethics committee application, and B. Carminati for her feedback on data anonymization and privacy considerations.</p>
AdsMT: Multi-modal Transformer for Predicting Global Minimum Adsorption Energy
<p>We built three Global Minimum Adsorption Energy (GMAE) benchmark datasets named OCD-GMAE, Alloy-GMAE and FG-GMAE from OC20-Dense, Catalysis Hub, and `functional groups' (FG)-dataset datasets through strict data cleaning, and each data point represents a unique combination of catalyst surface and adsorbate. These new benchmark datasets can be beneficial for future ML study on GMAE prediction.</p> <p>In addition, a similar data cleaning procedure was employed on the OC20 dataset to create a new dataset named OC20-LMAE, which comprises surface/adsorbate pairings along with their local minimum adsorption energies (LMAE). The OC20-LMAE dataset contains 363,937 data points and serves as an effective resource for model pretraining.</p>
Sensor-based Pallet Activity Recognition in Logistics (SPARL Version 2) - A multi-modal Dataset
<p>SPARL is a freely accessible data set for sensor-based activity recognition of pallets in logistics. The data set consists of 20 recordings from three scenarios. A description of the scenarios can be found in the protocol file.</p> <p>Four different sensors were used simultaneously for all recordings:</p> <ul> <li>MSR Electronics MSR 145 <ul> <li>Sampling rate 50 Hz</li> </ul> </li> <li>MBIENTLAB MetaMotionS <ul> <li>Sampling rate 100 Hz</li> </ul> </li> <li>Kistler KiDaQ Module 5512A <ul> <li>Sampling rate 100 kHz</li> <li>the raw data is also downsampled to 5 kHz and 20 kHz for easier processing </li> </ul> </li> <li>Holybro Flightcontroller PX4FMU <ul> <li>The board uses two accelerometers and two gyroscopes, all with a sampling rate of 1000 Hz <ul> <li>Accelerometer 1: IvenSense MPU6000 </li> <li>Accelerometer 2: STMicroelectronics LSM303D </li> <li>Gyroscope 1: IvenSense MPU6000</li> <li>Gyroscope 2: STMicroelectronics L3GD20</li> </ul> </li> </ul> </li> </ul> <p>The recordings were accompanied by three logitech Mevo Start cameras, of which all recordings are included anonymously in the data set. </p> <p>The videos were annotated by one person in each frame. For this purpose, the annotation tool SARA was used, which can be found <a href="../records/8189341">here</a>. The JSON schema used for annotation is also included in the SPARL dataset. The R code used our evaluation can be found in <a title="https://github.com/bommert/WGTL24" href="https://github.com/bommert/WGTL24">GitHub</a>.</p> <p>If you have any questions about the dataset, please contact: sven.franke@tu-dortmund.de</p> <p><strong>If you use this dataset for research, please cite the following paper: “Data-driven, sensor-based taxonomy for environmental life cycle assessment of pallets”, Nr. 20 (2024): Logistics Journal: Proceedings, DOI: <a href="http://dx.doi.org/10.2195/lj_proc_franke_en_202410_01" target="_blank" rel="noopener">10.2195/lj_proc_franke_en_202410_01</a></strong></p>
REHAB24-6: A multi-modal dataset of physical rehabilitation exercises
<p>To enable the evaluation of HPE models and the development of exercise feedback systems, we produced a new rehabilitation dataset (REHAB24-6). The main focus is on a diverse range of exercises, views, body heights, lighting conditions, and exercise mistakes. With the publicly available RGB videos, skeleton sequences, repetition segmentation, and exercise correctness labels, this dataset offers the most comprehensive testbed for exercise-correctness-related tasks.</p> <h2>Contents</h2> <ul> <li>65 recordings (184,825 frames, 30 FPS): <ul> <li>RGB videos from two cameras (<code>videos.zip</code>, horizontal = Camera17, vertical = Camera18);</li> <li>3D and 2D projected positions of 41 motion capture marker (<code><2/3>d_markers.zip</code>, marker labels in <code>marker_names.txt</code>);</li> <li>3D and 2D projected positions of 26 skeleton joints (<code><2/3>d_joints.zip</code>, joint labels in <code>joint_names.txt</code>);</li> </ul> </li> <li>Annotation of 1,072 exercise repetitions (<code>Segmentation.csv</code>, indexed based <strong>only on</strong> 30 FPS data, described in <code>Segmentation.txt</code>): <ul> <li>Temporal segmentation (start/end frame, most between 2–5 seconds);</li> <li>Binary correctness label (around 90 from each category in each exercise, except Ex3 with around 50);</li> <li>Exercise direction (around 90 from each direction in each exercise);</li> <li>Lighting conditions label.</li> </ul> </li> </ul> <h2>Recording Conditions</h2> <p>Our laboratory setup included 18 synchronized sensors (2 RGB video cameras, 16 ultra-wide motion capture cameras) spread around an 8.2 × 7 m room. The RGB cameras were located in the corners of the room, one in a horizontal position (hor.), providing a larger field of view (FoV), and one in a vertical (ver.), resulting in a narrower FoV. Both types of cameras were synchronized with a sampling frequency of 30 frames per second (FPS).</p> <p>The subjects wore motion capture body suits with 41 markers attached to them, which were detected by optical cameras. The OptiTrack Motive 2.3.0 software inferred the 3D positions of the markers in virtual centimeters and converted them into a skeleton with 26 joints, forming our human pose 3D ground truth (GT).</p> <p>To acquire a 2D version of the ground truth in pixel coordinates, we applied a projection of the virtual coordinates into the camera using the simplified pinhole model. We estimated the parameters for this projection as follows. First, the virtual position of the cameras was estimated using measuring tape and knowledge of the virtual origin. Then, the orientation of the cameras was optimized by matching the virtual marker positions with their position in the videos.</p> <p>We also simulated changes in lighting conditions: a few videos were shot in the natural evening light, which resulted in worse visibility, while the rest were under artificial lighting.</p> <h2>Exercises</h2> <p>10 subjects participated in our recording and consented to release the data publicly: 6 males and 4 females of different ages (from 25 to 50) and fitness levels. A physiotherapist instructed the subjects on how to perform the exercises so that at least five repetitions were done in what he deemed the correct way and five more incorrectly. The participants had a certain degree of freedom, e.g., in which leg they used in Ex4 and Ex5. Similarly, the physiotherapist suggested different exercise mistakes for each subject.</p> <ul> <li><strong>Ex1 = Arm abduction</strong>: sideway raising of the straightened right arm;</li> <li><strong>Ex2 = Arm VW</strong>: fluent transition of arms between V (arms straight up) and W (elbows down, hands up) shape;</li> <li><strong>Ex3 = Push-ups</strong>: push-ups with hands on a table;</li> <li><strong>Ex4 = Leg abduction</strong>: sideway raising of the straightened leg;</li> <li><strong>Ex5 = Leg lunge</strong>: pushing a knee of the back leg down while keeping a right angle on the front knee;</li> <li><strong>Ex6 = Squats</strong>.</li> </ul> <p>Every exercise was also executed in two directions, resulting in different views of the subject depending on the camera. Facing the horizontal camera resulted in a front view for that camera and a profile from the other. Facing the wall between the cameras shows the subject from half-profile in both cameras. A rare direction, only used for push-ups due to the use of the table, was facing the vertical camera, with the views being reversed compared to the first orientation.</p> <h2>Citation</h2> <p>Cite the related conference paper:</p> <p>Černek, A., Sedmidubsky, J., Budikova P.: REHAB24-6: Physical Therapy Dataset for Analyzing Pose Estimation Methods. 17th International Conference on Similarity Search and Applications (SISAP). Springer, 14 pages, 2024.</p> <h2>License</h2> <p>This dataset is for academic or non-profit organization noncomercial research use only. By using you agree to appropriately reference the paper above in any publication making of its use. For comercial purposes contact us at info@visioncraft.ai</p>
A foundation model enpowered by a multi-modal prompt engine for universal seismic geobody interpretation across surveys
<p>A multi-type geobody dataset for training SAG model, including channel, paloekarst, salt body, and so on.</p> <p>A foundation model enpowered by a multi-modal prompt engine for universal seismic geobody interpretation across surveys (<a href="https://arxiv.org/abs/2409.04962">[2409.04962] A foundation model enpowered by a multi-modal prompt engine for universal seismic geobody interpretation across surveys (arxiv.org)</a>)</p> <p> </p>
Alternation emerges as a multi-modal strategy for turbulent odor navigation - Dataset
<p>Dataset from 3D direct numerical simulation of odor evolution in a turbulent channel flow.</p> <p>The dataset contains two .mat files with nose (z ~ 50 cm) and ground (z = 0) level 2D slices;</p> <p>coordinates.mat with the coordinates in X and Y directions;</p> <p>and a Jupyter Notebook file (Read_nose_ground_dataset) to read and plot the 2D fields.<br><br>Updated on May 20th 2025<br>3D velocity data have been added to the dataset.</p>
MOVING: a Multi-MOdal dataset of EEG signals and VIrtual Glove hand trackING
<p>A new Multi-modal dataset comprising neural EEG signals and kinematic data associated with three hand movements — open/close, finger tapping, and wrist rotation — along with a rest period. The dataset, obtained from eleven subjects using a 32-channel dry wireless EEG system, also includes synchronized kinematic data captured by a Virtual Glove (VG) system equipped with two orthogonal Leap Motion Controllers. The use of these two devices allows for fast assembly (~ 1 minute) while introducing more noise than the gold standard devices for data acquisition. The data set, obtained from 11 subjects using a 32-channel dry wireless EEG system, also includes synchronized kinematic data captured by a Virtual Glove (VG) system equipped with two orthogonal Leap Motion Controllers. </p> <p>For citation please refer to the paper:<br>Mattei, E.; Lozzi, D.; Di Matteo, A.; Cipriani, A.; Manes, C.; Placidi, G. MOVING: A Multi-Modal Dataset of EEG Signals and Virtual Glove Hand Tracking. Sensors 2024,24, 5207. https://doi.org/10.3390/s24165207 </p> <p><strong>References</strong>:</p> <p>Placidi, Giuseppe. "<em>A smart virtual glove for the hand telerehabilitation.</em>" Computers in Biology and Medicine 37.8 (2007): 1100-1107.</p> <p>Placidi, Giuseppe, et al. "<em>Measurements by a LEAP-based virtual glove for the hand rehabilitation.</em>" Sensors 18.3 (2018): 834.</p> <p>Placidi, Giuseppe, et al. "<em>Patient–therapist cooperative hand telerehabilitation through a novel framework involving the virtual glove system.</em>" Sensors 23.7 (2023): 3463.</p>
WildfireSpreadTS: A dataset of multi-modal time series for wildfire spread prediction
<p>We present a <strong>multi-temporal</strong>, <strong>multi-modal</strong> remote-sensing dataset for predicting <strong>how active wildfires will spread</strong> at a resolution of 24 hours. The dataset consists of <strong>13.607 images</strong> across 607 fire events in the United States from January 2018 to October 2021. For each fire event, the dataset contains a <strong>full time series of daily observations</strong>, containing detected active fires and variables related to <strong>fuel, topography and weather conditions</strong>.</p><h2>Documentation</h2><p><i><strong>WildfireSpreadTS_Documentation.pdf</strong></i> includes further details about the dataset, following Gebru et al.'s <strong>"Datasheets for Datasets"</strong> framework. This documentation is similar to the supplementary material of the associated NeurIPS paper, excluding only information about experimental setup and results. For full details, please refer to the associated paper. </p><h2>Code: Getting started</h2><p>Get started working with the dataset at <a href="https://github.com/SebastianGer/WildfireSpreadTS">https://github.com/SebastianGer/WildfireSpreadTS</a>. </p><p>The code includes a <strong>PyTorch Dataset</strong> and <strong>Lightning DataModule </strong>to allow for easy access. We recommend converting the GeoTIFF files provided here to HDF5 files (bigger files, but much faster). The necessary code is also available in the repository.</p><p> </p><p>This work is funded by Digital Futures in the project EO-AI4GlobalChange. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at C3SE partially funded by the Swedish Research Council through grant agreement no. 2022-06725.</p>
VISIONE Feature Repository for VBS: Multi-Modal Features and Detected Objects from VBSLHE Dataset
<p>This repository contains a diverse set of features extracted from the VBSLHE dataset (laparoscopic gynecology) . These features will be utilized in the VISIONE system [Amato et al. 2023, Amato et al. 2022] in the next editions of the Video Browser Showdown (VBS) competition (<a href="https://www.videobrowsershowdown.org/">https://www.videobrowsershowdown.org/</a>). </p> <p>We used a snapshot of the dataset provided by the Medical University of Vienna and Toronto that can be downloaded using the instructions provided at <a href="https://download-dbis.dmi.unibas.ch/mvk/">https://download-dbis.dmi.unibas.ch/mvk/</a>. It comprises 75 video files. We divided each video into video shots with a maximum duration of 5 seconds.</p> <p>This repository is released under a Creative Commons Attribution license. If you use it in any form for your work, please cite the following paper:</p> <blockquote> <p>@inproceedings{amato2023visione, title={VISIONE at Video Browser Showdown 2023}, author={Amato, Giuseppe and Bolettieri, Paolo and Carrara, Fabio and Falchi, Fabrizio and Gennaro, Claudio and Messina, Nicola and Vadicamo, Lucia and Vairo, Claudio}, booktitle={International Conference on Multimedia Modeling}, pages={615--621}, year={2023}, organization={Springer} } </p> </blockquote> <p> </p> <p>This repository (v2) comprises the following files:</p> <ul> <li><em><strong>msb.tar.gz </strong></em> contains tab-separated files (.tsv) for each video. Each tsv file reports, for each video segment, the timestamp and frame number marking the start/end of the video segment, along with the timestamp of the extracted middle frame and the associated identifier ("id_visione").</li> <li><em><strong>extract-keyframes-from-msb.tar.gz</strong></em> contains a Python script designed to extract the middle frame of each video segment from the MSB files. To run the script successfully, please ensure that you have the original VBSLHE videos available.</li> <li><em><strong>features-aladin.tar.gz†</strong></em><strong> </strong>contains <a href="https://github.com/mesnico/ALADIN">ALADIN</a> [Messina N. et al. 2022] features extracted for all the segment's middle frames.</li> <li><em><strong>features-clip-laion.tar.gz†</strong></em> contains <a href="https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K">CLIP ViT-H/14 - LAION-2B </a>[Schuhmann et al. 2022] features extracted for all the segment's middle frames.</li> <li><em><strong>features-clip-openai.tar.gz† </strong></em>contains <a href="https://huggingface.co/openai/clip-vit-large-patch14">CLIP ViT-L/14</a> [Radford et al. 2021] features extracted for all the segment's middle frames.</li> <li><em><strong>features-clip2video.tar.gz† </strong></em>contains <a href="https://github.com/CryhanFang/CLIP2Video">CLIP2Video</a> [Fang H. et al. 2021] extracted for all the video segments. <strong> </strong></li> <li><em><strong>objects-frcnn-oiv4.tar.gz* </strong></em>contains the objects detected using <a href="http://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1">Faster R-CNN+Inception ResNet</a> (trained on the Open Images V4 [Kuznetsova et al. 2020]).</li> <li><em><strong>objects-mrcnn-lvis.tar.gz*</strong></em> contains the objects detected using Mask R-CNN [He et al. 2017] (trained on LVIS).</li> <li><em><strong>objects-vfnet64-coco.tar.gz*</strong></em> contains the objects detected using VfNet [Zhang et al. 2021] (trained on COCO dataset).</li> </ul> <p>*Please be sure to use the <strong>v2 version </strong>of this repository, since v1 feature files may contain inconsistencies that have now been corrected</p> <p><em><strong>*Note on the object annotations:</strong></em> Within an object archive, there is a jsonl file for each video, where each row contains a record of a video segment (the <em>"_id"</em> corresponds to the <em>"id_visione"</em> used in the msb.tar.gz) . Additionally, there are three arrays representing the objects detected, the corresponding scores, and the bounding boxes. The format of these arrays is as follows:</p> <ul> <li><em>"object_class_names"</em>: vector with the class name of each detected object.</li> <li><em>"object_scores"</em>: scores corresponding to each detected object.</li> <li><em>"object_boxes_yxyx"</em>: bounding boxes of the detected objects in the format <em>(ymin, xmin, ymax, xmax).</em></li> </ul> <p> </p> <p><em><strong>†Note on the cross-modal features: </strong></em>The extracted multi-modal features (ALADIN, CLIPs, CLIP2Video) enable internal searches within the VBSLHE dataset using the query-by-image approach (features can be compared with the dot product). However, to perform searches based on free text, the text needs to be transformed into the joint embedding space according to the specific network being used (see links above). Please be aware that t<strong>he service for transforming text into features is not provided within this repository and should be developed independently using the original feature repositories linked above.</strong></p> <p>We have plans to release the code in the future, allowing the reproduction of the VISIONE system, including the instantiation of all the services to transform text into cross-modal features. However, this work is still in progress, and the code is not currently available.</p> <p> </p> <p><strong>References:</strong></p> <p>[Amato et al. 2023] Amato, G.et al., 2023, January. VISIONE at Video Browser Showdown 2023. In International Conference on Multimedia Modeling (pp. 615-621). Cham: Springer International Publishing.</p> <p>[Amato et al. 2022] Amato, G. et al. (2022). VISIONE at Video Browser Showdown 2022. In: , et al. MultiMedia Modeling. MMM 2022. Lecture Notes in Computer Science, vol 13142. Springer, Cham. </p> <p>[Fang H. et al. 2021] Fang H. et al., 2021. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097.</p> <p>[He et al. 2017] He, K., Gkioxari, G., Dollár, P. and Girshick, R., 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 2961-2969).</p> <p>[Kuznetsova et al. 2020] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A. and Duerig, T., 2020. The open images dataset v4. International Journal of Computer Vision, 128(7), pp.1956-1981.</p> <p>[Lin et al. 2014] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P. and Zitnick, C.L., 2014, September. Microsoft coco: Common objects in context. In European conference on computer vision (pp. 740-755). Springer, Cham.</p> <p>[Messina et al. 2022] Messina N. et al., 2022, September. Aladin: distilling fine-grained alignment scores for efficient image-text matching and retrieval. In Proceedings of the 19th International Conference on Content-based Multimedia Indexing (pp. 64-70).</p> <p>[Radford et al. 2021] Radford A. et al., 2021, July. Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.</p> <p>[Schuhmann et al. 2022] Schuhmann C. et al., 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35, pp.25278-25294.</p> <p>[Zhang et al. 2021] Zhang, H., Wang, Y., Dayoub, F. and Sunderhauf, N., 2021. Varifocalnet: An iou-aware dense object detector. In Proceedings of the IEEE/CV</p>
VISIONE Feature Repository for VBS: Multi-Modal Features and Detected Objects from MVK Dataset
<p>This repository contains a diverse set of features extracted from the marine video (underwater) dataset (MVK) . These features were utilized in the VISIONE system [Amato et al. 2023, Amato et al. 2022] during the latest editions of the Video Browser Showdown (VBS) competition (<a href="https://www.videobrowsershowdown.org/">https://www.videobrowsershowdown.org/</a>). </p> <p>We used a snapshot of the MVK dataset from 2023, that can be downloaded using the instructions provided at <a href="https://download-dbis.dmi.unibas.ch/mvk/">https://download-dbis.dmi.unibas.ch/mvk/</a>. It comprises 1,372 video files. We divided each video into 1 second segments. </p> <p>This repository is released under a Creative Commons Attribution license. If you use it in any form for your work, please cite the following paper:</p> <blockquote> <pre>@inproceedings{amato2023visione, title={VISIONE at Video Browser Showdown 2023}, author={Amato, Giuseppe and Bolettieri, Paolo and Carrara, Fabio and Falchi, Fabrizio and Gennaro, Claudio and Messina, Nicola and Vadicamo, Lucia and Vairo, Claudio}, booktitle={International Conference on Multimedia Modeling}, pages={615--621}, year={2023}, organization={Springer} }</pre> </blockquote> <p> </p> <p>This repository comprises the following files:</p> <ul> <li><strong><em>msb.tar.gz </em></strong> contains tab-separated files (.tsv) for each video. Each tsv file reports, for each video segment, the timestamp and frame number marking the start/end of the video segment, along with the timestamp of the extracted middle frame and the associated identifier ("id_visione"). </li> <li><em><strong>extract-keyframes-from-msb.tar.gz</strong></em> contains a Python script designed to extract the middle frame of each video segment from the MSB files. To run the script successfully, please ensure that you have the original MVK videos available.</li> <li><strong><em>features-aladin.tar.gz<sup>†</sup></em> </strong>contains <a href="https://github.com/mesnico/ALADIN">ALADIN</a> [Messina N. et al. 2022] features extracted for all the segment's middle frames. </li> <li><em><strong>features-clip-laion.tar.gz<sup>†</sup></strong></em> contains <a href="https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K">CLIP ViT-H/14 - LAION-2B </a>[Schuhmann et al. 2022] features extracted for all the segment's middle frames.</li> <li><em><strong>features-clip-openai.tar.gz<sup>†</sup> </strong></em>contains <a href="https://huggingface.co/openai/clip-vit-large-patch14">CLIP ViT-L/14</a> [Radford et al. 2021] features extracted for all the segment's middle frames. </li> <li><em><strong>features-clip2video.tar.gz<sup>†</sup> </strong></em>contains <a href="https://github.com/CryhanFang/CLIP2Video">CLIP2Video</a> [Fang H. et al. 2021] extracted for all the 1s video segments. <strong> </strong></li> <li><em><strong>objects-frcnn-oiv4.tar.gz<sup>*</sup> </strong></em>contains the objects detected using <a href="http://tfhub.dev/google/faster_rcnn/openimages_v4/inception_resnet_v2/1">Faster R-CNN+Inception ResNet</a> (trained on the Open Images V4 [Kuznetsova et al. 2020]). </li> <li><em><strong>objects-mrcnn-lvis.tar.gz<sup>*</sup></strong></em> contains the objects detected using Mask R-CNN [He et al. 2017] (trained on LVIS).</li> <li><em><strong>objects-vfnet64-coco.tar.gz<sup>*</sup></strong></em> contains the objects detected using VfNet [Zhang et al. 2021] (trained on COCO dataset).</li> </ul> <p>*Please be sure to use the <strong>v2 version </strong>of this repository, since v1 feature files may contain inconsistencies that have now been corrected</p> <p><em><strong>*Note on the object annotations:</strong></em> Within an object archive, there is a jsonl file for each video, where each row contains a record of a video segment (the <em>"_id"</em> corresponds to the <em>"id_visione"</em> used in the msb.tar.gz) . Additionally, there are three arrays representing the objects detected, the corresponding scores, and the bounding boxes. The format of these arrays is as follows:</p> <ul> <li><em>"object_class_names"</em>: vector with the class name of each detected object.</li> <li><em>"object_scores"</em>: scores corresponding to each detected object.</li> <li><em>"object_boxes_yxyx"</em>: bounding boxes of the detected objects in the format <em>(ymin, xmin, ymax, xmax).</em></li> </ul> <p> </p> <p><em><strong><sup>†</sup>Note on the cross-modal features: </strong></em>The extracted multi-modal features (ALADIN, CLIPs, CLIP2Video) enable internal searches within the MVK dataset using the query-by-image approach (features can be compared with the dot product). However, to perform searches based on free text, the text needs to be transformed into the joint embedding space according to the specific network being used (see links above). Please be aware that t<strong>he service for transforming text into features is not provided within this repository and should be developed independently using the original feature repositories linked above.</strong></p> <p>We have plans to release the code in the future, allowing the reproduction of the VISIONE system, including the instantiation of all the services to transform text into cross-modal features. However, this work is still in progress, and the code is not currently available.</p> <p> </p> <p><strong>References:</strong></p> <p>[Amato et al. 2023] Amato, G.et al., 2023, January. VISIONE at Video Browser Showdown 2023. In International Conference on Multimedia Modeling (pp. 615-621). Cham: Springer International Publishing.</p> <p>[Amato et al. 2022] Amato, G. et al. (2022). VISIONE at Video Browser Showdown 2022. In: , et al. MultiMedia Modeling. MMM 2022. Lecture Notes in Computer Science, vol 13142. Springer, Cham. </p> <p>[Fang H. et al. 2021] Fang H. et al., 2021. Clip2video: Mastering video-text retrieval via image clip. arXiv preprint arXiv:2106.11097.</p> <p>[He et al. 2017] He, K., Gkioxari, G., Dollár, P. and Girshick, R., 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision (pp. 2961-2969).</p> <p>[Kuznetsova et al. 2020] Kuznetsova, A., Rom, H., Alldrin, N., Uijlings, J., Krasin, I., Pont-Tuset, J., Kamali, S., Popov, S., Malloci, M., Kolesnikov, A. and Duerig, T., 2020. The open images dataset v4. International Journal of Computer Vision, 128(7), pp.1956-1981.</p> <p>[Lin et al. 2014] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P. and Zitnick, C.L., 2014, September. Microsoft coco: Common objects in context. In European conference on computer vision (pp. 740-755). Springer, Cham.</p> <p>[Messina et al. 2022] Messina N. et al., 2022, September. Aladin: distilling fine-grained alignment scores for efficient image-text matching and retrieval. In Proceedings of the 19th International Conference on Content-based Multimedia Indexing (pp. 64-70).</p> <p>[Radford et al. 2021] Radford A. et al., 2021, July. Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.</p> <p>[Schuhmann et al. 2022] Schuhmann C. et al., 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35, pp.25278-25294.</p> <p>[Zhang et al. 2021] Zhang, H., Wang, Y., Dayoub, F. and Sunderhauf, N., 2021. Varifocalnet: An iou-aware dense object detector. In Proceedings of the IEEE/CV</p>
Multi-modal dataset for music genre recognition based on six different modalities for LMD-aligned and SLAC datasets
<p>Multi-modal dataset for music genre recognition based on six different modalities for the LMD-aligned [1] and SLAC [2] datasets. Further details are provided in [3].</p> <p><strong>Descriptions of files</strong></p> <table> <thead> <tr> <th scope="col">Link</th> <th scope="col">Description</th> </tr> </thead> <tbody> <tr> <td><a href="https://zenodo.org/record/5651429/files/LMD-aligned_Filelist.arff">LMD-aligned_Filelist.arff</a></td> <td>File list with 1575 music tracks selected from the LMD-aligned dataset [1] with tagtraum genre annotations [4] (only a subset of LMD-aligned is used, which includes only pieces for which all six modalities were accessible, and which includes only well-represented genres)</td> </tr> <tr> <td><a href="https://zenodo.org/record/5651429/files/LMD-aligned_ExtractedFeatures.tar.gz">LMD-aligned_ExtractedFeatures.tar.gz</a></td> <td>Raw audio signal and model-based features extracted with AMUSE [5]</td> </tr> <tr> <td><a href="https://zenodo.org/record/5651429/files/LMD-aligned_ProcessedFeatures.tar.gz">LMD-aligned_ProcessedFeatures.tar.gz</a></td> <td>Processed features: audio signal and model-based features aggregated for 4 s time frames with 2 s step size / all other features (see the table below) with the same values for all time frames</td> </tr> <tr> <td><a href="https://zenodo.org/record/5651429/files/LMD-aligned_Datasets.tar.gz">LMD-aligned_Datasets.tar.gz</a></td> <td>Training, optimization, and test datasets for 3 splits for the recognition of 5 genres in [3]</td> </tr> <tr> <td><a href="https://zenodo.org/record/5651429/files/SLAC_Filelist.arff">SLAC_Filelist.arff</a></td> <td>File list with 250 music tracks from the SLAC dataset [2] (genres and sub-genres are provided in the folder structure)</td> </tr> <tr> <td><a href="https://zenodo.org/record/5651429/files/SLAC_ExtractedFeatures.tar.gz">SLAC_ExtractedFeatures.tar.gz</a></td> <td>Raw audio signal and model-based features extracted with AMUSE [5]</td> </tr> <tr> <td><a href="https://zenodo.org/record/5651429/files/SLAC_ProcessedFeatures.tar.gz">SLAC_ProcessedFeatures.tar.gz</a></td> <td>Processed features: audio signal and model-based features aggregated for 4 s time frames with 2 s step size / all other features (see the table below) with the same values for all time frames</td> </tr> <tr> <td><a href="https://zenodo.org/record/5651429/files/SLAC_Datasets.tar.gz">SLAC_Datasets.tar.gz</a></td> <td>Training, optimization, and test datasets for 3 splits for the recognition of 5 genres and 10 sub-genres in [3]</td> </tr> </tbody> </table> <p><strong>Modalities and feature sub-groups</strong></p> <table> <thead> <tr> <th scope="col">Modality</th> <th scope="col">Sub-group</th> <th scope="col"> <p>Dimensions in processed</p> <p>features of LMD-aligned</p> </th> <th scope="col"> <p>Dimensions in processed</p> <p>features of SLAC</p> </th> </tr> </thead> <tbody> <tr> <td>Audio signal</td> <td>Low-level</td> <td>1-524</td> <td>1-524</td> </tr> <tr> <td>Audio signal</td> <td>Semantic</td> <td>525-810</td> <td>525-810</td> </tr> <tr> <td>Audio signal</td> <td>Structural complexity</td> <td>811-908</td> <td>811-908</td> </tr> <tr> <td>Model-based</td> <td>Instruments</td> <td>909-1018</td> <td>909-1018</td> </tr> <tr> <td>Model-based</td> <td>Moods</td> <td>1019-1146</td> <td>1019-1146</td> </tr> <tr> <td>Model-based</td> <td>Various</td> <td>1147-1402</td> <td>1147-1402</td> </tr> <tr> <td>Playlists</td> <td>Genres</td> <td>1403-1973</td> <td>1403-1973</td> </tr> <tr> <td>Playlists</td> <td>Styles</td> <td>1974-1695</td> <td>1974-1695</td> </tr> <tr> <td>Symbolic</td> <td>Pitch</td> <td>1696-1757</td> <td>1696-1757</td> </tr> <tr> <td>Symbolic</td> <td>Melodic</td> <td>1758-1781</td> <td>1758-1781</td> </tr> <tr> <td>Symbolic</td> <td>Chords</td> <td>1782-1836</td> <td>1782-1836</td> </tr> <tr> <td>Symbolic</td> <td>Rhythm</td> <td>1837-1935</td> <td>1837-1935</td> </tr> <tr> <td>Symbolic</td> <td>Tempo</td> <td>1936-1963</td> <td>1936-1963</td> </tr> <tr> <td>Symbolic</td> <td>Instrument presence</td> <td>1964-2441</td> <td>1964-2441</td> </tr> <tr> <td>Symbolic</td> <td>Instruments</td> <td>2442-2456</td> <td>2442-2456</td> </tr> <tr> <td>Symbolic</td> <td>Texture</td> <td>2457-2480</td> <td>2457-2480</td> </tr> <tr> <td>Symbolic</td> <td>Dynamics</td> <td>2481-2484</td> <td>2481-2484</td> </tr> <tr> <td>Album covers</td> <td>SIFT</td> <td>2485-2584</td> <td>2485-2584</td> </tr> <tr> <td>Lyrics</td> <td>jLyrics descriptors</td> <td>2585-2603</td> <td>2585-2671</td> </tr> <tr> <td>Lyrics</td> <td>Bag-of-Words</td> <td>2604-2703</td> <td> </td> </tr> <tr> <td>Lyrics</td> <td>Doc2Vec</td> <td>2704-2803</td> <td> </td> </tr> </tbody> </table>
OpenPack: Public multi-modal dataset for packaging work recognition in logistics domain
<p><strong>OpenPack</strong> is an open-access logistics dataset for human activity recognition, which contains human movement and package information from 16 subjects in four scenarios. Human movement information is subdivided into three types of data, acceleration, physiological, and depth-sensing. The package information includes the size and number of items included in each packaging job. </p> <p>In the "Humanware laboratory" at IST Osaka University, with the supervision of industrial engineers, an experiment to mimic logistic center labor was designed. 12 workers with previous packaging experience and 4 without experience performed a set of packaging tasks according to an instruction manual from a real-life logistics center. During the different scenarios, subjects were recorded while performing packing operations using Lidar, Kinect, and Realsense depth sensors while wearing 4 ATR IMU devices and 2 Empatica E4 wearable sensors. Besides sensor data, this dataset contains timestamp information collected from the hand terminal used to register product, packet, and address label codes as well as package details that can be useful to relate operations to specific packages.</p> <p>The 4 different scenarios include; sequential packing, worker-decided sequence changes, pre-ordered item packing, and time-sensitive stressors. Each of the subjects performed 20 packing jobs in 5 work sessions for a total of 100 packing jobs. <strong>53+</strong> hours of packaging operations have been labeled into 10 global operation classes and 16 sub-action classes for this dataset. Action classes are not unique to each operation but may only appear in one or two operations. </p> <p>You can find information on how to use this dataset at: <a href="https://open-pack.github.io/">https://open-pack.github.io/</a>. For details on how this dataset was collected please check the following publication "OpenPack: A Large-Scale Dataset for Recognizing Packaging Works in IoT-Enabled Logistic Environments" <a href="https://doi.ieeecomputersociety.org/10.1109/PerCom59722.2024.10494448">10.1109/PerCom59722.2024.10494448</a>.</p> <p> </p> <p><strong>Full Dataset</strong></p> <p>In this repository, the data and label files are contained in separate files for each worker. Each worker's file contains; IMU, E4, 2d keypoint, 3d keypoint, annotation, and system-related<em> </em>data<em>.</em></p> <p><em><strong>Preprocessed Dataset (IMU with operation and action Labels)</strong></em></p> <p>We have received many comments that it was difficult to combine multiple workers' IMU and annotation data. Therefore, we have created several CSV files containing the four IMU's sensor data and the operation labels in a single file. These files are now included as "imu-with-operation-action-labels.zip". </p> <p><em><strong>Preprocessed Dataset (Kinect 2D and 3D keypoint data with operation and action Labels)</strong></em></p> <p>We have received several requests for a preprocessed dataset containing only specific types of keypoint data with its assigned operation and action labels. Two new preprocessed files have been added for 2D and 3D keypoint data extracted from the frontal view Kinect camera. These files are:</p> <p>"<a href="11059235" target="_blank" rel="noopener noreferrer">kinect-2d-kpt-with-operation-action-labels.zip</a>", and</p> <p>"<a href="11059235" target="_blank" rel="noopener noreferrer">kinect-3d-kpt-with-operation-action-labels.zip</a>".</p> <p> </p> <p>Work is continuously being done to update and improve this dataset. When downloading and using this dataset please verify that the version is up to date with the latest release. The latest release <strong>[1.1.0]</strong> was uploaded on 24/04/2024. </p> <p><strong><em>Changes LOG:</em></strong></p> <ul> <li>v1.0.0: Add tutorial preprocessed dataset for IMU data with operation labels.</li> <li>v1.1.0: Update preprocessed datasets. (Include Kinect 2d and 3d keypoint data with Operation and action labels)</li> </ul> <p> </p> <p><strong>We hosted an activity recognition competition using this dataset (OpenPack v0.3.x) awarded at a PerCom 2023 Workshop! The task was very simple: Recognize 10 work operations from the OpenPack dataset. You can refer to this website for coding materials relevant to this dataset. </strong><a href="https://open-pack.github.io/challenge2022"><strong>https://open-pack.github.io/challenge2022</strong></a></p>
FAPM: Functional annotation of proteins using multi-modal models beyond structural modeling
<p>Assigning accurate property labels to proteins, like functional terms and catalytic activity, is challenging, especially for proteins without homologs and "tail labels" with few known examples. Unlike previous methods that mainly focused on protein sequence features, we use a pretrained large natural language model to understand the semantic meaning of protein labels. Specifically, we introduce FAPM, a contrastive multi-modal model that links natural language with protein sequence language. This model combines a pretrained protein sequence model with a pretrained large language model to generate labels, such as Gene Ontology (GO) functional terms and catalytic activity predictions, in natural language. Our results show that FAPM excels in understanding protein properties, outperforming models based solely on protein sequences or structures. It achieves state-of-the-art performance on public benchmarks and in-house experimentally annotated phage proteins, which often have few known homologs. Additionally, FAPM's flexibility allows it to incorporate extra text prompts, like taxonomy information, enhancing both its predictive performance and explainability. This novel approach offers a promising alternative to current methods that rely on multiple sequence alignment for protein annotation.</p>
Multi-modal image analysis for large scale cancer tissue studies within IMMUcan: multiplex immunofluorescence images
<p>In cancer research, multiplexed imaging has enabled the in-depth characterization of the tumor microenvironment (TME) and how it relates to patient prognosis. However, standardized, multi-modal data from large numbers of patients to identify robust biomarkers is missing. To provide such data across five cancer indications, the IMMUcan consortium performs broad molecular and cellular spatial profiling of thousands of cancer samples. Two reproducible and scalable workflows have been developed for whole slide multiplexed immunofluorescence (mIF) and imaging mass cytometry (IMC) to overcome challenges of reproducibility and scalability. For mIF we developed IFQuant, a web-based tool optimized for user-friendliness and reproducibility. This Zenodo record contains the mIF images and IFQuant settings to reproduce the results presented in the referenced publication. The companion IMC dataset is available as a joint Zenodo record.</p>
ScienceDex guides
Understand access before you commit
These curated guides explain access requirements, typical timelines, costs, and reuse considerations for widely used research datasets.
Allen Brain Atlas
Allen Brain Atlas is an Allen Institute collection of brain map atlases, datasets, APIs, and analysis tools covering mouse, human, and non-human primate brain resources.
Annotated Behaviour and Observability Dataset (ABODe)
ABODe is a University of Edinburgh DataShare dataset for behavior classification in group-housed mice using home-cage video, identities, bounding boxes, ground-plate positions, and annotator labels.
DANDI Archive for NWB datasets
DANDI is a BRAIN Initiative archive for publishing and sharing neurophysiology data, including electrophysiology, optophysiology, and behavioral data packaged as NWB and related standards.
International Brain Laboratory public data
The International Brain Laboratory public data releases expose standardized mouse decision-making experiments, including Neuropixels recordings, widefield calcium imaging, behavior, and session metadata accessed through the ONE API.
OpenNeuro
OpenNeuro is a free, open platform for sharing neuroimaging datasets, with public search, dataset pages, and download paths for web, S3, DataLad, and the OpenNeuro CLI.