Synthetic Multimodal Dataset for Daily Life Activities

<p><strong>Outline</strong></p> <ul> <li>This dataset is originally created for the&nbsp;<a href="https://challenge.knowledge-graph.jp/2022/">Knowledge Graph Reasoning Challenge for Social Issue</a>s (KGRC4SI)</li> <li>Video data that simulates daily life actions in a virtual space from Scenario Data.</li> <li>Knowledge graphs, and transcriptions of the Video Data content (&quot;who&quot; did what &quot;action&quot; with what &quot;object,&quot; when and where, and the resulting &quot;state&quot; or &quot;position&quot; of the object).</li> <li>Knowledge Graph Embedding Data are created for reasoning based on machine learning&nbsp;</li> <li>This data&nbsp;is open to the public as open data</li> </ul> <p><strong>Details</strong></p> <ul> <li> <p><a href="https://github.com/KnowledgeGraphJapan/KGRC-RDF/blob/kgrc4si/Movie">Videos</a></p> <ul> <li>mp4 format</li> <li>203&nbsp;action scenarios</li> <li>For each scenario, there is a character rear view (file name ending in 0), an indoor camera switching view (file name ending in 1), and a fixed camera view placed in each corner of the room (file name ending in 2-5). Also, for each action scenario, data was generated for a minimum of 1 to a maximum of 7 patterns with different room layouts (scenes). A total of 1,218&nbsp;videos</li> <li>Videos with slowly moving characters simulate the movements of elderly people.</li> </ul> </li> <li> <p><a href="https://github.com/KnowledgeGraphJapan/KGRC-RDF/blob/kgrc4si/RDF">Knowledge Graphs</a></p> <ul> <li>RDF format</li> <li>203&nbsp;knowledge graphs corresponding to the videos</li> <li>Includes schema and location supplement information</li> <li>The schema is described below</li> <li><a href="http://kgrc4si.ml:7200/sparql">SPARQL endpoints</a>&nbsp;and&nbsp;<a href="https://github.com/KnowledgeGraphJapan/KGRC-RDF/tree/kgrc4si#%E3%83%8A%E3%83%AC%E3%83%83%E3%82%B8%E3%82%B0%E3%83%A9%E3%83%95%E3%81%AE%E4%BD%BF%E7%94%A8%E6%96%B9%E6%B3%95">query examples</a>&nbsp;are available</li> </ul> </li> <li> <p><a href="https://github.com/KnowledgeGraphJapan/KGRC-RDF/blob/kgrc4si/Program">Script Data</a></p> <ul> <li>txt format</li> <li>Data provided to VirtualHome2KG to generate videos and knowledge graphs</li> <li>Includes the action title and a brief description in text format.</li> </ul> </li> <li>Embedding <ul> <li>Embedding Vectors in TransE, ComplEx, and RotatE. Created with DGL-KE (<a href="https://dglke.dgl.ai/doc/">https://dglke.dgl.ai/doc/</a>)</li> <li>Embedding Vectors created with jRDF2vec (<a href="https://github.com/dwslab/jRDF2Vec">https://github.com/dwslab/jRDF2Vec</a>).</li> </ul> </li> </ul> <p><strong>Specification of Ontology</strong></p> <ul> <li>Please refer to the&nbsp;specification for descriptions of all classes, instances, and properties:&nbsp;<a href="https://aistairc.github.io/VirtualHome2KG/vh2kg_ontology.html">https://aistairc.github.io/VirtualHome2KG/vh2kg_ontology.htm</a></li> </ul> <p><strong>Related Resources</strong></p> <ul> <li><a href="https://www.youtube.com/watch?v=Ajbn8hNXiZ8&amp;list=PLHaRK-B0LUwjvrPgmIBTrf3DsPhmdnFTW">KGRC4SI Final Presentations with automatic English subtitles (YouTube)</a></li> <li><a href="https://github.com/aistairc/VirtualHome2KG">VirtualHome2KG (Software)</a></li> <li><a href="https://github.com/aistairc/virtualhome_unity_aist">VirtualHome-AIST (Unity</a>)</li> <li><a href="https://github.com/aistairc/virtualhome_aist">VirtualHome-AIST (Python API</a>)</li> <li><a href="https://github.com/aistairc/virtualhome2kg_visualization">Visualization Tool</a>&nbsp;(Software)</li> <li><a href="https://github.com/aistairc/virtualhome2kg_generation">Script Editor</a>&nbsp;(Software)</li> </ul>