PyKEEN Benchmarking Experiment Model Files

Model Weights <p>This repository provides weights of the models from the benchmarking study conducted in <a href="https://arxiv.org/abs/2006.13365">"Bringing Light Into the Dark: A Large-scale Evaluation of Knowledge Graph Embedding Models Under a Unified Framework"</a> which have been upgraded to compatible with <a href="https://github.com/pykeen/pykeen/releases/tag/v1.9.0">PyKEEN 1.9</a>.</p> <p>The weights are organized as zipfiles, which are named by the dataset-interaction function configuration. For each of these combinations, we chose the best according to validation Hits@10 to include into this repository. For each model, we have three files:</p> <ol> <li><code>configuration.json</code> contains the (pipeline) configuration used to train the model. It can loaded as </li> </ol> <div class="sourceCode"> <pre class="sourceCode python"><code class="sourceCode python"><span><a href="#cb1-1"></a><span class="im">import</span> pathlib</span> <span><a href="#cb1-2"></a><span class="im">import</span> json</span> <span><a href="#cb1-3"></a>configuration <span class="op">=</span> json.loads(pathlib.Path(<span class="st">&quot;configuration.json&quot;</span>).read_text())</span></code></pre> </div> <p>Since the configuration is intended for the <em>pipeline</em>, we need some custom code to re-create the model without re-training it.</p> <div class="sourceCode"> <pre class="sourceCode python"><code class="sourceCode python"><span><a href="#cb2-1"></a><span class="im">from</span> pykeen.datasets <span class="im">import</span> get_dataset</span> <span><a href="#cb2-2"></a><span class="im">from</span> pykeen.models <span class="im">import</span> ERModel, model_resolver</span> <span><a href="#cb2-3"></a></span> <span><a href="#cb2-4"></a>configuration <span class="op">=</span> configuration[<span class="st">&quot;pipeline&quot;</span>]</span> <span><a href="#cb2-5"></a><span class="co"># load the triples factory</span></span> <span><a href="#cb2-6"></a>dataset <span class="op">=</span> get_dataset(</span> <span><a href="#cb2-7"></a> dataset<span class="op">=</span>configuration[<span class="st">&quot;dataset&quot;</span>], dataset_kwargs<span class="op">=</span>configuration.get(<span class="st">&quot;dataset_kwargs&quot;</span>, <span class="va">None</span>)</span> <span><a href="#cb2-8"></a>)</span> <span><a href="#cb2-9"></a>model: ERModel <span class="op">=</span> model_resolver.make(</span> <span><a href="#cb2-10"></a> configuration[<span class="st">&quot;model&quot;</span>], configuration[<span class="st">&quot;model_kwargs&quot;</span>], triples_factory<span class="op">=</span>dataset.training</span> <span><a href="#cb2-11"></a>)</span></code></pre> </div> <p>Note, that this only creates the model instance, but does <em>not</em> load the weights, yet.</p> <ol> <li><code>state_dict.pt</code> contains the weights, stored via <a href="https://pytorch.org/docs/stable/generated/torch.save.html"><code>torch.save</code></a>. They can be loaded via</li> </ol> <div class="sourceCode"> <pre class="sourceCode python"><code class="sourceCode python"><span><a href="#cb3-1"></a><span class="im">import</span> torch</span> <span><a href="#cb3-2"></a>state_dict <span class="op">=</span> torch.load(<span class="st">&quot;state_dict.pt&quot;</span>)</span></code></pre> </div> <p>We can load these weights into the model by using <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html#torch.nn.Module.load_state_dict"><code>Module.load_state_dict</code></a> </p> <div class="sourceCode"> <pre class="sourceCode python"><code class="sourceCode python"><span><a href="#cb4-1"></a>model.load_state_dict(state_dict, strict<span class="op">=</span><span class="va">False</span>)</span></code></pre> </div> <p>Note that we set <code>strict=False</code>, since the exported weights do <em>not</em> contain regularizers' state, while the re-instantiated models may have regularizers.</p> <ol> <li><code>results.json</code> contains the results obtained by the original runs. It can be read by</li> </ol> <div class="sourceCode"> <pre class="sourceCode python"><code class="sourceCode python"><span><a href="#cb5-1"></a><span class="im">import</span> pathlib</span> <span><a href="#cb5-2"></a><span class="im">import</span> json</span> <span><a href="#cb5-3"></a>configuration <span class="op">=</span> json.loads(pathlib.Path(<span class="st">&quot;results.json&quot;</span>).read_text())</span></code></pre> </div> <p>Note that some of the recently added metrics are not available in those results.</p>