Pre-Processed Power Grid Frequency Time Series

<p><strong>Overview</strong><br> This repository contains ready-to-use frequency time series as well as the corresponding pre-processing scripts in python. The data covers three synchronous areas of the European power grid:</p> <ul> <li>Continental Europe</li> <li>Great Britain</li> <li>Nordic</li> </ul> <p>This work is part of the paper &quot;Predictability of Power Grid Frequency&quot;[1]. Please cite this paper, when using the data and the code. For a detailed documentation of the pre-processing procedure we refer to the supplementary material of the paper.</p> <p><strong>Data sources</strong><br> We downloaded the frequency recordings from publically available repositories of three different Transmission System Operators (TSOs).</p> <ul> <li><strong>Continental Europe </strong>[2]: We downloaded the data from the German TSO <em>TransnetBW GmbH,</em> which retains the Copyright on the data, but allows to re-publish it upon request [3].</li> <li><strong>Great Britain </strong>[4]: The download was supported by National Grid ESO Open Data, which belongs to the British TSO <em>National Grid</em>. They publish the frequency recordings under the NGESO Open License [5].</li> <li><strong>Nordic</strong> [6]: We obtained the data from the Finish TSO <em>Fingrid</em>, which provides the data under the open license CC-BY 4.0 [7].</li> </ul> <p><strong>Content of the repository</strong></p> <p><strong>A) Scripts</strong></p> <ol> <li>In the &quot;Download_scripts&quot; folder you will find three scripts to automatically download frequency data from the TSO&#39;s websites.</li> <li>In &quot;convert_data_format.py&quot; we save the data with corrected timestamp formats. Missing data is marked as NaN (processing step (1) in the supplementary material of [1]).</li> <li>In &quot;clean_corrupted_data.py&quot; we load the converted data and identify corrupted recordings. We mark them as NaN and clean some of the resulting data holes (processing step (2) in the supplementary material of [1]).</li> </ol> <p>The python scripts run with Python 3.7 and with the packages found in &quot;requirements.txt&quot;.</p> <p><strong>B) Yearly converted and cleansed data</strong><br> The folders &quot;&lt;year&gt;_converted&quot; contain the output of &quot;convert_data_format.py&quot; and &quot;&lt;year&gt;_cleansed&quot; contain the output of &quot;clean_corrupted_data.py&quot;.</p> <ul> <li><strong>File type</strong>: The files are zipped csv-files, where each file comprises one year.</li> <li><strong>Data format</strong>: The files contain two columns. The second column contains the frequency values in Hz.&nbsp; The first one represents the time stamps in the format <em>Year-Month-Day Hour-Minute-Second</em>, which is given as naive local time. The local time refers to the following time zones and includes Daylight Saving Times (python time zone in brackets): <ul> <li>TransnetBW: Continental European Time (<em>CE)</em></li> <li>Nationalgrid: Great Britain (<em>GB</em>)</li> <li>Fingrid: Finland (<em>Europe/Helsinki</em>)</li> </ul> </li> <li><strong>NaN representation</strong>: We mark corrupted and missing data as &quot;NaN&quot; in the csv-files.</li> </ul> <p><strong>Use cases</strong><br> We point out that this repository can be used in two different was:</p> <ul> <li><strong>Use pre-processed data</strong>: You can directly use the converted or the cleansed data. Note however, that both data sets include segments of NaN-values due to missing and corrupted recordings. Only a very small part of the NaN-values were eliminated in the cleansed data to not manipulate the data too much.</li> </ul> <ul> <li><strong>Produce your own cleansed data</strong>: Depending on your application, you might want to cleanse the data in a custom way. You can easily add your custom cleansing procedure in &quot;clean_corrupted_data.py&quot; and then produce cleansed data from the raw data in &quot;&lt;year&gt;_converted&quot;.</li> </ul> <p><strong>License</strong></p> <p>This work is licensed under multiple licenses, which are located in the &quot;LICENSES&quot; folder.</p> <ul> <li>We release the code in the folder &quot;Scripts&quot; under the MIT license .</li> <li>The pre-processed data in the subfolders &quot;**/Fingrid&quot; and &quot;**/Nationalgrid&quot; are licensed under CC-BY 4.0.</li> <li>TransnetBW originally did not publish their data under an open license. We have explicitly received the permission to publish the pre-processed version from TransnetBW. However, we cannot publish our pre-processed version under an open license due to the missing license of the original TransnetBW data.</li> </ul> <p><strong>Changelog</strong><br> Version 2:</p> <ul> <li>Add time zone information to description</li> <li>Include new frequency data</li> <li>Update references</li> <li>Change folder structure to yearly folders</li> </ul> <p>Version 3:</p> <ul> <li> <p>Correct TransnetBW files for missing data in May 2016</p> </li> </ul>