A computational intelligence approach to predict energy demand using Random Forest in a Cloudera cluster

<p>Society&rsquo;s energy consumption has shot up in recent years, making the prediction of&nbsp;its demand a current challenge to ensure an efficient and responsible use. Artificial intelligence&nbsp;techniques have proven to be potential tools in handling tedious tasks and making sense of&nbsp;large-scale data to make better business decisions in different areas of knowledge. In this article,&nbsp;the use of random forests algorithms in a Big Data environment is proposed for households energy&nbsp;demand forecasting. The predictions are based on the use of information from different sources,&nbsp;confirming a fundamental role of socioeconomic data in consumer&rsquo;s behaviours. On the other&nbsp;hand, the use of Big Data architectures is proposed to perform horizontal and vertical scaling of&nbsp;the solution to be used in real environments. Finally, a tool for high-resolution predictions with&nbsp;great efficiency is introduced, which enables energy management in a very accurate way.</p> <p>Raw data is incuded in data.csv. This file contains half hourly home electricity consumption registers for 4404&nbsp;households with fix tariffs (not subject to dynamic time of use) for a period between November 2011 and February 2014. Original information was acquired from the Low Carbon London project led by UK Power Networks (https://data.london.gov.uk/dataset/smartmeter-energy-use-data-in-london-households)</p> <p>RFResults.zip contains the energy predictions for each ACORN group using the generated Random Forest algorithm. For this purpose, the first 613 days of a total of 818 observations of each group were considered for training and the last 205 days for testing.</p> <p>Meteorological data was adquired from the darksky app (https://darksky.net).&nbsp;These data are included in the weather_hourly_darksky.csv</p> <p>uk_bank_holidays. xlsx contains the dated of UK bank holidays for the studied period, used as additional variable related to occupancy</p>