Data cleaning using unstructured data

<p>In this project, we work on repairing three datasets:</p> <ul> <li>Trials design: This dataset was obtained from the European Union Drug Regulating Authorities Clinical Trials Database (<a href="https://eudract.ema.europa.eu/" target="_blank" rel="noopener">EudraCT</a>) register and the ground truth was created from external registries. In the dataset, multiple countries, identified by the attribute <code>country_protocol_code</code>, conduct the same clinical trials which is identified by <code>eudract_number</code>. Each clinical trial has a <code>title</code> that can help&nbsp; find informative details about the design of the trial.</li> <li>Trials population: This dataset delineates the demographic origins of participants in <a href="https://eudract.ema.europa.eu/" target="_blank" rel="noopener">clinical trials</a> primarily conducted across European countries. This dataset include structured attributes indicating whether the trial pertains to a specific gender, age group or healthy volunteers. Each of these categories is labeled as (`1') or (`0') respectively denoting whether it is included in the trials or not. It is important to note that the population category should remain consistent across all countries conducting the same clinical trial identified by an <code>eudract_number</code>. The ground truth samples in the dataset were established by aligning information about the trial populations provided by external registries, specifically the <a href="https://clinicaltrials.gov/" target="_blank" rel="noopener">CT.gov</a> database and the <a href="https://drks.de/search/en" target="_blank" rel="noopener">German Trials</a> database.&nbsp; Additionally, the dataset comprises other unstructured attributes that categorize the inclusion criteria for trial participants such as <code>inclusion</code>.</li> <li>Allergens: This dataset contains information about products and their allergens. The data was collected from the German version of the `<a href="https://www.alnatura.de/de-de/maerkte/" target="_blank" rel="noopener">Alnatura</a>' (Access date: 24 November, 2020), a free database of food products from around the world `<a href="https://world.openfoodfacts.org/data" target="_blank" rel="noopener">Open Food Facts</a>', and the websites: `<a href="https://migipedia.migros.ch/en">Migipedia</a>', '<a href="https://www.piccantino.com/">Piccantino</a>', and &nbsp;`<a href="http://das-ist-drin.de/">Das Ist Drin</a>'. There may be overlapping products across these websites.&nbsp; Each product in the dataset is identified by a unique <code><em>code</em></code>.&nbsp; Samples with the same <code><em>code</em></code> represent the same product but are extracted from a differentb <code>source</code>. The allergens are indicated by (&lsquo;2&rsquo;) if present, or (&lsquo;1&rsquo;) if there are traces of it, and (&lsquo;0&rsquo;) if it is absent in a product. The dataset also includes information on <code>ingredients</code> in the products. Overall, the dataset comprises categorical structured data describing the presence, trace, or absence of specific allergens, and unstructured text describing ingredients.&nbsp;</li> </ul> <p>N.B: Each '.zip' file contains a set of 5 '.csv' files which are part of the afro-mentioned datasets:</p> <ul> <li>"{dataset_name}_train.csv": samples used for the ML-model training. (e.g "allergens_train.csv")</li> <li>"{dataset_name}_test.csv": samples used to test the the ML-model performance. (e.g "allergens_test.csv")</li> <li>"{dataset_name}_golden_standard.csv": samples represent the ground truth of the test samples. (e.g "allergens_golden_standard.csv")</li> <li>"{dataset_name}_parker_train.csv": samples repaired using <a href="https://gitlab.com/ledc/ledc-sigma/-/blob/master/docs/repair.md#parker-repair" target="_blank" rel="noopener">Parker Engine</a> used for the ML-model training. (e.g "allergens_parker_train.csv")</li> <li>"{dataset_name}_parker_train.csv": samples repaired using Parker Engine used to test the the ML-model performance. (e.g "allergens_parker_test.csv")</li> </ul>