FlyWire Whole-brain Connectome Connectivity Data

<p>This repository contains the connectivity data for the FlyWire Connectome release. Currently, the latest release is version 783 (see also codex.flywire.ai).</p> <p>The synapses represent a combination of four different data releases, which are combined by the FlyWire whole-brain connectome release. The synapses (as points in space) were detected and published by <a href="https://www.nature.com/articles/s41592-021-01183-7">Buhmann et al., 2021</a> who made use of a cleft segmentation produced by <a href="https://link.springer.com/chapter/10.1007/978-3-030-00934-2_36">Heinrich et al., 2018</a>. The neurotransmitter for these synapses was then predicted and released by <a href="https://www.cell.com/cell/fulltext/S0092-8674(24)00307-6">Eckstein, Bates et al., 2024</a>. The segmentation and neuron IDs (= root IDs) were proofread by the FlyWire consortium and are released by&nbsp;<a href="https://www.nature.com/articles/s41586-024-07558-y">Dorkenwald et al., 2024</a> as part of the <a href="https://www.nature.com/collections/hgcfafejia">FlyWire connectome paper package</a>.&nbsp;</p> <p>Because multiple methods were involved in the production of this resource, the description of the methods is distributed across these manuscripts. We provide a summary in Dorkenwald et al., 2024 (Methods-&gt;Synaptic connections).</p> <p>Some of the files made available use feather as a file format. See a code example for reading these files, including a chunk-wise streaming to handle the large file.</p> <h3>flywire_synapses_783.feather</h3> <p>a pandas dataframe with all ~130 million synapses, their locations, neurotransmitter predictions and pre and postsynaptic partners (=root ids). This table contains all synapses that passed the thresholds (see methods section in <a href="https://www.nature.com/articles/s41586-024-07558-y">Dorkenwald et al., 2024</a>), but not all synapses were associated with proofread neurons (e.g., see Discussion-&gt;Limitations of our reconstruction in <a href="https://www.nature.com/articles/s41586-024-07558-y">Dorkenwald et al., 2024</a>). Hence, not all root IDs in this table will have a match in the proofread root IDs array. This table is provided for completeness, and to allow calculations about total synaptic input and output of neurons independent of other limitations.&nbsp;</p> <p>Columns:</p> <ul> <li>id: synapse ID</li> <li>pre_pt_root_id: presynaptic neuron ID</li> <li>post_pt_root_id: postsynaptic neuron ID</li> <li>connection_score: score assigned by Buhmann et al.; higher is better. We did not use this score to threshold synapses in any analysis</li> <li>cleft_score: score derived from the cleft segmentation by Heinrich et al.; higher is better. We used a threshold of 50 for all analyses and the released dataset. Synapses with lower score are not made available but can be made available on demand.</li> <li>gaba: probability for neurotransmitter=GABA</li> <li>ach: probability for neurotransmitter=Acetylcholine</li> <li>glut: probability for neurotransmitter=Glutamate</li> <li>oct: probability for neurotransmitter=Octopamine</li> <li>ser: probability for neurotransmitter=Serotonin</li> <li>da: probability for neurotransmitter=Dopamine</li> <li>neuropil: the name of the neuropil associated with this synapse. Symmetric neuropils contain a hemisphere annotation after '_'. E.g., ME_L is the medulla in the left hemisphere. For mapping long names, see Ext. Data Fig. 1 or <a href="https://codex.flywire.ai/app/neuropils">https://codex.flywire.ai/app/neuropils</a></li> <li>post_pt_position_{x,y,z}: Coordinate within the postsynaptic neuron (synapses were identified with two points, one in each neuron). Coordinates are in nanometers.</li> <li>pre_pt_position_{x,y,z}: Coordinate within the presynaptic neuron (synapses were identified with two points, one in each neuron). Coordinates are in nanometers.</li> </ul> <p>&nbsp;</p> <h3>per_neuron_neuropil_count_post_783.feather</h3> <p>a pandas dataframe containing the number of postsynapses per neuropil and segment id, i.e. this is a summarized version of&nbsp;<em>flywire_synapses_783.feather</em></p> <p>Columns:</p> <ul> <li>post_pt_root_id: segment ID</li> <li>neuropil: neuropil name. Symmetric neuropils contain a hemisphere annotation after '_'. E.g., ME_L is the medulla in the left hemisphere. For a mapping to long names see Ext. Data Fig. 1 or <a href="https://codex.flywire.ai/app/neuropils">https://codex.flywire.ai/app/neuropils</a></li> <li>Count: number of synapses for this segment ID and neuropil&nbsp;</li> </ul> <h3>&nbsp;</h3> <h3>per_neuron_neuropil_count_pre_783.feather</h3> <p>a pandas dataframe containing the number of presynapses per neuropil and segment id, i.e. this is a summarized version of&nbsp;<em>flywire_synapses_783.feather</em></p> <p>Columns:</p> <ul> <li>pre_pt_root_id: segment ID</li> <li>neuropil: neuropil name. Symmetric neuropils contain a hemisphere annotation after '_'. E.g., ME_L is the medulla in the left hemisphere. For mapping long names, see Ext. Data Fig. 1 or <a href="https://codex.flywire.ai/app/neuropils">https://codex.flywire.ai/app/neuropils</a></li> <li>Count: number of synapses for this segment ID and neuropil&nbsp;</li> </ul> <h3>&nbsp;</h3> <h3>proofread_root_ids_783.npy</h3> <p>an array of all proofread neuron ids (=root ids)</p> <p>&nbsp;</p> <h3>proofread_connections_783.feather</h3> <p>a pandas dataframe containing the proofread subset from&nbsp;<em>flywire_synapses_783.feather&nbsp;</em>and summarized per neuron-neuron pair and neuropil, i.e. this table contains one entry per neuron-neuron pair and neuropil if there is 1 or more synapses for a given combination</p> <p>Columns:</p> <ul> <li>pre_pt_root_id: presynaptic neuron ID</li> <li>post_pt_root_id: postsynaptic neuron ID</li> <li>neuropil: neuropil name. Symmetric neuropils contain a hemisphere annotation after '_'. E.g., ME_L is the medulla in the left hemisphere. For mapping long names, see Ext. Data Fig. 1 or <a href="https://codex.flywire.ai/app/neuropils">https://codex.flywire.ai/app/neuropils</a></li> <li>syn_count: number of synapses between these two neurons in this neuropil</li> <li>gaba_avg: average probability across the synapses for neurotransmitter=GABA</li> <li>ach_avg: average probability across the synapses for neurotransmitter=Acetylcholine</li> <li>glut_avg: average probability across the synapses for neurotransmitter=Glutamate</li> <li>oct_avg: average probability across the synapses for neurotransmitter=Octopamine</li> <li>ser_avg: average probability across the synapses for neurotransmitter=Serotonin</li> <li>da_avg: average probability across the synapses for neurotransmitter=Dopamine</li> </ul> <p>&nbsp;</p> <h3>Code for reading and streaming feather files</h3> <p>Read feather files with pandas:</p> <p><code>import pandas as pd</code></p> <p><code>df = pd.read_feather(path)</code></p> <p>&nbsp;</p> <p>Stream large feather files in chunks:</p> <p><code>import pyarrow.feather as feather</code></p> <p><code>table = feather.read_table(path)</code></p> <p><code># Total number of rows in the Feather file</code><br><code>num_rows = table.num_rows</code></p> <p><code># Define chunk size</code><br><code>chunk_size = 1000</code></p> <p><code># Read and process the data in chunks</code><br><code>for i in range(0, num_rows, chunk_size):</code><br><code>&nbsp; &nbsp; end_row = min(i + chunk_size, num_rows)</code><br><code>&nbsp; &nbsp; chunk = table.slice(i, end_row - i) &nbsp;# Slice the table from i to end_row</code><br><code>&nbsp; &nbsp; </code></p> <p><code>&nbsp; &nbsp; # Convert to pandas DataFrame if needed</code><br><code>&nbsp; &nbsp; df_chunk = chunk.to_pandas()</code><br><code>&nbsp; &nbsp;&nbsp;</code><br><code>&nbsp; &nbsp; # Now you can process each chunk DataFrame as needed</code><br><code>&nbsp; &nbsp; print(df_chunk.head())</code></p>