Small example Eiger 2X 16M data set from Diamond Light Source I04 revised for HDRMX Gold Standard Discussion

<p>Revised useful small (488 frame) Eiger data set recorded during routine testing, useful for software testing as it is small. Data recorded is from a thaumatin crystal by Graeme Winter,&nbsp; The original dataset is&nbsp; <a href="https://zenodo.org/record/3385862">https://zenodo.org/record/3385862</a>&nbsp; which contains two Eiger MX datasets, Therm_6_1 and Them_6_2, each with a data file and two versions each of the metadata -- a &quot;..._master.h5&quot; file and a &quot;....nxs&quot; file.&nbsp; The former are the usual Eiger metadata files using exposed external links to connect the metadata to the date, and the latter are HDF5-1.10 VDS files.&nbsp; This revision has the same data as the original Therm_6_2 data, but now includes with the &quot;master.h5&quot; file a &quot;..._master_rev.h5&quot; file and with the &quot;.nxs&quot; file a &quot;..._rev.nxs&quot; VDS file.</p> <p>The purpose to the changes in the &quot;..._rev&quot; files is to provide a supporting example for the HDRMX discusssion of a new proposed Eiger &quot;gold standard&quot; to improve the ability to process Eiger MX data collected at one facility at other facilties, by ensuring that sufficient metadata is stored with all datasets.</p> <p>The changes were made by the following script</p> <pre>cp Therm_6_2.nxs Therm_6_2_rev.nxs cp Therm_6_2_master.h5 Therm_6_2_master_rev.h5 export LD_LIBRARY_PATH=$HOME/lib export HDF5_PLUGIN_PATH=$HOME/lib export PATH=$HOME/bin:$PATH h5copy -i Therm_6_2_rev.nxs -o Therm_6_2_master_rev.h5 -s /entry/instrument/name -d /entry/instrument/name -f ref h5copy -i Therm_6_2_rev.nxs -o Therm_6_2_master_rev.h5 -s /entry/instrument/source -d /entry/source -f ref h5copy -i Therm_6_2_rev.nxs -o Therm_6_2_rev.nxs -s /entry/instrument/source -d /entry/source -f ref export end_time=`h5dump -d &quot;/entry/end_time&quot; Therm_6_2_master.h5 | grep &quot;:&quot; | sed &#39;s/^.........//&#39;|sed &#39;s/.\$//&#39;` echo &quot;end_time: $end_time&quot; python &lt;&lt; &#39;EOL&#39; import h5py import numpy as np import os end_time=os.environ[&#39;end_time&#39;] fvds = h5py.File(&#39;Therm_6_2_rev.nxs&#39;,&#39;r+&#39;) fmaster = h5py.File(&#39;Therm_6_2_master_rev.h5&#39;,&#39;r+&#39;) fvds_keys=fvds.keys() fmaster_keys=fmaster.keys() fvds_entry=fvds[&#39;entry&#39;] fmaster_entry=fmaster[&#39;entry&#39;] fvds_entry_keys=fvds_entry.keys() fmaster_entry_keys=fmaster_entry.keys() fvds_entry_instrument=fvds[&#39;entry&#39;][&#39;instrument&#39;] fmaster_entry_instrument=fmaster[&#39;entry&#39;][&#39;instrument&#39;] fvds_entry_instrument_keys=fvds_entry_instrument.keys() fmaster_entry_instrument_keys=fmaster_entry_instrument.keys() fvds_entry_instrument_name=(fvds[&#39;entry&#39;][&#39;instrument&#39;][&#39;name&#39;]) fmaster_entry_instrument_name=(fmaster[&#39;entry&#39;][&#39;instrument&#39;][&#39;name&#39;]) fvds_entry_instrument_short_name=fvds_entry_instrument.attrs[&#39;short_name&#39;] fmaster_entry_instrument_short_name=fmaster_entry_instrument.attrs[&#39;short_name&#39;] zero_offset=fmaster_entry_instrument[&#39;detector&#39;][&#39;module&#39;][&#39;fast_pixel_direction&#39;].attrs[&#39;offset&#39;] fmaster_det_z=fmaster_entry_instrument[&#39;transformations&#39;][&#39;det_z&#39;] fvds_det_z=fvds_entry_instrument[&#39;transformations&#39;][&#39;det_z&#39;] print(&#39;fvds_keys: &#39;,fvds_keys) print(&#39;fmaster_keys: &#39;,fmaster_keys) print(&#39;fvds_entry_keys: &#39;,fvds_entry_keys) print(&#39;fmaster_entry_keys: &#39;,fmaster_entry_keys) print(&#39;fvds_entry_instrument_keys: &#39;,fvds_entry_instrument_keys) print(&#39;fmaster_entry_instrument_keys: &#39;,fmaster_entry_instrument_keys) print(&#39;fvds_entry_instrument_name: &#39;,fvds_entry_instrument_name) print(&#39;fmaster_entry_instrument_name: &#39;,fmaster_entry_instrument_name) print(&#39;fvds_entry_instrument_short_name: &#39;,fvds_entry_instrument_short_name) print(&#39;fmaster_entry_instrument_short_name: &#39;,fmaster_entry_instrument_short_name) print(&#39;fmaster_entry_instrument_detector_module_fast_pixel_direction_offset: &#39;,zero_offset) print(&#39;fmaster_entry_instrument_detector_detector_z_det_z: &#39;,fmaster_det_z) print(&#39;fmaster_entry_end_time: &#39;,end_time) fmaster.attrs.modify(&#39;file_time&#39;,np.string_(end_time)) fmaster.attrs.modify(&#39;file_name&#39;,np.string_(&#39;Therm_6_2_master_rev.h5&#39;)) fmaster.attrs.modify(&#39;HDF5_Version&#39;,np.string_(&#39;hdf5-1.8.18&#39;)) fvds.attrs.modify(&#39;file_time&#39;,np.string_(end_time)) fvds.attrs.modify(&#39;file_name&#39;,np.string_(&#39;Therm_6_2_master_rev.h5&#39;)) fvds.attrs.modify(&#39;HDF5_Version&#39;,np.string_(&#39;hdf5-1.10.5&#39;)) fvds_entry_instrument_name.attrs.modify(&#39;short_name&#39;,np.string_(fvds_entry_instrument.attrs[&#39;short_name&#39;])) fmaster_entry_instrument_name.attrs.modify(&#39;short_name&#39;,np.string_(fmaster_entry_instrument.attrs[&#39;short_name&#39;])) fmaster_entry_instrument[&#39;attenuator&#39;][&#39;attenuator_transmission&#39;].attrs.modify(&#39;units&#39;,np.string_(&quot;&quot;)) fmaster_entry_instrument[&#39;detector&#39;][&#39;count_time&#39;].attrs.modify(&#39;units&#39;,np.string_(&quot;s&quot;)) fvds_entry_instrument_name.attrs.modify(&#39;short_name&#39;,np.string_(fvds_entry_instrument.attrs[&#39;short_name&#39;])) fvds_entry_instrument[&#39;attenuator&#39;][&#39;attenuator_transmission&#39;].attrs.modify(&#39;units&#39;,np.string_(&quot;&quot;)) fvds_entry_instrument[&#39;detector&#39;][&#39;count_time&#39;].attrs.modify(&#39;units&#39;,np.string_(&quot;s&quot;)) fmaster_det_z.attrs.modify(&#39;offset&#39;,zero_offset) fvds_det_z.attrs.modify(&#39;offset&#39;,zero_offset) fmaster_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;phi&#39;].attrs.modify(&#39;offset&#39;,zero_offset) fmaster_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;chi&#39;].attrs.modify(&#39;offset&#39;,zero_offset) fmaster_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;sam_x&#39;].attrs.modify(&#39;offset&#39;,zero_offset) fmaster_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;sam_y&#39;].attrs.modify(&#39;offset&#39;,zero_offset) fmaster_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;sam_z&#39;].attrs.modify(&#39;offset&#39;,zero_offset) fmaster_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;omega&#39;].attrs.modify(&#39;offset&#39;,zero_offset) fvds_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;phi&#39;].attrs.modify(&#39;offset&#39;,zero_offset) fvds_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;chi&#39;].attrs.modify(&#39;offset&#39;,zero_offset) fvds_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;sam_x&#39;].attrs.modify(&#39;offset&#39;,zero_offset) fvds_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;sam_y&#39;].attrs.modify(&#39;offset&#39;,zero_offset) fvds_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;sam_z&#39;].attrs.modify(&#39;offset&#39;,zero_offset) fvds_entry[&#39;sample&#39;][&#39;transformations&#39;][&#39;omega&#39;].attrs.modify(&#39;offset&#39;,zero_offset) print(fmaster[&#39;entry&#39;][&#39;instrument&#39;][&#39;name&#39;].attrs[&#39;short_name&#39;]) print(fmaster[&#39;entry&#39;][&#39;instrument&#39;][&#39;name&#39;].attrs[&#39;short_name&#39;].shape) print(fmaster[&#39;entry&#39;][&#39;instrument&#39;][&#39;name&#39;].attrs[&#39;short_name&#39;].dtype) del fvds_entry_instrument.attrs[&#39;short_name&#39;] del fmaster_entry_instrument.attrs[&#39;short_name&#39;] del fmaster_entry_instrument[&#39;source&#39;] fvds.close() fmaster.close() quit() EOL $HOME/bin/nxvalidate -a NXmx -l /home/yaya/hdrmx_rev_29Sep19/hdrmx/definitions&nbsp; Therm_6_2_master_rev.h5 $HOME/bin/nxvalidate -a NXmx -l /home/yaya/hdrmx_rev_29Sep19/hdrmx/definitions&nbsp; Therm_6_2_rev.nxs </pre> <p>The revised cnxvalidate and definitions are available on github</p> <p><a href="https://github.com/HDRMX/cnxvalidate.git">https://github.com/HDRMX/cnxvalidate.git</a></p> <p><a href="https://github.com/HDRMX/definitions.git">https://github.com/HDRMX/definitions.git</a></p> <p>&nbsp;</p> <p>&nbsp;</p>