ptypy.experiment.hdf5_loader.Hdf5LoaderFast#

class ptypy.experiment.hdf5_loader.Hdf5LoaderFast(pars=None, **kwargs)#

Bases: Hdf5Loader

__init__(pars=None, **kwargs)#: hdf5 data loader

Methods

`__init__`([pars])	hdf5 data loader
`auto`(frames)	Repeated calls to this function will process the data.
`check`([frames, start])	Override in subclass for custom implementation
`compute_scan_mapping_and_trajectory`(...)	This horrendous block of logic is all to do with making a semi-intelligent guess at what the data looks like.
`correct`(raw, weights, common)	Override in subclass for custom implementation
`get_corrected_intensities`(weights, ...)	Corrects the intensities for normalisation and padding
`get_data_chunk`(chunksize[, start])	This function prepares a container that is compatible to data package.
`initialize`()	Begins the Data preparation and intended as the first method that does read-write access on (large) data.
`load`(indices)	Override in subclass for custom implementation
`load_common`()	Override in subclass for custom implementation
`load_mapped_and_arbitrary_scan`(indices)
`load_mapped_and_raster_scan`(indices)
`load_multiprocessing`(src_slices)
`load_positions`()	Override in subclass for custom implementation
`load_unmapped_raster_scan`(indices)
`load_weight`()	Override in subclass for custom implementation
`post_initialize`()	Placeholder.
`report`([what, shout])	Make a report on internal structure.
`subtract_dark`(raw, dark)	Subtract dark current from a raw frame and truncate negative values

Attributes

`CODES`
`DEFAULT`
`EOS`
`METAKEYS`	Keys to store in meta param
`WAIT`
`abort`
`end_of_scan`
`frames_accessible`
`num_frames`	Total number of frames to prepare / load.
`min_frames`	Minimum number of frames to prepare / load with call of `auto()`
`info`	`Param` container that stores all input parameters.

METAKEYS = ['version', 'num_frames', 'label', 'shape', 'psize', 'energy', 'center', 'distance']#: Keys to store in meta param

auto(frames)#

Repeated calls to this function will process the data.

Parameters:

frames (int) – Number of frames to process.

Returns:

one of the following

WAIT, if scan’s end is not reached, but no data could be prepared yet
EOS, if scan’s end is reached
a data package otherwise

Return type:

variable

check(frames=None, start=None)#

Override in subclass for custom implementation

This method checks how many frames the preparation routine may process, starting from frame start at a request of frames.

This method is supposed to return the number of accessible frames for preparation and should determine if data acquisition for this scan is finished. Its main purpose is to allow for a data acquisition scheme, where the number of frames is not known when PtyScan is constructed, i.e. a data stream or an on-the-fly reconstructions.

Note

If num_frames is set on __init__() of the subclass, this method can be left as it is.

Parameters:

frames (int or None) – Number of frames requested.
start (int or None) – Scanpoint index to start checking from.

Returns:

frames_accessible (int) – Number of frames readable.
end_of_scan (int or None) – is one of the following, - 0, end of the scan is not reached - 1, end of scan will be reached or is - None, can’t say

compute_scan_mapping_and_trajectory(data_shape, positions_fast_shape, positions_slow_shape)#: This horrendous block of logic is all to do with making a semi-intelligent guess at what the data looks like.

correct(raw, weights, common)#

Override in subclass for custom implementation

Place holder for dark and flatfield correction. If load already provides data in the form of photon counts, and no frame specific weight is needed, this method may be left as it is.

May get merged with load in future.

Returns:: data, weights – Flat and dark-corrected data dictionaries. These dictionaries must have the same keys as the input raw and contain corrected frames (data) and statistical weights (weights) which are zero for invalid or masked pixel other the number of detector counts that correspond to one photon count
Return type:: dict

get_corrected_intensities(weights, intensities, index, indexed_frame_slice)#: Corrects the intensities for normalisation and padding

get_data_chunk(chunksize, start=None)#

This function prepares a container that is compatible to data package.

This function is called from the auto() function.

info#: Param container that stores all input parameters.

initialize()#

Begins the Data preparation and intended as the first method that does read-write access on (large) data. Does the following:

Creates a *.ptyd data file at location specified by dfile (master node only)
Calls load_weight(), load_positions() load_common() (master node only for load_parallel==None or load_parallel=='data')
Sets num_frames if needed
Calls post_initialize()

load(indices)#

Override in subclass for custom implementation

Loads data according to node specific scanpoint indices that have been determined by LoadManager or otherwise.

Returns:: raw, positions, weight – Dictionaries whose keys are the given scan point indices and whose values are the respective frame / position according to the scan point index. weight and positions may be empty
Return type:: dict

Note

This is the most important method to change when subclassing PtyScan. Most often it suffices to override the constructor and this method to create a subclass suited for a specific experiment.

load_common()#

Override in subclass for custom implementation

Called in initialize()

Loads anything and stores that in a dict. This dict will be available to all processes after initialize() through the attribute common

The purpose of this method is the same as load_weight() and load_positions() except for that the contents of common have no built-in effect of the behavior in the processing other than the user specifies it in py:meth:load

If load_parallel is set to all or common`, this function is executed by all nodes, otherwise the master node executes this function and broadcasts the results to other nodes.

Returns:: common
Return type:: dict

load_positions()#

Override in subclass for custom implementation

Called in initialize()

Loads all positions for all diffraction patterns in this scan. The positions loaded here will be available by all processes through the attribute self.positions. If you specify position on a per frame basis in load() , this function has no effect.

If theoretical positions positions_theory are provided in the initial parameter set DEFAULT, specifying positions here has NO effect and will be ignored.

The purpose of this function is to avoid reloading and parallel reads on files that may require intense parsing to retrieve the information, e.g. long SPEC log files. If parallel reads or log file parsing for each set of frames is not a time critical issue of the subclass, reimplementing this function can be ignored and it is recommended to only reimplement the load() method.

If load_parallel is set to all or common`, this function is executed by all nodes, otherwise the master node executes this function and broadcasts the results to other nodes.

Returns:: positions – A (N,2)-array where N is the number of positions.
Return type:: ndarray

Note

Be aware that this method sets attribute num_frames in the following manner.

If num_frames == None : num_frames = N.
If num_frames < N , no effect.
If num_frames > N : num_frames = N.

load_weight()#

Override in subclass for custom implementation

Called in initialize()

Loads a common (2d)-weight for all diffraction patterns. The weight loaded here will be available by all processes through the attribute self.weight2d. If a per-frame-weight is specified in load() , this function has no effect.

The purpose of this function is to avoid reloading and parallel reads. If that is not critical to the implementation, reimplementing this function in a subclass can be ignored.

If load_parallel is set to all or common`, this function is executed by all nodes, otherwise the master node executes this function and broadcasts the results to other nodes.

Returns:: weight2d – A two-dimensional array with a shape compatible to the raw diffraction data frames
Return type:: ndarray

Note

For now, weights will be converted to a mask, mask = weight2d > 0 for use in reconstruction algorithms. It is planned to use a general weight instead of a mask in future releases.

min_frames#: Minimum number of frames to prepare / load with call of auto()

num_frames#: Total number of frames to prepare / load. Set by num_frames

post_initialize()#

Placeholder. Called at the end of initialize() by all processes.

Use this method to benefit from ‘hard-to-retrieve but now available’ information after initialize.

report(what=None, shout=True)#: Make a report on internal structure.

static subtract_dark(raw, dark)#: Subtract dark current from a raw frame and truncate negative values