Generic Catalog Reader (GCR)

class GCR.BaseGenericCatalog(**kwargs)

Abstract base class for all catalog classes.

Attributes

native_filter_quantities

quantities that can be used in native filters

Methods

add_derived_quantity(derived_quantity, func, …)	Add a derived quantify modifier.
add_modifier_on_derived_quantities(…)	Deprecated.
add_quantity_modifier(quantity, modifier[, …])	Add a quantify modifier.
del_quantity_modifier(quantity)	Delete a quantify modifier.
first_available(*quantities)	Return the first available quantity in the input arguments.
get_catalog_info([key, default])	Get information from the catalog config file.
get_input_kwargs([key, default])	Deprecated.
get_normalized_quantity_modifier(quantity)	Retrive a quantify modifier, normalized.
get_quantities(quantities[, filters, …])	Fetch quantities from this catalog.
get_quantity_info(quantity[, key, default])	Get information of a certain quantity.
get_quantity_modifier(quantity)	Retrive a quantify modifier.
has_quantities(quantities[, include_native])	Check if ALL quantities specified are available in this catalog
has_quantity(quantity[, include_native])	Check if quantity is available in this catalog
list_all_native_quantities([with_info])	Return a list of all available native quantities in this catalog.
list_all_quantities([include_native, with_info])	Return a list of all available quantities in this catalog.

_generate_native_quantity_list()

To be implemented by subclass. Must return an iterable of all native quantity names.

_get_quantity_info_dict(quantity, default=None)

To be implemented by subclass. Must return a dictionary for existing quantity. For non-existing quantity must return default.

_iter_native_dataset(native_filters=None)

To be implemented by subclass. Must be a generator. Must yield a callable, native_quantity_getter. This function must iterate over subsets of rows, not columns!

native_filters will either be None or a GCRQuery object (unless self.native_filter_string_only is set to True, in which case native_filters will either be None or a tuple of strings). If native_filters is a GCRQuery object, you can use native_filters.check_scalar(scalar_dict) to check if scalar_dict satisfies native_filters.

Must take a single argument of a native quantity name (unless _obtain_native_data_dict is edited accordingly) Should assume the argument is valid. Must return a numpy 1d array.

static _obtain_native_data_dict(native_quantities_needed, native_quantity_getter)

This method can be overwritten if native_quantity_getter has a different behavior, for example, if native_quantity_getter can retrive multiple quantities at once, this method can return `python dict(zip(native_quantities_needed, native_quantity_getter(native_quantities_needed))) `

_subclass_init(**kwargs)

To be implemented by subclass. Must return None. Must accept any keyword argument (i.e., must have **kwargs). This method is called during __init__().

add_derived_quantity(derived_quantity, func, *quantities)

Add a derived quantify modifier.

Parameters

derived_quantitystr

name of the derived quantity to be added

funccallable

function to calculate the derived quantity the number of arguments should equal number of following quantities

quantitieslist of str

quantities to pass to the callable

add_modifier_on_derived_quantities(new_quantity, func, *quantities)

Deprecated. Use add_derived_quantity instead.

add_quantity_modifier(quantity, modifier, overwrite=False)

Add a quantify modifier. Consider useing the high-level function add_derived_quantity instead!

Parameters

quantitystr

name of the derived quantity to add

modifierNone or str or tuple

If the quantity modifier is a tuple of length >=2 and the first element is a callable, it should be in the formate of (callable, native quantity 1, native quantity 2, …). And the modifier would work as callable(native quantity 1, native quantity 2, …) If the quantity modifier is None, the quantity will be used as the native quantity name Otherwise, the modifier would be use directly as a native quantity name

overwritebool, optional

If False and quantity are already specified in _quantity_modifiers, raise an ValueError

del_quantity_modifier(quantity)

Delete a quantify modifier.

Parameters

quantitystr

name of the derived quantity to delete

first_available(*quantities)

Return the first available quantity in the input arguments. Return None if none of them is available.

get_catalog_info(key=None, default=None)

Get information from the catalog config file. If key is None, return the full dict.

get_input_kwargs(key=None, default=None)

Deprecated. Use get_catalog_info instead.

Get information from the catalog config file. If key is None, return the full dict.

get_normalized_quantity_modifier(quantity)

Retrive a quantify modifier, normalized. This function would also return a tuple, with the first item a callable, and the rest native quantity names

Parameters

quantitystr

name of the derived quantity to get

Returns

tuple(callable, quantity1, quantity2…)

get_quantities(quantities, filters=None, native_filters=None, return_iterator=False)

Fetch quantities from this catalog.

Parameters

quantitiesstr or list of str or tuple of str

quantities to fetch

filterslist of tuple, or GCRQuery instance, optional

filters to apply. Each filter should be in the format of (callable, str, str, …)

native_filterslist of tuple, optional

Native filters to apply. Each filter should be in the format of (callable, str, str, …)

return_iteratorbool, optional

if True, return an iterator that iterates over the native format, default is False

Returns

quantitiesdict, or iterator of dict (when return_iterator is True)

get_quantity_info(quantity, key=None, default=None)

Get information of a certain quantity. If key is None, return the full dict for that quantity.

get_quantity_modifier(quantity)

Retrive a quantify modifier.

Parameters

quantitystr

name of the derived quantity to get

Returns

quantity_modifier

has_quantities(quantities, include_native=True)

Check if ALL quantities specified are available in this catalog

Parameters

quantitiesiterable

a list of quantity names to check

include_nativebool, optional

whether or not to include native quantity names when checking

Returns

has_quantitiesbool

True if the quantities are all available; otherwise False

has_quantity(quantity, include_native=True)

Check if quantity is available in this catalog

Parameters

quantitystr

a quantity name to check

include_nativebool, optional

whether or not to include native quantity names when checking

Returns

has_quantitybool

True if the quantities are all available; otherwise False

list_all_native_quantities(with_info=False)

Return a list of all available native quantities in this catalog.

If with_info is True, return a dict with quantity info.

See also: list_all_quantities

list_all_quantities(include_native=False, with_info=False)

Return a list of all available quantities in this catalog.

If include_native is True, includes native quantities. If with_info is True, return a dict with quantity info.

See also: list_all_native_quantities

property native_filter_quantities

quantities that can be used in native filters

class GCR.CompositeCatalog(catalog_instances, catalog_identifiers=None, matching_methods=None, only_use_master_attr=None, always_return_masked_array=False, **kwargs)

Bases: GCR.base.BaseGenericCatalog

Composite Catalog class

Parameters

catalog_instancesA list of CompositeSpecs instance.

Alternatively, a list of dictionaries that can be used to init CompositeSpecs. See CompositeSpecs docstring for details.

always_return_masked_arraybool, optional (default: False)

If set to True, always return masked array unless the catalogs have matching formats.

Attributes

catalogs

main

master

native_filter_quantities

quantities that can be used in native filters

Methods

add_derived_quantity(derived_quantity, func, …)	Add a derived quantify modifier.
add_modifier_on_derived_quantities(…)	Deprecated.
add_quantity_modifier(quantity, modifier[, …])	Add a quantify modifier.
del_quantity_modifier(quantity)	Delete a quantify modifier.
first_available(*quantities)	Return the first available quantity in the input arguments.
get_catalog_info([key, default])	Get information from the catalog config file.
get_input_kwargs([key, default])	Deprecated.
get_normalized_quantity_modifier(quantity)	Retrive a quantify modifier, normalized.
get_quantities(quantities[, filters, …])	Fetch quantities from this catalog.
get_quantity_info(quantity[, key, default])	Get information of a certain quantity.
get_quantity_modifier(quantity)	Retrive a quantify modifier.
has_quantities(quantities[, include_native])	Check if ALL quantities specified are available in this catalog
has_quantity(quantity[, include_native])	Check if quantity is available in this catalog
list_all_native_quantities([with_info])	Return a list of all available native quantities in this catalog.
list_all_quantities([include_native, with_info])	Return a list of all available quantities in this catalog.

class GCR.GCRQuery(*queries)

Bases: easyquery.Query

Attributes

variable_names

Get all variable names required for this query

Methods

__call__(table[, column_slice])	Use the current Query object to create a mask (a boolean array) for table.
check_scalar(scalar_dict)	check if scalar_dict satisfy query
copy()	Create a copy of the current Query object.
count(table)	Use the current Query object to count the number of entries in table that satisfy queries.
filter(table[, column_slice])	Use the current Query object to create a mask (a boolean array) for table.
mask(table)	Use the current Query object to count the number of entries in table that satisfy queries.

GCR.dict_to_numpy_array(d)

Convert a dict of 1d array to a numpy recarray