fdi.dataset package

Subpackages

Submodules

fdi.dataset.abstractcomposite module

class fdi.dataset.abstractcomposite.AbstractComposite(**kwds)[source]

Bases: fdi.dataset.attributable.Attributable, fdi.dataset.listener.EventListener, fdi.dataset.composite.Composite, fdi.dataset.datawrapper.DataWrapperMapper

an annotatable and attributable subclass of Composite.

Composite inherits annotatable via EventListener via DataContainer.

__init__(**kwds)[source]
string(level=0, width=0, tablefmt='grid', tablefmt1='simple', tablefmt2='rst', extra=False, matprint=None, trans=True, beforedata='', heavy=True, center=- 1, **kwds)

matprint: an external matrix print function

Parameters
  • trans (print 2D matrix transposed. default is True.)

  • ——-

toString(level=0, width=0, tablefmt='grid', tablefmt1='simple', tablefmt2='rst', extra=False, matprint=None, trans=True, beforedata='', heavy=True, center=- 1, **kwds)[source]

matprint: an external matrix print function

Parameters
  • trans (print 2D matrix transposed. default is True.)

  • ——-

fdi.dataset.annotatable module

class fdi.dataset.annotatable.Annotatable(description='UNKNOWN', **kwds)[source]

Bases: object

An Annotatable object is an object that can give a human readable description of itself.

__init__(description='UNKNOWN', **kwds)[source]
getDescription()[source]

gets the description of this Annotatable object.

Returns

The current description.

Return type

string

setDescription(newDescription)[source]

sets the description of this Annotatable object.

Parameters

newDescription (string) – The new description.

fdi.dataset.arraydataset module

class fdi.dataset.arraydataset.ArrayDataset(data=None, unit=None, description=None, typ_=None, typecode=None, version=None, zInfo=None, alwaysMeta=True, **kwds)[source]

Bases: fdi.dataset.dataset.GenericDataset, collections.abc.Iterable, fdi.dataset.shaped.Shaped

Special dataset that contains a single Array Data object.

mh: If omit the parameter names during instanciation, e.g. ArrayDataset(a, b, c), the assumed order is data, unit, description. mh: contains a sequence which provides methods count(), index(), remove(), reverse(). A mutable sequence would also need append(), extend(), insert(), pop() and sort().

Data

the payload data of this dataset. Default is None, Returns ——-

Initializes an ArrayDataset.

Default `None` will initialize MetaData Parameters to their default values. If data is not None and has shape (len applies), shape MDP is set to the actual data shape.

__init__(data=None, unit=None, description=None, typ_=None, typecode=None, version=None, zInfo=None, alwaysMeta=True, **kwds)[source]

Initializes an ArrayDataset.

Default `None` will initialize MetaData Parameters to their default values. If data is not None and has shape (len applies), shape MDP is set to the actual data shape.

append(*args, **kwargs)[source]

appends to data.

count(*args, **kwargs)[source]

returns size.

extend(*args, **kwargs)[source]

extend data.

index(*args, **kwargs)[source]

returns the index of a value.

pop(*args, **kwargs)[source]

revomes and returns value

remove(*args, **kwargs)[source]

removes value at first occurrence.

setData(data)[source]
string(level=0, tablefmt='grid', tablefmt1='simple', tablefmt2='simple', width=0, param_widths=None, matprint=None, trans=True, center=- 1, heavy=True, **kwds)

matprint: an external matrix print function trans: print 2D matrix transposed. default is True.

toString(level=0, tablefmt='grid', tablefmt1='simple', tablefmt2='simple', width=0, param_widths=None, matprint=None, trans=True, center=- 1, heavy=True, **kwds)[source]

matprint: an external matrix print function trans: print 2D matrix transposed. default is True.

class fdi.dataset.arraydataset.Column(*args, typ_='Column', **kwds)[source]

Bases: fdi.dataset.arraydataset.ArrayDataset, fdi.dataset.listener.ColumnListener

A Column is a the vertical cut of a table for which all cells have the same signature.

A Column contains raw ArrayData, and optionally a description and unit. example:

table = TableDataset()
table.addColumn("Energy",Column(data=[1,2,3,4],description="desc",unit='eV'))

Initializes an ArrayDataset.

Default `None` will initialize MetaData Parameters to their default values. If data is not None and has shape (len applies), shape MDP is set to the actual data shape.

fdi.dataset.arraydataset_datamodel module

fdi.dataset.attributable module

class fdi.dataset.attributable.Attributable(meta=None, zInfo=None, alwaysMeta=False, **kwds)[source]

Bases: fdi.dataset.metadataholder.MetaDataHolder

An Attributable object is an object that has the notion of meta data.

MetaData Porperties (MDPs) are Attributes that store their properties in te metadata table.

Pick out arguments listed in zInfo then put updated parameters into MetaData meta.

meta: meta data container. zInfo: configuration alwaysMeta: always treat parameters as MetaDataProperties (MDPs).

__init__(meta=None, zInfo=None, alwaysMeta=False, **kwds)[source]

Pick out arguments listed in zInfo then put updated parameters into MetaData meta.

meta: meta data container. zInfo: configuration alwaysMeta: always treat parameters as MetaDataProperties (MDPs).

makeMdpMethods(outputfile=None)[source]

Generates a mix-in class file according to MetaData Property list.

property meta
setMdp(name, value, met=None)[source]
setMeta(newMetadata)[source]

Replaces the current MetaData with specified argument.

Product will override this to add listener when meta is replaced. _defaults which usually is self.zInfo is added to new meta so str(meta) can ommit the parameters with default value.

setParameters(params)[source]

Set a group of name-value pairs to the object as properties.

params: a dictionary of name:value where value is a subclass of AbstractParameter. value can be the value of a registered MDP. type will be used if typ_ is given as the name.

fdi.dataset.attributable.MdpInfo = {}

Names not for mormal properties.

class fdi.dataset.attributable.MetaDataProperties[source]

Bases: object

Mix-in place-holder for Class denifitions that has not their own property.py generated by :meth: makeMdpMethods.

fdi.dataset.attributable.Reserved_Property_Names = ['history', 'meta', 'refs', 'dataset', 'zInfo', '_MDP', 'extraMdp', 'alwaysMeta', 'toString', 'string', 'yaml', 'tree']

These MetaData Parameters (MDPs) and vital attrbutes are Set By Parent classes: | Special MDPs and attrbutes | set by parent Classes | attribute holder | | ‘meta’ | Attributable | _meta | | ‘description’ | Annotatable | description | | ‘data’ and ‘shape’ | DataWrapper | _data | | ‘listeners’ | EventSender | _listener |

fdi.dataset.attributable.addMetaDataProperty(cls)[source]

mh: Add MDP to a class so that although they are metadata, they can be accessed by for example, productfoo.creator.

dynamic properties see https://stackoverflow.com/a/2584050 https://stackoverflow.com/a/1355444

fdi.dataset.attributable.make_class_properties(attrs, reserved=None)[source]

Generates class properties source code string according to given attribute names.

fdi.dataset.attributable.value2parameter(name, value, descriptor)[source]

returns a parameter with correct type and attributes according to its value and name.

value: type must be compatible with data_type. For example [0, 0] is wrong; Vector2d([0, 0)] is right if data_type``==``vector2d. descriptor: is zInfo(‘metadata’] or zInfo[‘dataset’][xxx]

fdi.dataset.baseproduct module

class fdi.dataset.baseproduct.BaseProduct(description='UNKNOWN', typ_='BaseProduct', level='ALL', creator='UNKNOWN', creationDate=FineTime(Unknown), rootCause='UNKNOWN', version='0.8', FORMATV='1.6.0.10', zInfo=None, **kwds)[source]

Bases: fdi.dataset.abstractcomposite.AbstractComposite, fdi.dataset.copyable.Copyable, fdi.dataset.listener.EventSender

A BaseProduct is the starting point of te whole product tree, and a generic result that can be passed on between processes.

In general a Product contains zero or more datasets, history, optional reference pointers and metadata some required metadata fields. A Product is expected to fully describe itself; this includes the way how this product was achieved (its history). As it is the result of a process, it should be able to save to and restore from an Archive device.

Many times a Product may contain a single dataset and for this purpose the first dataset entry can be accessed by the getDefault() method. Note that the datasets may be a composite of datasets by themselves.

A built-in attributes in Model[‘metadata’] (“MetaData Parameter” or MDP) can be accessed with e.g. p.creator, or p.meta[‘creator’].value:

p.creator='foo'
assert p.creatur=='foo'
assert p.meta['creator']=='foo'
p.meta['creator']=Parameter('bar')
assert p.meta['creator']==Parameter('bar')

BaseProduct class schema 1.6 inheriting [None].

Automatically generated from fdi/dataset/resources/BaseProduct.yml on 2021-12-13 09:38:07.494042.

Description: FDI base class data model

property FORMATV
accept(visitor)[source]

Hook for adding functionality to meta data object through visitor pattern.

property creationDate
property creator
property description
property history

xx must be a property for self.xx = yy to work in super class after xx is set as a property also by a subclass.

property level
property rootCause
string(level=0, tablefmt='grid', tablefmt1='rst', tablefmt2='rst', matprint=None, trans=True, beforedata='', **kwds)

like AbstractComposite but with history

targetChanged(event)[source]

Informs that an event has happened in a target of any type. Paremeters ———-

toString(level=0, tablefmt='grid', tablefmt1='rst', tablefmt2='rst', matprint=None, trans=True, beforedata='', **kwds)[source]

like AbstractComposite but with history

property type
property version

fdi.dataset.booleanparameter module

fdi.dataset.browseproduct module

class fdi.dataset.browseproduct.BrowseProduct(description='UNKNOWN', typ_='BrowseProduct', level='ALL', creator='UNKNOWN', creationDate=FineTime(Unknown), rootCause='UNKNOWN', version='0.8', FORMATV='1.6.0.1', zInfo=None, **kwds)[source]

Bases: fdi.dataset.baseproduct.BaseProduct

BrowseProduct class schema 1.6 inheriting [‘BaseProduct’].

Automatically generated from fdi/dataset/resources/BrowseProduct.yml on 2021-12-13 09:38:07.543630.

Description: Container of media data for browsing.

Generally a Product (inheriting BaseProduct) has project-wide attributes and can be extended to define a plethora of specialized products.

Initializes instances with more metadata as attributes, set to default values.

Put description keyword argument here to allow e.g. BaseProduct(“foo”) and description=’foo’

__init__(description='UNKNOWN', typ_='BrowseProduct', level='ALL', creator='UNKNOWN', creationDate=FineTime(Unknown), rootCause='UNKNOWN', version='0.8', FORMATV='1.6.0.1', zInfo=None, **kwds)[source]

Initializes instances with more metadata as attributes, set to default values.

Put description keyword argument here to allow e.g. BaseProduct(“foo”) and description=’foo’

fdi.dataset.classes module

class fdi.dataset.classes.Classes[source]

Bases: object

A dictionary of class names and their class objects that are allowed to be deserialized.

An fdi package built-in dictionary (in the format of locals() output) is kept internally. Users who need add more deserializable class can for example:

Define new classes ``class Myclass():

….``

update Classes Classes.classes.update({'myClasses': MyClass})

and use new_instance = Classes.mapping['MyClass']

For a new package with many classes:

Import user classes in a python file for example projectclasses.py:

`` Class PC(Classes):

module_class = {

‘mypackage.mymodule’: [‘MyClass1’, ‘MyClass2’],

} # from another module defining a dict of modulename-Classobj pairs try:

from mypackage.mymodule import pairs

except (ImportError, ModuleNotFoundError) as e:

logger.info(e)

else:

module_class.update(pairs)

_package = {} _classes = {}

``

To use: `` from fdi.dataset.classes import Classes from my.package.projectclasses import PC prjcls = Classes.mapping Classes.updateMapping(PC.updateMapping())

new_instance = prjcls[‘MyClass1’]

class fdi.dataset.classes.Classes_meta(*args, **kwds)[source]

Bases: type

metaclass for ‘classproperty’. # https://stackoverflow.com/a/1800999

Class is initialized with built-in classes by default.

__init__(*args, **kwds)[source]

Class is initialized with built-in classes by default.

importModuleClasses(rerun=False, exclude=None, ignore_error=False, verbose=False)[source]

The set of deserializable classes in module_class is maintained by hand.

Do nothing if the classes mapping is already made so repeated calls will not cost more time.

rerun: set to True to force re-import. If the module-class list has never been imported, it will be imported regardless rerun. exclude: modules whose names (without ‘.’) are in exclude are not imported.

property mapping

Returns the dictionary of classes allowed for deserialization, including the fdi built-ins and user added classes.

Will update the classes if the list is empty

module_class = {'fdi.dataset.arraydataset': ['ArrayDataset', 'Column'], 'fdi.dataset.baseproduct': ['BaseProduct'], 'fdi.dataset.browseproduct': ['BrowseProduct'], 'fdi.dataset.dataset': ['GenericDataset', 'CompositeDataset'], 'fdi.dataset.datatypes': ['Vector', 'Vector2D', 'Quaternion'], 'fdi.dataset.dateparameter': ['DateParameter'], 'fdi.dataset.deserialize': ['deserialize'], 'fdi.dataset.eq': ['DeepEqual'], 'fdi.dataset.finetime': ['FineTime', 'FineTime1', 'utcobj'], 'fdi.dataset.history': ['History'], 'fdi.dataset.listener': ['ListnerSet'], 'fdi.dataset.mediawrapper': ['MediaWrapper'], 'fdi.dataset.metadata': ['AbstractParameter', 'Parameter', 'MetaData'], 'fdi.dataset.numericparameter': ['NumericParameter', 'BooleanParameter'], 'fdi.dataset.odict': ['ODict'], 'fdi.dataset.product': ['Product'], 'fdi.dataset.readonlydict': ['ReadOnlyDict'], 'fdi.dataset.serializable': ['Serializable'], 'fdi.dataset.stringparameter': ['StringParameter'], 'fdi.dataset.tabledataset': ['TableDataset', 'IndexedTableDataset'], 'fdi.dataset.testproducts': ['TP', 'TC', 'TM'], 'fdi.dataset.unstructureddataset': ['UnstrcturedDataset'], 'fdi.pal.context': ['AbstractContext', 'Context', 'MapContext', 'RefContainer', 'ContextRuleException'], 'fdi.pal.productref': ['ProductRef'], 'fdi.pal.query': ['AbstractQuery', 'MetaQuery', 'StorageQuery'], 'fdi.pal.urn': ['Urn']}
reloadClasses()[source]

re-import classes in list.

updateMapping(c=None, rerun=False, exclude=None, verbose=False, ignore_error=False)[source]

Updates classes mapping. Make the package mapping if it has not been made.

fdi.dataset.classes.logger = <Logger fdi.dataset.classes (WARNING)>

this has to be in a different file where other interface classes are defined to avoid circular dependency (such as , Serializable.

Type

Note

fdi.dataset.collectionsMockUp module

class fdi.dataset.collectionsMockUp.ContainerMockUp[source]

Bases: object

class fdi.dataset.collectionsMockUp.MappingMockUp[source]

Bases: object

class fdi.dataset.collectionsMockUp.SequenceMockUp[source]

Bases: object

fdi.dataset.composite module

class fdi.dataset.composite.Composite(data=None, **kwds)[source]

Bases: fdi.dataset.datawrapper.DataContainer, fdi.dataset.serializable.Serializable, collections.abc.MutableMapping

A container of named Datasets.

This container can hold zero or more datasets, each of them stored against a unique name. The order of adding datasets to this Composite is important, that is: the keySet() method will return a set of labels of the datasets in the sequence as they were added. Note that replacing a dataset with the same name, will keep the order.

DeepEqual must stay to the left of MutableMapping so its __eq__ will get to run. Serializable becomes a parent for having __setstate__.

Data

default None will init with a dict.

__init__(data=None, **kwds)[source]
containsKey(name)[source]

Returns true if this map contains a mapping for the specified name.

get(name, default=Invalid)[source]

Returns the dataset to which this composite maps the specified name.

If the attrbute does not exist and default unspecified, raise a KeyError.

Parameters

default: assigne to None or anything for missing name.

getDatasetNames()

Returns a list view of the keys contained in this composite.

getDefault()[source]

Convenience method that returns the first dataset belonging to this composite.

getSets()[source]

Provide access to the Map < String, Dataset > . mh: api from CompositeDataset

isEmpty()[source]

Returns true if this map contains no key - value mappings.

keySet()[source]

Returns a list view of the keys contained in this composite.

remove(name)[source]

Removes the mapping for this name from this composite. mh: returns None if name is None or item does not exist.

set(name, dataset)[source]

Associates the specified dataset with the specified key in this map(optional operation). If the map previously contained a mapping for this key, the old dataset is replaced by the specified dataset. this composite does not permit null or empty keys.

size()[source]

Returns the number of key - value mappings in this map.

string(level=0, matprint=None, trans=True, **kwds)
toString(level=0, matprint=None, trans=True, **kwds)[source]
class fdi.dataset.composite.UserDictAdapter(data=None, *args, **kwds)[source]

Bases: collections.UserDict

Adapter class to make UserDict cooperative to multiple inheritance and take data keyword arg.

REf. https://rhettinger.wordpress.com/2011/05/26/super-considered-super/

Parameters

data: initialize UserDict.

__init__(data=None, *args, **kwds)[source]
Parameters

data: initialize UserDict.

fdi.dataset.copyable module

class fdi.dataset.copyable.Copyable(**kwds)[source]

Bases: object

Interface for objects that can make a copy of themselves.

__init__(**kwds)[source]
copy()[source]

Makes a deep copy of itself.

fdi.dataset.dataset module

class fdi.dataset.dataset.CompositeDataset(**kwds)[source]

Bases: fdi.dataset.listener.MetaDataListener, fdi.dataset.abstractcomposite.AbstractComposite

An CompositeDataset is a Dataset that contains zero or more named Datasets. It allows to build arbitrary complex dataset structures.

It also defines the iteration ordering of its children, which is the order in which the children were inserted into this dataset.

__init__(**kwds)[source]
class fdi.dataset.dataset.Dataset(**kwds)[source]

Bases: fdi.dataset.attributable.Attributable, fdi.dataset.datawrapper.DataContainer, fdi.dataset.serializable.Serializable, fdi.dataset.listener.MetaDataListener

Attributable and annotatable information data container that can be be part of a Product.

developer notes: The intent is that developers do not derive from this interface

directly. Instead, they should inherit from one of the generic datasets that this package provides:

mh: GenericDataset, UnstrcturedDataset ArrayDataset. TableDataset or CompositeDataset.

__init__(**kwds)[source]
accept(visitor)[source]

Hook for adding functionality to object through visitor pattern. Parameter ———

string(level=0, tablefmt='grid', tablefmt1='simple', tablefmt2='rst', param_widths=None, width=0, matprint=None, trans=True, heavy=True, center=- 1, **kwds)

matprint: an external matrix print function trans: print 2D matrix transposed. default is True. Parameter ———

toString(level=0, tablefmt='grid', tablefmt1='simple', tablefmt2='rst', param_widths=None, width=0, matprint=None, trans=True, heavy=True, center=- 1, **kwds)[source]

matprint: an external matrix print function trans: print 2D matrix transposed. default is True. Parameter ———

class fdi.dataset.dataset.GenericDataset(**kwds)[source]

Bases: fdi.dataset.dataset.Dataset, fdi.dataset.typed.Typed, fdi.dataset.datawrapper.DataWrapper

mh: Contains one typed data item with a unit and a typecode.

__init__(**kwds)[source]
fdi.dataset.dataset.make_title_meta_l0(self, level=0, heavy=True, center=0, html=False, **kwds)[source]

make toString title and metadata.

Heavy

use bold symbols for separaters.

Center

0 for no centering; -1 for centering with metadata table; other for str.center(<center>.

fdi.dataset.datatypes module

fdi.dataset.datatypes.DataTypes = {'': 'None', 'baseProduct': 'BaseProduct', 'binary': 'int', 'boolean': 'bool', 'byte': 'int', 'finetime': 'FineTime', 'finetime1': 'FineTime1', 'float': 'float', 'hex': 'int', 'integer': 'int', 'list': 'list', 'mapContext': 'MapContext', 'product': 'Product', 'quaternion': 'Quaternion', 'short': 'int', 'string': 'str', 'tuple': 'tuple', 'vector': 'Vector', 'vector2d': 'Vector2D'}

maps class type names to human-friendly types

class fdi.dataset.datatypes.Quaternion(components=None, **kwds)[source]

Bases: fdi.dataset.datatypes.Vector

Quaternion with 4-component data.

invoked with no argument results in a vector of [0, 0, 0, 0] components

__init__(components=None, **kwds)[source]

invoked with no argument results in a vector of [0, 0, 0, 0] components

class fdi.dataset.datatypes.Vector(components=None, **kwds)[source]

Bases: fdi.dataset.quantifiable.Quantifiable, fdi.dataset.serializable.Serializable, fdi.dataset.eq.DeepcmpEqual

N dimensional vector.

If unit, description, type etc meta data is needed, use a Parameter.

A Vector can compare with a value whose type is in DataTypes, the quantity being used is the magnitude.

invoked with no argument results in a vector of [0, 0, 0] components.

__init__(components=None, **kwds)[source]

invoked with no argument results in a vector of [0, 0, 0] components.

property components

for property getter

getComponents()[source]

Returns the actual components that is allowed for the components of this vector.

setComponents(components)[source]

Replaces the current components of this vector.

string(level=0, **kwds)
toString(level=0, **kwds)[source]
class fdi.dataset.datatypes.Vector2D(components=None, **kwds)[source]

Bases: fdi.dataset.datatypes.Vector

Vector with 2-component data

invoked with no argument results in a vector of [0, 0] components

__init__(components=None, **kwds)[source]

invoked with no argument results in a vector of [0, 0] components

fdi.dataset.datatypes.cast(val, typ_, namespace=None)[source]

casts the input value to type specified, which is in DataTypeNames.

For example ‘binary’ type ‘0x9’ is casted to int 9.

namespace: default is Classes.mapping.

fdi.dataset.datatypes.logger = <Logger fdi.dataset.datatypes (WARNING)>

Allowed data (Parameter and Dataset) types and the corresponding classe names. The keys are mnemonics for humans; the values are type(x).__name__.

fdi.dataset.datatypes.lookup_bd(t)[source]

fdi.dataset.datawrapper module

class fdi.dataset.datawrapper.DataContainer(data=None, **kwds)[source]

Bases: fdi.dataset.annotatable.Annotatable, fdi.dataset.copyable.Copyable, fdi.dataset.eq.DeepcmpEqual, collections.abc.Container

A DataContainer is a composite of data and description.

mh: note that There is no metadata. Implemented partly from AbstractDataWrapper.

data: a Container. Default is None.

__init__(data=None, **kwds)[source]

data: a Container. Default is None.

property data
getData()[source]

Returns the data in this dw

Returns

Return type

self._data or None if ‘._data’ is missing. e.g. subclass overriding setData calling getData from its setData that is called by ‘this.__init__’.

hasData()[source]

Returns whether this data wrapper has data.

Returns

Return type

False if data does not exist or is None or empty, True if otherwise.

setData(data)[source]

Replaces the current DataData with specified argument. mh: subclasses can override this to add listener whenevery data is replaced.

class fdi.dataset.datawrapper.DataWrapper(*args, **kwds)[source]

Bases: fdi.dataset.datawrapper.DataContainer, fdi.dataset.quantifiable.Quantifiable

A DataWrapper is a composite of data, unit and description.

mh: note that all data are in the same unit. There is no metadata. Implemented from AbstractDataWrapper.

__init__(*args, **kwds)[source]
class fdi.dataset.datawrapper.DataWrapperMapper(*args, **kwds)[source]

Bases: object

Object holding a map of data wrappers.

__init__(*args, **kwds)[source]
getDataWrappers()[source]

Gives the data wrappers, mapped by name.

fdi.dataset.dateparameter module

class fdi.dataset.dateparameter.DateParameter(value=None, description='UNKNOWN', default=None, valid=None, **kwds)[source]

Bases: fdi.dataset.metadata.Parameter, fdi.dataset.typecoded.Typecoded

has a FineTime as the value.

Set up a parameter whose value is a point in TAI time.

__init__(value=None, description='UNKNOWN', default=None, valid=None, **kwds)[source]

Set up a parameter whose value is a point in TAI time.

setDefault(default)[source]

accept any type that a FineTime does.

setValue(value)[source]

accept any type that a FineTime does.

class fdi.dataset.dateparameter.DateParameter1(value=None, description='UNKNOWN', default=None, valid=None, **kwds)[source]

Bases: fdi.dataset.dateparameter.DateParameter

Like DateParameter but usese FineTime1.

Set up a parameter whose value is a point in TAI time.

setDefault(default)[source]

accept any type that a FineTime1 does.

setValue(value)[source]

accept any type that a FineTime1 does.

fdi.dataset.deserialize module

class fdi.dataset.deserialize.IntDecoder(*, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, strict=True, object_pairs_hook=None)[source]

Bases: json.decoder.JSONDecoder

adapted from https://stackoverflow.com/questions/45068797/how-to-convert-string-int-json-into-real-int-with-json-loads modified to also convert keys in dictionaries.

object_hook, if specified, will be called with the result of every JSON object decoded and its return value will be used in place of the given dict. This can be used to provide custom deserializations (e.g. to support JSON-RPC class hinting).

object_pairs_hook, if specified will be called with the result of every JSON object decoded with an ordered list of pairs. The return value of object_pairs_hook will be used instead of the dict. This feature can be used to implement custom decoders that rely on the order that the key and value pairs are decoded (for example, collections.OrderedDict will remember the order of insertion). If object_hook is also defined, the object_pairs_hook takes priority.

parse_float, if specified, will be called with the string of every JSON float to be decoded. By default this is equivalent to float(num_str). This can be used to use another datatype or parser for JSON floats (e.g. decimal.Decimal).

parse_int, if specified, will be called with the string of every JSON int to be decoded. By default this is equivalent to int(num_str). This can be used to use another datatype or parser for JSON integers (e.g. float).

parse_constant, if specified, will be called with one of the following strings: -Infinity, Infinity, NaN. This can be used to raise an exception if invalid JSON numbers are encountered.

If strict is false (true is the default), then control characters will be allowed inside strings. Control characters in this context are those with character codes in the 0-31 range, including '\t' (tab), '\n', '\r' and '\0'.

decode(s)[source]
class fdi.dataset.deserialize.IntDecoderOD(*, object_hook=None, parse_float=None, parse_int=None, parse_constant=None, strict=True, object_pairs_hook=None)[source]

Bases: fdi.dataset.deserialize.IntDecoder

object_hook, if specified, will be called with the result of every JSON object decoded and its return value will be used in place of the given dict. This can be used to provide custom deserializations (e.g. to support JSON-RPC class hinting).

object_pairs_hook, if specified will be called with the result of every JSON object decoded with an ordered list of pairs. The return value of object_pairs_hook will be used instead of the dict. This feature can be used to implement custom decoders that rely on the order that the key and value pairs are decoded (for example, collections.OrderedDict will remember the order of insertion). If object_hook is also defined, the object_pairs_hook takes priority.

parse_float, if specified, will be called with the string of every JSON float to be decoded. By default this is equivalent to float(num_str). This can be used to use another datatype or parser for JSON floats (e.g. decimal.Decimal).

parse_int, if specified, will be called with the string of every JSON int to be decoded. By default this is equivalent to int(num_str). This can be used to use another datatype or parser for JSON integers (e.g. float).

parse_constant, if specified, will be called with one of the following strings: -Infinity, Infinity, NaN. This can be used to raise an exception if invalid JSON numbers are encountered.

If strict is false (true is the default), then control characters will be allowed inside strings. Control characters in this context are those with character codes in the 0-31 range, including '\t' (tab), '\n', '\r' and '\0'.

fdi.dataset.deserialize.constructSerializable(obj, lookup=None, debug=False)[source]

mh: reconstruct object from the output of jason.loads(). Recursively goes into nested class instances that are not encoded by default by JSONEncoder, instantiate and fill in variables. Objects to be deserialized must have their classes loaded. _STID cannot have module names in it (e.g. dataset.Product) or locals()[classname] or globals()[classname] will not work.

fdi.dataset.deserialize.decode_str(a0)[source]
fdi.dataset.deserialize.deserialize(js, lookup=None, debug=False, usedict=True)[source]

Loads classes with _STID from the results of serialize.

if usedict is True dict insted of ODict will be used.

fdi.dataset.deserialize.deserialize_args(all_args, not_quoted=False, first_string=True, serialize_out=False)[source]

parse the command path to get positional and keywords arguments.

  1. if not_quoted is True, split everythine to the left of first { with Serialize_Args_Sep append the part startin from the {. mark=’{’

  2. else after splitting all_args with Serialize_Args_Sep: mark=’%7B%22’ (quote(‘{‘))

Scan from left. if all_args[i] not start with mark

Conversion rules: |all_args[i]| converted to | | else | convert (case insensitive) and move on to the next segment | | string not starting with `'0x'` | quote |

  • else decode_str() if `not_quoted==False` else only substitute SAS_Avatar with Serialize_Args_Sep. Then deserialize() this segment to become `{'apiargs':list, 'foo':bar ...}`, append value of `apiargs` to the converted-list above, remove the `apiargs`-`val` pair.

  • return 200 as the reurn code followed by the converted-list and the deserialized `dict`.

All_args

a list of path segments for the args list.

First_string

Do not try to change the type of the first arg (assumed to be the function/method name).

fdi.dataset.deserialize.encode_str(a0)[source]

quote to remove general url offenders then use a mostly harmless str to substitute Serialize_Args_Sep.

fdi.dataset.deserialize.logger = <Logger fdi.dataset.deserialize (WARNING)>

this has to be in a different file where other interface classes are defined to avoid circular dependency (such as , Serializable.

Type

Note

fdi.dataset.deserialize.serialize_args(*args, not_quoted=False, **kwds)[source]

Serialize all positional and keywords arguements as they would appear in a function call. Arguements are assumed to have been placed in the same order of a valid function/method call. They are scanned from left to right from args[i] i = 0, 1,… to kwds[j] j = 0, 1, …

  • Scan args from i=0. if is of args[i] is of bool, int, float types, convert with str, if str(), convert with encode_str(), if bytes or bytearray’ types, with ```0x```+`hex(), save to the convered-list, and move on to the next element.

  • else if finding a segment not of any of the above types,

** put this and the rest of `args` as the `value` in `{'apiargs':value}`, ** and append kwds key-val pairs after this pair, ** serialize the disctionary with serialize() and encode_str() ** append the result to the converted-list. ** break from the args scan loop. * if args scan loop reaches its end, if kwds is not empty, serialize it with serialize() and encode_str(), or scanning reaches the end of args. * append the result to the converted-list. * join the converted-list with Serialize_Args_Sep. * return the result string

fdi.dataset.eq module

exception fdi.dataset.eq.CircularCallError[source]

Bases: RuntimeError

fdi.dataset.eq.DeepEqual

alias of fdi.dataset.eq.DeepcmpEqual

class fdi.dataset.eq.DeepcmpEqual(*args, **kwds)[source]

Bases: object

Equality tested by deepcmp.

Must pass args so DataWrapper in Composite can get data.

__init__(*args, **kwds)[source]

Must pass args so DataWrapper in Composite can get data.

equals(obj, **kwds)[source]

compares using `deepcmp.

class fdi.dataset.eq.EqualDict(**kwds)[source]

Bases: object

mh: Can compare key-val pairs of another object with self. False if compare with None or exceptions raised, e.g. obj does not have items()

equals(obj, verbose=False)[source]
class fdi.dataset.eq.EqualODict(**kwds)[source]

Bases: object

mh: Can compare order and key-val pairs of another object with self. False if compare with None or exceptions raised, e.g. obj does not have items()

equals(obj, verbose=False)[source]
class fdi.dataset.eq.StateEqual(*args, **kwds)[source]

Bases: object

Equality tested by hashed state.

Must pass args so DataWrapper in Composite can get data.

__init__(*args, **kwds)[source]

Must pass args so DataWrapper in Composite can get data.

equals(obj, **kwds)

compares hash.

hash(**kwds)[source]
fdi.dataset.eq.deepcmp(obj1, obj2, seenlist=None, verbose=False, eqcmp=False)[source]

Recursively descends into obj1’s every component and compares with its counterpart in obj2.

Factors includes; * if they are the same object * type * quick string * `__eq__` or `__cmp__` if requested * state from `__getstate__` * quick length * members if is `Mapping`, `Sequence` ( set, list, dict, ordereddict, UserDict … ) * properties/attributes in `__dict__`

Detects cyclic references.

Returns

  • None if finds no difference, a string of explanation

  • otherwise.

  • eqcmp: if True, use __eq__ or __cmp__ if the objs have them. If False only use as the last resort. default True.

fdi.dataset.eq.xhash(hash_list=None, seenlist=None, verbose=None)[source]

get the hash of a tuple of hashes of all members of given sequence.

Hash_list

use instead of self.getstate__()

Verbose

set to trace.

fdi.dataset.finetime module

class fdi.dataset.finetime.FineTime(date=None, format=None, **kwds)[source]

Bases: fdi.dataset.copyable.Copyable, fdi.dataset.eq.DeepcmpEqual, fdi.dataset.serializable.Serializable

Atomic time (SI seconds) elapsed since the TAI epoch of 1 January 1958 UT2. The resolution is one microsecond and the allowable range is: epoch + /-290, 000 years approximately.

This has the following advantages, compared with the standard class:

It has better resolution(microseconds) Time differences are correct across leap seconds It is immutable.

Initiate with a UTC date or an integer TAI.

date; time to be set to. Acceptable types: int for TAI, datetime.datetime, string for ISO format date-time, or bytes-like classed that can get string by calling its decode(encoding=’utf-8’)

DEFAULT_FORMAT = '%Y-%m-%dT%H:%M:%S.%f'
EPOCH = datetime.datetime(1958, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)

number of TAI units in a second

RESOLUTION = 1000000
RETURNFMT = '%s.%06d'
TIMESPEC = 'microseconds'
UTC_LOW_LIMIT = datetime.datetime(1958, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)
__init__(date=None, format=None, **kwds)[source]

Initiate with a UTC date or an integer TAI.

date; time to be set to. Acceptable types: int for TAI, datetime.datetime, string for ISO format date-time, or bytes-like classed that can get string by calling its decode(encoding=’utf-8’)

classmethod datetimeToFineTime(dtm)[source]

Return given Python Datetime in FineTime to the precision of the input. Rounded to the last digit. Unit is decided by RESOLUTION.

getFormat()[source]

format cannot be a property name as it is a built so`@format.setter` is not allowed.

getTime()[source]

Returns the time related to this object.

isoutc(format='%Y-%m-%dT%H:%M:%S.%f')[source]

Returns a String representation of this objet in ISO format without timezone. sub-second set to TIMESPEC.

If tai is None return ‘’. ;format: time format. default ‘%Y-%m-%dT%H:%M:%S’ prints like 2019-02-17T12:43:04.577000

microsecondsSinceEPOCH()[source]

Return the rounded integer number of microseconds since the epoch: 1 Jan 1958.

setTime(time)[source]

Sets the time of this object.

If an integer is given, it will be taken as the TAI. If a datetime object or a string code is given, the timezone will be set to UTC. Only when current TAI is 0, so a non-zero instance is immutable. Violation gets a TypeError.

string(level=0, width=0, **kwds)

Returns a String representation of this object according to self.format. level: 0 prints like 2019-02-17T12:43:04.577000 TAI(…) width: if non-zero, insert newline to break simplified output into shorter lines. For level=0 it is ` #TODO `

subtract(time)[source]

Subract the specified time and return the difference in microseconds.

property time
toDate(tai=None)

Return given FineTime as a Python Datetime.

tai: if not given or given as None, return this object’s time as a Python Datetime.

toDatetime(tai=None)[source]

Return given FineTime as a Python Datetime.

tai: if not given or given as None, return this object’s time as a Python Datetime.

toString(level=0, width=0, **kwds)[source]

Returns a String representation of this object according to self.format. level: 0 prints like 2019-02-17T12:43:04.577000 TAI(…) width: if non-zero, insert newline to break simplified output into shorter lines. For level=0 it is ` #TODO `

class fdi.dataset.finetime.FineTime1(*args, **kwds)[source]

Bases: fdi.dataset.finetime.FineTime

Same as FineTime but Epoch is 2017-1-1 0 UTC and unit is millisecond

Initiate with a UTC date or an integer TAI.

date; time to be set to. Acceptable types: int for TAI, datetime.datetime, string for ISO format date-time, or bytes-like classed that can get string by calling its decode(encoding=’utf-8’)

EPOCH = datetime.datetime(2017, 1, 1, 0, 0, tzinfo=datetime.timezone.utc)

number of TAI units in a second

RESOLUTION = 1000
RETURNFMT = '%s.%03d'
TAI_AT_EPOCH = 1861920037000000
TIMESPEC = 'milliseconds'
classmethod datetimeToFineTime(dtm)[source]

Return given Python Datetime in FineTime to the precision of the input. Rounded to the last digit. Unit is decided by RESOLUTION.

toDatetime(tai=None)[source]

Return given FineTime as a Python Datetime.

tai: if not given or given as None, return this object’s time as a Python Datetime.

fdi.dataset.history module

class fdi.dataset.history.History(other=None, **kwds)[source]

Bases: fdi.dataset.dataset.CompositeDataset

Public interface to the history dataset. Contains the main methods for retrieving a script and copying the history.

mh: The copy constructor is better not be implemented. Use copy() instead. Remember: not only copies the datasets, but also changes the history ID in the metadata and relevant table entries to indicate that this a new independent product of which the history may change.

__init__(other=None, **kwds)[source]

mh: The copy constructor is better not be implemented. Use copy() instead. Remember: not only copies the datasets, but also changes the history ID in the metadata and relevant table entries to indicate that this a new independent product of which the history may change.

accept(visitor)[source]

Hook for adding functionality to meta data object through visitor pattern.

getOutputVar()[source]

Returns the final output variable of the history script.

getScript()[source]

Creates a Jython script from the history.

getTaskHistory()[source]

Returns a human readable formatted history tree.

saveScript(file)[source]

Saves the history script to a file.

fdi.dataset.indexed module

class fdi.dataset.indexed.Indexed(indexPattern=None, **kwds)[source]

Bases: object

Abstract class with an index table for efficient records look-up.

indexPattern: specifies which columns to use to do look up.

__init__(indexPattern=None, **kwds)[source]

indexPattern: specifies which columns to use to do look up.

clearToc()[source]

Clears the index table of content.

getColumnsToLookup()[source]

returns an iterator that gives a number of sequences to looking up over.

Default is a tuple of the data attributes of every columns specified by indexPattern. To be overrided for different data model.

property indexPattern
property toc

returns the index table of content.

updateToc(which=None, for_records=None)[source]

Build index in format specified in indexPattern for retrieving record. which: an iterator that gives a number of sequences to looking up over. Default is getColumnsToLookup(). for_records: a list or a Slice of record (row) numbers. Those are changed records that caused updating. default is all records.

vLookUp(key, return_index=True, multiple=False)[source]

Similar to Excel VLOOKUP, return all records (rows) that match the key. key: taken as a dictionary key unless multiple is True. return_index: if True (default) return index in the array of columns. multiple: if True (default is False) loop through key as a sequence of keys and return a sequece.

fdi.dataset.invalid module

class fdi.dataset.invalid.InvalidType[source]

Bases: object

fdi.dataset.listener module

class fdi.dataset.listener.ColumnListener(**kwds)[source]

Bases: fdi.dataset.listener.EventListener

Listener for events occuring in a Column.

Available types:

* DESCRIPTION_CHANGED
* UNIT_CHANGED
* DATA_CHANGED

Cause is always null.

__init__(**kwds)[source]
class fdi.dataset.listener.DatasetEvent(source, target, typ_, change, cause, rootCause, **kwds)[source]

Bases: fdi.dataset.serializable.Serializable

__init__(source, target, typ_, change, cause, rootCause, **kwds)[source]
string(level=0, **kwds)
toString(level=0, **kwds)[source]
class fdi.dataset.listener.DatasetEventSender(**kwds)[source]

Bases: fdi.dataset.listener.EventSender

__init__(**kwds)[source]
class fdi.dataset.listener.DatasetListener(**kwds)[source]

Bases: fdi.dataset.listener.EventListener

Listener for events occuring in MetaData. Available types:

* DESCRIPTION_CHANGED, METADATA_CHANGED (all datasets)
* DATA_CHANGED, UNIT_CHANGED (ArrayDataset)
* COLUMN_ADDED, COLUMN_REMOVED, COLUMN_CHANGED, ROW_ADDED, VALUE_CHANGED (TableDataset)
* DATASET_ADDED, DATASET_REMOVED, DATASET_CHANGED (CompositeDataset)

Possible causes:

* not null (METADATA_CHANGED, COLUMN_CHANGED, DATASET_CHANGED)
* null (rest)

Warning: The listener handler must be a class attribute in order to create an object hard reference. See DatasetBaseListener.

__init__(**kwds)[source]
class fdi.dataset.listener.EventListener(*args, **kwds)[source]

Bases: fdi.dataset.annotatable.Annotatable

Generic interface for listeners that will listen to anything

targetChanged(*args, **kwargs)[source]

Informs that an event has happened in a target of any type. Paremeters ———-

class fdi.dataset.listener.EventSender(**kwds)[source]

Bases: object

adapted from Peter Thatcher’s https://stackoverflow.com/questions/1092531/event-system-in-python/1096614#1096614

Parameters

Returns

__init__(**kwds)[source]
Parameters

Returns

addListener(listener, cls=<class 'fdi.dataset.listener.EventListener'>)[source]

Adds a listener to this. Paremeters ———-

fire(*args, **kwargs)[source]
getListenerCount()[source]
getListeners()[source]

Returns the current Listeners. Paremeters ———-

property listeners
Parameters

Returns

removeListener(listener)[source]

Removes a listener from this. Paremeters ———-

setListeners(listeners)[source]

Replaces the current Listeners with specified argument. Paremeters ———-

class fdi.dataset.listener.ListnerSet(data=None, *args, **kwds)[source]

Bases: fdi.dataset.serializable.Serializable, fdi.dataset.eq.DeepcmpEqual, collections.UserList

Mutable collection of Listeners of an EvenSender.

Parameters

data: default is None for a list.

__init__(data=None, *args, **kwds)[source]
Parameters

data: default is None for a list.

equals(obj, verbose=False)[source]

compares with another one. :Parameters: Returns

geturns(remove=None)[source]

Returns the current urns. :Parameters: Returns

seturns(urns)[source]

Replaces the current urn with specified argument.

string(level=0, alist=False, **kwds)
Returns

  • LIST[TUPLE(OBJ)] or STRXS

  • A list of member-repre tuples or a string of all depending on alist.

toString(level=0, alist=False, **kwds)[source]
Returns

  • LIST[TUPLE(OBJ)] or STRXS

  • A list of member-repre tuples or a string of all depending on alist.

property urns
class fdi.dataset.listener.MetaDataListener(*args, **kwds)[source]

Bases: fdi.dataset.listener.EventListener

Listener for events occuring in MetaData. Available types:

* PARAMETER_ADDED
* PARAMETER_REMOVED
* PARAMETER_CHANGED

Possible causes: not null (for PARAMETER_CHANGED, if parameter internally changed) null (for PARAMETER_CHANGED, when set is called with a previous existing parameter, and rest)

Warning: The listener handler must be a class attribute in order to create an object hard reference. See DatasetBaseListener.

__init__(*args, **kwds)[source]
class fdi.dataset.listener.ParameterListener(**kwds)[source]

Bases: fdi.dataset.listener.EventListener

Listener for events occuring in a Parameter. Available types:

* DESCRIPTION_CHANGED
* UNIT_CHANGED
* VALUE_CHANGED
* UNKOWN_ATTRIBUTE_CHANGED

Cause is always null.

Warning: The listener handler must be a class attribute in order to create an object hard reference. See DatasetBaseListener.

__init__(**kwds)[source]
class fdi.dataset.listener.ProductListener(**kwds)[source]

Bases: fdi.dataset.listener.EventListener

Listener for events occuring in Product. Available types:

* METADATA_CHANGED
* DATASET_ADDED
* DATASET_REMOVED
* DATASET_CHANGED

Possible causes:

* not null (METADATA_CHANGED, DATASET_CHANGED)
* null (METADATA_CHANGED, DATASET_REMOVED, DATASET_CHANGED)

Warning: The listener handler must be a class attribute in order to create an object hard reference. See DatasetBaseListener.

__init__(**kwds)[source]

fdi.dataset.mediawrapper module

class fdi.dataset.mediawrapper.MediaWrapper(data=None, unit=None, description=None, typ_='image/png', source=None, typecode=None, version=None, zInfo=None, alwaysMeta=True, **kwds)[source]

Bases: fdi.dataset.arraydataset.ArrayDataset

A MediaWrapper contains raw, usually binary, data in specific format.

Initializes media data wrapped in ArrayDataset.

typ_: www style string that follows Content-Type: `. Default is `imagw/png.

__init__(data=None, unit=None, description=None, typ_='image/png', source=None, typecode=None, version=None, zInfo=None, alwaysMeta=True, **kwds)[source]

Initializes media data wrapped in ArrayDataset.

typ_: www style string that follows Content-Type: `. Default is `imagw/png.

fdi.dataset.mediawrapper_datamodel module

fdi.dataset.messagequeue module

class fdi.dataset.messagequeue.MqttRelayListener(topics=None, host=None, port=None, username=None, passwd=None, callback=None, clean_session=None, client_id=None, userdata=None, qos=1, **kwds)[source]

Bases: fdi.dataset.listener.EventListener

Generic interface for sending anything heard to an MQTT message queue.

Starts a MQTT message queue and forward everything in the arguement list to the MQ serialized.

host, port, username, passwd: if any is not provided, it is looked up in `config[‘mqtt’].

__init__(topics=None, host=None, port=None, username=None, passwd=None, callback=None, clean_session=None, client_id=None, userdata=None, qos=1, **kwds)[source]

Starts a MQTT message queue and forward everything in the arguement list to the MQ serialized.

host, port, username, passwd: if any is not provided, it is looked up in `config[‘mqtt’].

targetChanged(*args, **kwargs)[source]

Informs that an event has happened in a target of any type.

class fdi.dataset.messagequeue.MqttRelaySender(topics=None, host=None, port=None, username=None, passwd=None, callback=None, clean_session=None, client_id=None, userdata=None, qos=1, **kwds)[source]

Bases: fdi.dataset.listener.EventSender

Gets MQTT messages and forwards to listeners.

Starts a MQTT message queue and forward everything in the arguement list to the MQ serialized.

host, port, username, passwd: if any is not provided, it is looked up in `config[‘mqtt’].

__init__(topics=None, host=None, port=None, username=None, passwd=None, callback=None, clean_session=None, client_id=None, userdata=None, qos=1, **kwds)[source]

Starts a MQTT message queue and forward everything in the arguement list to the MQ serialized.

host, port, username, passwd: if any is not provided, it is looked up in `config[‘mqtt’].

fdi.dataset.messagequeue.on_message(client, userdata, msg)[source]

fdi.dataset.metadata module

class fdi.dataset.metadata.AbstractParameter(value=None, description='UNKNOWN', **kwds)[source]

Bases: fdi.dataset.annotatable.Annotatable, fdi.dataset.copyable.Copyable, fdi.dataset.eq.DeepcmpEqual, fdi.dataset.listener.DatasetEventSender, fdi.dataset.serializable.Serializable

Parameter is the interface for all named attributes in the MetaData container.

A Parameter is a variable with associated information about its description, unit, type, valid ranges, default, format code etc. Type can be numeric, string, datetime, vector.

Often a parameter shows a property. So a parameter in the metadata of a dataset or product is often called a property.

Default value=None, description=’UNKNOWN’

Constructed with no argument results in a parameter of None value and ‘UNKNOWN’ description ‘’. With a signle argument: arg -> value, ‘UNKNOWN’ as default-> description. With two positional arguments: arg1-> value, arg2-> description. Type is set according to value’s. Unsuported parameter types will get a NotImplementedError.

__init__(value=None, description='UNKNOWN', **kwds)[source]

Constructed with no argument results in a parameter of None value and ‘UNKNOWN’ description ‘’. With a signle argument: arg -> value, ‘UNKNOWN’ as default-> description. With two positional arguments: arg1-> value, arg2-> description. Type is set according to value’s. Unsuported parameter types will get a NotImplementedError.

accept(visitor)[source]

Adds functionality to classes of this type.

getValue()[source]

Gets the value of this parameter as an Object.

getValueAsString()[source]

Value as string for building the string representation of the parameter.

hash()[source]

hash and equality derived only from the value of the parameter.

because Python does not allow overriding __eq__ without setting hash to None.

setValue(value)[source]

Replaces the current value of this parameter.

string(level=0, alist=False, **kwds)

alist: returns a dictionary string representation of parameter attributes.

toString(level=0, alist=False, **kwds)[source]

alist: returns a dictionary string representation of parameter attributes.

property value

for property getter

fdi.dataset.metadata.Default_Extra_Param_Width = 10
Attribute | Defining Module | Holder Variable |
‘description’ | Annotatable | description |
typ_’ | Typed | _type |
‘unit’ | Quantifiable | ‘_unit’ |
‘typecode’ | Typecoded | ‘_typecode’ |
class fdi.dataset.metadata.MetaData(copy=None, defaults=None, **kwds)[source]

Bases: fdi.dataset.listener.ParameterListener, fdi.dataset.composite.Composite, fdi.dataset.copyable.Copyable, fdi.dataset.listener.DatasetEventSender

A container of named Parameters. A MetaData object can have one or more parameters, each of them stored against a unique name. The order of adding parameters to this container is important, that is: the keySet() method will return a set of labels of the parameters in the sequence as they were added. Note that replacing a parameter with the same name, will keep the order.

Table_Widths = [{'name': 15, 'value': 18, 'unit': 6, 'type': 8, 'valid': 17, 'default': 15, 'code': 4, 'description': 17}]
__init__(copy=None, defaults=None, **kwds)[source]
accept(visitor)[source]

Hook for adding functionality to meta data object through visitor pattern.

clear()[source]

Removes all the key - parameter mappings.

remove(name)[source]

add eventhandling

set(name, newParameter)[source]

Saves the parameter and adds eventhandling.

In a parameter name, dot or other invalid characters (when the name is used as a property name) is ignored.

Raises TypeError if not given Parameter (sub) class object.

string(level=0, tablefmt='grid', tablefmt1='simple', tablefmt2='rst', extra=False, param_widths=None, width=0, **kwds)

return string representation of metada.

level: 0 is the most detailed, 2 is the least, tablefmt: format string in packae tabulate, for level==0, tablefmt1 for level1, tablefmt2: format of 2D table data. param_widths: controls how the attributes of every parameter are displayed in the table cells. If is set to -1, there is no cell-width limit. For finer control set a dictionary of parameter attitute names and how many characters wide its tsble cell is, 0 for ommiting the attributable. Default is MetaData.Table_Widths[0]. e.g.

``{‘name’: 8, ‘value’: 17, ‘unit’: 7, ‘type’: 8,

‘valid’: 20, ‘default’: 17, ‘code’: 4, ‘description’: 15}``

toString(level=0, tablefmt='grid', tablefmt1='simple', tablefmt2='rst', extra=False, param_widths=None, width=0, **kwds)[source]

return string representation of metada.

level: 0 is the most detailed, 2 is the least, tablefmt: format string in packae tabulate, for level==0, tablefmt1 for level1, tablefmt2: format of 2D table data. param_widths: controls how the attributes of every parameter are displayed in the table cells. If is set to -1, there is no cell-width limit. For finer control set a dictionary of parameter attitute names and how many characters wide its tsble cell is, 0 for ommiting the attributable. Default is MetaData.Table_Widths[0]. e.g.

``{‘name’: 8, ‘value’: 17, ‘unit’: 7, ‘type’: 8,

‘valid’: 20, ‘default’: 17, ‘code’: 4, ‘description’: 15}``

class fdi.dataset.metadata.Parameter(value=None, description='UNKNOWN', typ_='', default=None, valid=None, **kwds)[source]

Bases: fdi.dataset.metadata.AbstractParameter, fdi.dataset.typed.Typed

Parameter is the interface for all named attributes in the MetaData container. It can have a value and a description. Default arguments: typ_=’’, default=None, valid=None. value=default, description=’UNKNOWN’

invoked with no argument results in a parameter of

None value and ‘UNKNOWN’ description ‘’. typ_ DataTypes[‘’], which is None. With a signle argument: arg -> value, ‘UNKNOWN’-> description. ParameterTypes-> typ_, hex values have integer typ_. Unsuported parameter types will get a NotImplementedError.

f With two positional arguments: arg1-> value, arg2-> description. ParameterTypes[‘’]-> typ_.

Unsuported parameter types will get a NotImplementedError. With three positional arguments: arg1 casted to DataTypes[arg3]-> value, arg2-> description. arg3-> typ_. Unsuported parameter types will get a NotImplementedError. Incompatible value and typ_ will get a TypeError. Parameters ———-

__init__(value=None, description='UNKNOWN', typ_='', default=None, valid=None, **kwds)[source]
invoked with no argument results in a parameter of

None value and ‘UNKNOWN’ description ‘’. typ_ DataTypes[‘’], which is None. With a signle argument: arg -> value, ‘UNKNOWN’-> description. ParameterTypes-> typ_, hex values have integer typ_. Unsuported parameter types will get a NotImplementedError.

f With two positional arguments: arg1-> value, arg2-> description. ParameterTypes[‘’]-> typ_.

Unsuported parameter types will get a NotImplementedError. With three positional arguments: arg1 casted to DataTypes[arg3]-> value, arg2-> description. arg3-> typ_. Unsuported parameter types will get a NotImplementedError. Incompatible value and typ_ will get a TypeError. Parameters ———-

accept(visitor)[source]

Adds functionality to classes of this type.

checked(value)[source]

Checks input value against self.type.

If value is none, returns it; else if type is not set, return value after setting type; If value’s type is a subclass of self’s type, return the value; If value’s and self’s types are both subclass of Number, returns value casted in self’s type.

property default
getDefault()[source]

Returns the default related to this object.

getValid()[source]

Returns the valid related to this object.

isValid()[source]
setDefault(default)[source]

Sets the default of this object.

Default is set directly if type is not set or default is None. If the type of default is not getType(), TypeError is raised.

setType(typ_)[source]

Replaces the current type of this parameter.

Default will be casted if not the same. Unsuported parameter types will get a NotImplementedError.

setValid(valid)[source]

Sets the valid of this object.

If valid is None or empty, set as None, else save in a way so the tuple keys can be serialized with JSON. [[[rangelow, ranehi], state1], [[range2low, r..]..]..]

setValue(value)[source]

Replaces the current value of this parameter.

If value is None set it to None (#TODO: default?) If given/current type is ‘’ and arg value’s type is in DataTypes both value and type are updated to the suitable one in DataTypeNames; or else TypeError is raised. If value type is not a subclass of given/current type, or

Incompatible value and type will get a TypeError.

split(into=None)[source]

split a multiple binary bit-masked parameters according to masks.

into: dictionary mapping bit-masks to the sub-name of the parameter. return: a dictionary mapping name of new parameters to its value.

string(level=0, alist=False, **kwds)

alist: returns a dictionary string representation of parameter attributes.

toString(level=0, alist=False, **kwds)[source]

alist: returns a dictionary string representation of parameter attributes.

property valid
validate(value=Invalid)[source]

checks if a match the rule set.

value: will be checked against the ruleset. Default is self._valid. returns: (valid value, rule name) for discrete and range rules. {mask: (valid val, rule name, mask_height, mask_width), …} for binary masks rules. (INVALID, ‘Invalid’) if no matching is found. (value, ‘Default’) if rule set is empty.

fdi.dataset.metadata.make_jsonable(valid)[source]
fdi.dataset.metadata.parameterDataClasses(tt)[source]

maps machine type names to class objects

fdi.dataset.metadataholder module

class fdi.dataset.metadataholder.MetaDataHolder(meta=None, **kwds)[source]

Bases: object

Object holding meta data.

Adds MetaData to the class. with defaults set to self.zInfo[‘metadata’].

__init__(meta=None, **kwds)[source]

Adds MetaData to the class. with defaults set to self.zInfo[‘metadata’].

getMeta()[source]

Returns the current MetaData container of this object. Cannot become a python property because setMeta is in Attributable

hasMeta()[source]

whether the metadata holder is present. During initialization subclass of MetaDataHolder may need to know if the metadata holder has been put in place with is method.

setMeta(meta)[source]

fdi.dataset.ndprint module

fdi.dataset.ndprint.ndprint(data, trans=True, maxElem=9223372036854775807, tablefmt3='plain', **kwds)[source]
makes a formated string of an N-dimensional array for printing.

The fastest changing index is the innerest list. E.g. A 2 by 3 matrix is [[1,2],[3,4],[5,6]] written as:

1 2 3 4 5 6

But if the matrix is a table, the cells in a column change the fastest,

and the columns are written vertically. So to print a matrix as a table, whose columns are the innerest list, set trans = True (default) then the matrix needs to be printed transposed:

1 3 5 2 4 6

tablefmt3

control 2d array printing. Default ‘plain’.

fdi.dataset.ndprint.padstr(s, w, just='left', pad=' ')[source]

fdi.dataset.numericparameter module

class fdi.dataset.numericparameter.BooleanParameter(value=None, description='UNKNOWN', default=None, valid=None, **kwds)[source]

Bases: fdi.dataset.metadata.Parameter

A Parameter that has a boolean as the value.

Set up a parameter whose value is a boolean

__init__(value=None, description='UNKNOWN', default=None, valid=None, **kwds)[source]

Set up a parameter whose value is a boolean

setDefault(default)[source]

accept any type that bool Vector does.

setValue(value)[source]

accept any type that bool does.

class fdi.dataset.numericparameter.NumericParameter(value=None, description='UNKNOWN', typ_='', default=None, unit=None, valid=None, typecode=None, **kwds)[source]

Bases: fdi.dataset.metadata.Parameter, fdi.dataset.quantifiable.Quantifiable

A Parameter that has a number as the value, a unit, and a typecode.

Set up a parameter whose value is a or a list of numbers.

typ_: type of the parameter value.

__init__(value=None, description='UNKNOWN', typ_='', default=None, unit=None, valid=None, typecode=None, **kwds)[source]

Set up a parameter whose value is a or a list of numbers.

typ_: type of the parameter value.

setDefault(default)[source]

accept any type that a Vector does.

setValue(value)[source]

accept any type that a Vector does.

fdi.dataset.odict module

class fdi.dataset.odict.ODict(*args, **kwds)[source]

Bases: collections.UserDict, fdi.dataset.serializable.Serializable, fdi.dataset.eq.DeepcmpEqual

Ordered dict that is not a subclass of dict and with a better __str__.

__init__(*args, **kwds)[source]
get(name)[source]

Raise a KeyError to change the default behavior of colections.Mapping to quietly return a None when a key is not found in the dict.

hash()[source]
string(level=0, keyval=None, **kwds)
level

default=0,

keyval

default=`None`. If set to a string, ODict class name is not shown and a label of given string is shown with each key as ‘label key:

‘.
tablefmt

=’rst’, tablefmt1=’simple’, tablefmt2=’simple’, matprint=None, trans=True, heavy=True

toString(level=0, keyval=None, **kwds)[source]
level

default=0,

keyval

default=`None`. If set to a string, ODict class name is not shown and a label of given string is shown with each key as ‘label key:

‘.
tablefmt

=’rst’, tablefmt1=’simple’, tablefmt2=’simple’, matprint=None, trans=True, heavy=True

fdi.dataset.product module

class fdi.dataset.product.Product(description='UNKNOWN', typ_='Product', level='ALL', creator='UNKNOWN', creationDate=FineTime(Unknown), rootCause='UNKNOWN', version='0.8', FORMATV='1.6.0.10', startDate=FineTime(Unknown), endDate=FineTime(Unknown), instrument='UNKNOWN', modelName='UNKNOWN', mission='_AGS', zInfo=None, **kwds)[source]

Bases: fdi.dataset.baseproduct.BaseProduct

Product class schema 1.6 inheriting [‘BaseProduct’].

Automatically generated from fdi/dataset/resources/Product.yml on 2021-12-13 09:38:07.593902.

Description: Project level product

Generally a Product (inheriting BaseProduct) has project-wide attributes and can be extended to define a plethora of specialized products.

Initializes instances with more metadata as attributes, set to default values.

Put description keyword argument here to allow e.g. BaseProduct(“foo”) and description=’foo’

__init__(description='UNKNOWN', typ_='Product', level='ALL', creator='UNKNOWN', creationDate=FineTime(Unknown), rootCause='UNKNOWN', version='0.8', FORMATV='1.6.0.10', startDate=FineTime(Unknown), endDate=FineTime(Unknown), instrument='UNKNOWN', modelName='UNKNOWN', mission='_AGS', zInfo=None, **kwds)[source]

Initializes instances with more metadata as attributes, set to default values.

Put description keyword argument here to allow e.g. BaseProduct(“foo”) and description=’foo’

property endDate
property instrument
property mission
property modelName
property startDate

fdi.dataset.quantifiable module

class fdi.dataset.quantifiable.Quantifiable(unit=None, typecode=None, **kwds)[source]

Bases: fdi.dataset.typecoded.Typecoded

A Quantifiable object is a numeric object that has a unit. $ x.unit = ELECTRON_VOLTS $ print x.unit eV [1.60218E-19 J]

Has a unit and a typecode (as in array.array.typecodes).

__init__(unit=None, typecode=None, **kwds)[source]

Has a unit and a typecode (as in array.array.typecodes).

getUnit()[source]

Returns the unit related to this object.

setUnit(unit)[source]

Sets the unit of this object.

property unit

fdi.dataset.readonlydict module

fdi.dataset.readonlydict.ReadOnlyDict

alias of fdi.dataset.readonlydict.frozendict

class fdi.dataset.readonlydict.bReadOnlyDict[source]

Bases: dict

Dictionary that is read-only and returns all values of dict type to ReadOnlyDict type so they, too, cannot be modified.

Based on https://stackoverflow.com/a/61700719/13472124

clear(*args, **kwargs)
pop(*args, **kwargs)
popitem(*args, **kwargs)
setdefault(*args, **kwargs)
update(*args, **kwargs)
class fdi.dataset.readonlydict.frozendict(*args, **kw)[source]

Bases: dict

Extended version of frozent dict.

This extended version of the code above will also “freeze” dictionaries and lists stored as values of the dictionary. If dictionaries/lists are found among the values, they will be handled recursively.

Oren Tirosh, Ero Carrera, Søren Løvborg

https://code.activestate.com/recipes/414283/

__init__(*args, **kw)[source]
property clear
property pop
property popitem
property setdefault
property update
fdi.dataset.readonlydict.make_readonly(d, excluded=None)[source]

Recursively make values of Mapping type at all levels read-only.

fdi.dataset.serializable module

class fdi.dataset.serializable.Serializable(*args, **kwds)[source]

Bases: object

mh: Can be serialized. Has a _STID instance property to show its class information.

__init__(*args, **kwds)[source]
fits(*args, **kwds)[source]

Get a FITS representation.

serializable()[source]

Can be encoded with serializableEncoder

serialized(indent=None)[source]
tree(*args, **kwds)[source]

Get a directory-tree-like representation.

yaml(*args, **kwds)[source]

Get a YAML representation.

class fdi.dataset.serializable.SerializableEncoder(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]

Bases: json.encoder.JSONEncoder

can encode parameter and product etc such that they can be recovered with deserialize(). Python 3 treats string and unicode as unicode, encoded with utf-8, byte blocks as bytes, encoded with utf-8. Python 2 treats string as str and unicode as unicode, encoded with utf-8, byte blocks as str, encoded with utf-8

Constructor for JSONEncoder, with sensible defaults.

If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, float or None. If skipkeys is True, such items are simply skipped.

If ensure_ascii is true, the output is guaranteed to be str objects with all incoming non-ASCII characters escaped. If ensure_ascii is false, the output can contain non-ASCII characters.

If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an OverflowError). Otherwise, no such check takes place.

If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.

If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.

If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation.

If specified, separators should be an (item_separator, key_separator) tuple. The default is (’, ‘, ‘: ‘) if indent is None and (‘,’, ‘: ‘) otherwise. To get the most compact JSON representation, you should specify (‘,’, ‘:’) to eliminate whitespace.

If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a TypeError.

default(obj)[source]
class fdi.dataset.serializable.SerializableEncoderAll(*, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, sort_keys=False, indent=None, separators=None, default=None)[source]

Bases: json.encoder.JSONEncoder

can encode parameter and product etc such that they can be recovered with deserialize(). Python 3 treats string and unicode as unicode, encoded with utf-8, byte blocks as bytes, encoded with utf-8. Python 2 treats string as str and unicode as unicode, encoded with utf-8, byte blocks as str, encoded with utf-8

Constructor for JSONEncoder, with sensible defaults.

If skipkeys is false, then it is a TypeError to attempt encoding of keys that are not str, int, float or None. If skipkeys is True, such items are simply skipped.

If ensure_ascii is true, the output is guaranteed to be str objects with all incoming non-ASCII characters escaped. If ensure_ascii is false, the output can contain non-ASCII characters.

If check_circular is true, then lists, dicts, and custom encoded objects will be checked for circular references during encoding to prevent an infinite recursion (which would cause an OverflowError). Otherwise, no such check takes place.

If allow_nan is true, then NaN, Infinity, and -Infinity will be encoded as such. This behavior is not JSON specification compliant, but is consistent with most JavaScript based encoders and decoders. Otherwise, it will be a ValueError to encode such floats.

If sort_keys is true, then the output of dictionaries will be sorted by key; this is useful for regression tests to ensure that JSON serializations can be compared on a day-to-day basis.

If indent is a non-negative integer, then JSON array elements and object members will be pretty-printed with that indent level. An indent level of 0 will only insert newlines. None is the most compact representation.

If specified, separators should be an (item_separator, key_separator) tuple. The default is (’, ‘, ‘: ‘) if indent is None and (‘,’, ‘: ‘) otherwise. To get the most compact JSON representation, you should specify (‘,’, ‘:’) to eliminate whitespace.

If specified, default is a function that gets called for objects that can’t otherwise be serialized. It should return a JSON encodable version of the object or raise a TypeError.

base = (<class 'str'>, <class 'int'>, <class 'float'>, <class 'bool'>, <class 'NoneType'>)
default(obj)[source]
iterencode(obj, **kwds)[source]
fdi.dataset.serializable.serialize(o, cls=None, **kwds)[source]

return JSON using special encoder SerializableEncoder

fdi.dataset.shaped module

class fdi.dataset.shaped.Shaped(**kwds)[source]

Bases: object

An object of specifiable kinds.

This class is for parameters, not for products/datasets to make ‘shape’ MDP.

Has a shape.

__init__(**kwds)[source]

Has a shape.

updateShape(**kwds)[source]

fdi.dataset.stringparameter module

class fdi.dataset.stringparameter.StringParameter(value=None, description='UNKNOWN', default=None, valid=None, typecode='B', **kwds)[source]

Bases: fdi.dataset.metadata.Parameter, fdi.dataset.typecoded.Typecoded

has a unicode string as the value, a typecode for length and char.

invoked with no argument results in a parameter of

None value and ‘UNKNOWN’ description ‘’. typ_ DataTypes[‘’], which is None. With a signle argument: arg -> value, ‘UNKNOWN’-> description. ParameterTypes-> typ_, hex values have integer typ_. Unsuported parameter types will get a NotImplementedError.

f With two positional arguments: arg1-> value, arg2-> description. ParameterTypes[‘’]-> typ_.

Unsuported parameter types will get a NotImplementedError. With three positional arguments: arg1 casted to DataTypes[arg3]-> value, arg2-> description. arg3-> typ_. Unsuported parameter types will get a NotImplementedError. Incompatible value and typ_ will get a TypeError. Parameters ———-

fdi.dataset.tabledataset module

class fdi.dataset.tabledataset.IndexedTableDataset(**kwds)[source]

Bases: fdi.dataset.indexed.Indexed, fdi.dataset.tabledataset.TableDataset

TableDataset with an index table for efficient row look-up.

__init__(**kwds)[source]
getColumnsToLookup()[source]

returns an iterator that gives a number of sequences to looking up over.

hashx()[source]
setData(data)[source]

sets name-column pairs from data and updates index if needed

vLookUp(key, return_index=True, multiple=False)[source]

Similar to Excel VLOOKUP, return all records (rows) that match the key. key: taken as a dictionary key unless multiple is True. return_index: if True (default) return index in the array of columns. multiple: if True (default is False) loop through key as a sequence of keys and return a sequece.

class fdi.dataset.tabledataset.TableDataset(data=None, description=None, typ_=None, version=None, zInfo=None, alwaysMeta=True, **kwds)[source]

Bases: fdi.dataset.dataset.CompositeDataset, fdi.dataset.tabledataset.TableModel, fdi.dataset.shaped.Shaped

Special dataset that contains a single Array Data object. A TableDataset is a tabular collection of Columns. It is optimized to work on array data.. The column-wise approach is convenient in many cases. For example, one has an event list, and each algorithm is adding a new field to the events (i.e. a new column, for example a quality mask).

Although mechanisms are provided to grow the table row-wise, one should use these with care especially in performance driven environments as this orthogonal approach (adding rows rather than adding columns) is expensive.

General Note:

For reasons of flexibility, memory consumption and performance, this class is not checking whether all columns are of the same length: this is the responsibility of the user/developer. See also the library documentation for more information about this.

Note on column names:

If a column is added without specifying a name, the name ColumnX is created, where X denotes the index of that column. Column name duplicity is not allowed.

Developers:

See “Writing special datasets or products” at the developer’s documentation also.

Please see also this selection example.

__init__(data=None, description=None, typ_=None, version=None, zInfo=None, alwaysMeta=True, **kwds)[source]
addColumn(name, column, col_des=True)[source]

Adds the specified column to this table, and attaches a name to it.

If the name is null, a dummy name “column”+column_count+1 is created, such that it can be accessed by getColumn(str).

If column name exists the corresponding column is substituted.

Parameters: name - column name. column - column to be added. col_des - if True (default) and if column description is ‘UNKNOWN’ or None, set to column name.

addRow(row, rows=False)[source]

Adds the specified map as a new row to this table.

row: mh: row is a dict with names as keys and row data as value. rows: append each element in row if the row data is a list.

property columnCount
getColumn(key)[source]

Returns the particular column, or a list of columns.

key: ref getColumnMap()

  • int/str: return the int-th/named column;

  • Slice object, list of columns of sliced column indices;

  • list of integers/strings: return a list of columns corresponding to the given column index/name, or where key is True. Example a.getColumn(Slice(3,,)), [2, 4], ['time', ``energy'].

  • list of booleans: return a list of columns where key is True. Example [True, False...].

getColumnMap(key=None)[source]

Returns a dict of column-names as the keys and the column(s) as the values.

key: return the following as the value for each key-value pair:

  • int: name-value where value is the int-th column.

  • Slice object, a list of name-columns from slicing the column index. Example a.getColumn(Slice(3,,));

  • Sequence of integers/strings: they are used as the column index/name to select the columns.

  • Sequence of booleans: columns where the corresponding boolean is True are chosen.

Default is to return all columns.

getRow(rowIndex)[source]

Returns a list containing the objects located in a particular row, or a list of rows.

rowIndex: ref getRowMap()

  • int: return the int-th row in a list of elements;

  • Slice object, list of integers, list of booleans: return a list of rows each represented by a tuple. Example a.getRow(Slice(3,,)), [2,4], [True, False...].

getRowMap(rowIndex)[source]

Returns a dict of column-names as the keys and the objects located at a particular row(s) as the values.

rowIndex: return the following as the value for each key-value pair:

  • int: the int-th row’s elements;

  • Slice object, a list of rows from slicing the column. Example a.getRow(Slice(3,,));

  • list of integers: they are used as the row index to select the rows.

  • list of booleans: rows where the corresponding boolean is True are chosen.

indexOf(key)[source]

Returns the index of specified column.

if the key is a Column, it looks for equal references (same column objects), not for equal values. If the key is a string, Returns the index of specified Column name.

items()[source]

for k,v in tabledataset.items()

keys()

Returns the column names.

property list
removeColumn(key)[source]

Removes the columns specified by key.

ref. getColumnMap on key usage.

removeRow(rowIndex)[source]

Removes a row with specified index from this table.

rowIndex: int or a Slice object. Example a.removeRow(Slice(3,,)). return: removed row data.

property rowCount
select(selection)[source]

Select a number of rows from this table dataset and return a new TableDataset object containing only the selected rows.

selection: to form a new Tabledataset with ref getRowMap()

setColumn(key, value)[source]

Replaces a column in this table with specified name to specified column if key is a string and exists, or if the key is an integer in 0 to the number of columns, insert at column-index=key, with the name ‘column’+key, else add a new coolumn.

setColumnCount(columnCount)[source]

cannot do this.

setData(data)[source]

sets name-column pairs from data.

Valid formd include: {str:Column, …} or [(str, [num, …], str)] or [(str, Column), …] or [[num …], [num …], …]

[{‘name’:str,’column’:Column}] form is deprecated.

Existing data will be discarded except when the provided data is a list of lists, where existing column names and units will remain but data replaced, and extra data items will form new columns named ‘column’+index (index counting from 1) with unit None.

setRowCount(rowCount)[source]

cannot do this.

string(level=0, tablefmt='grid', tablefmt1='simple', tablefmt2='rst', width=0, param_widths=None, matprint=None, trans=True, heavy=True, center=- 1, col_width=8, **kwds)

tablefmt2: format of 2D data, others see MetaData.toString.

toString(level=0, tablefmt='grid', tablefmt1='simple', tablefmt2='rst', width=0, param_widths=None, matprint=None, trans=True, heavy=True, center=- 1, col_width=8, **kwds)[source]

tablefmt2: format of 2D data, others see MetaData.toString.

class fdi.dataset.tabledataset.TableModel(**kwds)[source]

Bases: object

to interrogate a tabular data model

__init__(**kwds)[source]
getColumnClass(columnIndex)[source]

Returns the class for the first cell values in the column.

getColumnCount()[source]

Returns the number of columns in the model.

getColumnName(columnIndex)[source]

Returns the name of the column at columnIndex.

returns a set of columns if key is a slice.

getColumnNames()[source]

Returns the column names.

getRowCount()[source]

Returns the number of rows in the model.

getValueAt(rowIndex, columnIndex)[source]

Returns the value for the cell at columnIndex and rowIndex.

isCellEditable(rowIndex, columnIndex)[source]

Returns true if the cell at rowIndex and columnIndex is editable.

setValueAt(value, rowIndex, columnIndex)[source]

Sets the value in the cell at columnIndex and rowIndex to Value.

fdi.dataset.tabledataset.maybe2rows(header_names, units, col_width)[source]

makes one-row or two-row header

fdi.dataset.tabledataset_datamodel module

fdi.dataset.testproducts module

class fdi.dataset.testproducts.SP(description='UNKNOWN', typ_='SP', creator='UNKNOWN', version='9', creationDate=FineTime(Unknown), rootCause='UNKNOWN', startDate=FineTime(Unknown), endDate=FineTime(Unknown), instrument='UNKNOWN', modelName='UNKNOWN', mission='_AGS', zInfo=None, **kwds)[source]

Bases: fdi.dataset.product.Product

A subclass of Product for tests.

Initializes instances with more metadata as attributes, set to default values.

Put description keyword argument here to allow e.g. BaseProduct(“foo”) and description=’foo’

class fdi.dataset.testproducts.TC(*args, **kwds)[source]

Bases: fdi.pal.context.Context

class fdi.dataset.testproducts.TM(*args, **kwds)[source]

Bases: fdi.pal.context.MapContext

class fdi.dataset.testproducts.TP(description='UNKNOWN', typ_='Product', level='ALL', creator='UNKNOWN', creationDate=FineTime(Unknown), rootCause='UNKNOWN', version='0.8', FORMATV='1.6.0.10', startDate=FineTime(Unknown), endDate=FineTime(Unknown), instrument='UNKNOWN', modelName='UNKNOWN', mission='_AGS', zInfo=None, **kwds)[source]

Bases: fdi.dataset.product.Product

Initializes instances with more metadata as attributes, set to default values.

Put description keyword argument here to allow e.g. BaseProduct(“foo”) and description=’foo’

fdi.dataset.testproducts.get_demo_product()[source]

A complex context product as a reference for testing and demo.

```

|__ meta <MetaData> | |__ description <string> | |__ type <string> | |__ level <string> | |__ creator <string> | |__ creationDate <finetime> | |__ rootCause <string> | |__ version <string> | |__ FORMATV <string> | |__ speed <vector> | __ listeners <ListnerSet> |__ measurements <CompositeDataset> | |__ meta <MetaData> | | __ listeners <ListnerSet> | |__ Time_Energy_Pos <TableDataset> (5, 20) | | |__ meta <MetaData> | | | |__ description <string> | | | |__ shape <tuple> | | | |__ type <string> | | | |__ version <string> | | | |__ FORMATV <string> | | | __ listeners <ListnerSet> | | |__ Time <Column> (20,) | | |__ Energy <Column> (20,) | | |__ Error <Column> (20,) | | |__ y <Column> (20,) | | __ z <Column> (20,) | |__ calibration <ArrayDataset> (11, 11) | __ dset <str> |__ Environment Temperature <ArrayDataset> (7,) |__ Browse <image/png> (5976,) |__ refs <RefContainer> | |__ a_reference <ProductRef> | __ a_different_name <ProductRef> |__ history <History> | |__ PARAM_HISTORY <str> | |__ TASK_HISTORY <str> | __ meta <MetaData> | __ listeners <ListnerSet> __ listeners <ListnerSet>

fdi.dataset.testproducts.makeCal2D(width=11, height=11)[source]

fdi.dataset.typecoded module

class fdi.dataset.typecoded.Typecoded(typecode=None, **kwds)[source]

Bases: object

Has internal store type specified as TypeCode. ref doc of drray.Array.

Has a typecode.

Typecode is defined in array.array

__init__(typecode=None, **kwds)[source]

Has a typecode.

Typecode is defined in array.array

getTypecode()[source]

Returns the typecode related to this object.

setTypecode(typecode)[source]

Sets the typecode of this object.

property typecode

fdi.dataset.typed module

class fdi.dataset.typed.Typed(typ_=None, **kwds)[source]

Bases: object

An object of specifiable kinds.

This class is for parameters, not for products/datasets to make ‘type’ MDP.

Has a type.

__init__(typ_=None, **kwds)[source]

Has a type.

getType()[source]

Returns the actual type that is allowed for the value of this Parameter.

setType(typ_)[source]

Replaces the current type of this parameter.

property type

for property getter

fdi.dataset.unstructureddataset module

class fdi.dataset.unstructureddataset.UnstrcturedDataset(data=None, description=None, typ_=None, doctype=None, version=None, zInfo=None, alwaysMeta=True, **kwds)[source]

Bases: fdi.dataset.dataset.Dataset, fdi.dataset.copyable.Copyable

Container for data without pre-defined structure or organization..

MetaDataListener must stay to the left of AbstractComposite.

For xmltodict xml_attribs default to `False`.

__init__(data=None, description=None, typ_=None, doctype=None, version=None, zInfo=None, alwaysMeta=True, **kwds)[source]
fetch(paths, exe=['is'], not_quoted=True)[source]
getData()[source]

Optimized for _data being initialized to be _data by DataWrapper.

input(data, doctype=None, **kwds)[source]

Put data in the dataset.

Depending on doctype: * Default is None for arbitrarily nested Pyton data structure. * Use ‘json’ to have the input string loaded by json.loads, * ‘xml’ by xmltodict.parse.

jsonPath(expr, val='simple', sep='/', indent=None, *args, **kwds)[source]

Make a JSONPath query on the data.

Expr

JSONPath expression. Ref ‘jsonpath_ng’

Sep

‘’ or None for keeping jsonpath_ng format (e.g. a.b.[3].d; other string for substituting ‘.’ to the given string, with ‘[’ and ‘]’ removed. Default is ‘/’.

Val

‘context’ for returning the list of DatumInContext of find; ‘simple’ (default) for list of simple types of values and summarizing list and dict values; other for a list of un-treated `DatumInContext.value`s.

Indent

for json.dumps.

Returns

  • If val is `context`, return the list of DatumInContext of jsonpath_ng.ext.parse().find().

  • Else return a list of full_path-value pairs from the output of find().

  • * If val is `simple`, only node values of simple types are kept, list and dict types will show as ‘<list> length’ and ‘<dict> [keys [… [length]]]’, respectively.

  • * If val is `full`, the values of returned `list`s are un-treated `DatumInContext.value`s.

make_meta(print=False, **kwds)[source]
fdi.dataset.unstructureddataset.getCacheInfo()[source]
fdi.dataset.unstructureddataset.jexp(expr, *args, **kwds)[source]

fdi.dataset.unstructureddataset_datamodel module

fdi.dataset.vattribute module

fdi.dataset.yaml2python module

fdi.dataset.yaml2python.dependency_sort(descriptors)[source]

sort the descriptors so that everyone’s parents are to his right.

fdi.dataset.yaml2python.getCls(clp, rerun=True, exclude=None, ignore_error=True, verbose=False)[source]
fdi.dataset.yaml2python.getPython(val, indents, demo, onlyInclude, debug=False)[source]

make Model and init__() code strings from given data.

fdi.dataset.yaml2python.makeinitcode(dt, pval)[source]

python instanciation source code.

will be like “default: FineTime1(0)”

fdi.dataset.yaml2python.noParentsParents(pn)[source]
fdi.dataset.yaml2python.output(nm, d, fins, version, dry_run=False, verbose=False)[source]
fdi.dataset.yaml2python.params(val, indents, demo, onlyInclude, debug=False)[source]

generates python strng for val, a parameter with a set of attribute

val: as in `name:val` ``` nam:

data_type: string description: Description of this dataset default: UNKNOWN valid: ‘’

``` see getPython

fdi.dataset.yaml2python.readyaml(ypath, version=None, verbose=False)[source]

read YAML files in ypath.

output: nm is stem of file name. desc is descriptor, key being yaml[name]

fdi.dataset.yaml2python.removeParent(a, b)[source]

Returns the one who is the other one’s parent.

fdi.dataset.yaml2python.sq(s)[source]

add quote mark to string, depending on if ‘ or ” in the string.

fdi.dataset.yaml2python.yamlupgrade(descriptors, fins, ypath, version, dry_run=False, verbose=False)[source]
Parameters
  • descriptors: a list of nested dicts describing the data model.

  • version: current version. not that in the yaml to be modified.