fdi.pal package
Submodules
fdi.pal.backup module
fdi.pal.comparable module
fdi.pal.context module
- class fdi.pal.context.AbstractContext(*args, **kwds)[source]
Bases:
object
A special kind of Product that can hold references to other Products.
This abstract product introduces the lazy loading and saving of references to Products or ProductRefs that it is holding. It remembers its state. http://herschel.esac.esa.int/hcss-doc-15.0/load/hcss_drm/api/herschel/ia/pal/Context.html
Sets
rule
toNone
ifzInfo
does not haverefs
, else tozInfo['refs']
.- __init__(*args, **kwds)[source]
Sets
rule
toNone
ifzInfo
does not haverefs
, else tozInfo['refs']
.
- addRule(rule)[source]
Add to the rule that controls the products to be added into the context. The new rule will be old rule AND added rule.
- applyRule(*args)[source]
returns True if the input ProductRef passes rule.
If
refs
is not specified inzInfo
return True. If ``zInfo[‘refs’] is empty, return True.Subclasses can override this method according to what is written in zInfo.
- getAllRefs(recursive, includeContexts)[source]
Provides a set of the unique references stored in this context.
- hasDirtyReferences(storage)[source]
Returns a logical to specify whether this context has dirty references or not.
- isValid()[source]
Provides a mechanism to ensure whether it is valid to store this context in its current state.
- readDataset(storage, table, defaultPoolId)[source]
Reads a dataset with information within this context that is normally not accessible from the normal Product interface.
- property refs
Property
- refsChanged()[source]
Indicates that the references have been changed in memory, which marks this context as dirty.
- class fdi.pal.context.Context(*args, **kwds)[source]
Bases:
AbstractContext
,BaseProduct
See docstring of AbstractContext.
- applyRule(*args)[source]
returns True if the input ProductRef passes rule.
Default behavior is return the superclass.applyRule() result.
Subclasses can override this method according to what is written in zInfo.
- getAllRefs(recursive=False, includeContexts=True, seen=None)[source]
Provides a set of the unique references stored in this Context.
This includes references that are contexts, but not the contents of these subcontexts. This is equivalent to getAllRefs(recursive=false, includeContexts= true). recursive - if true, include references in subcontexts includeContexts - if true, include references to contexts, not including this one
- class fdi.pal.context.MapContext(*args, **kwds)[source]
Bases:
Context
Allows grouping Products into a map of (String, ProductRef) pairs. New entries can be added if they comply to the adding rules of this context. The default behaviour is to allow adding any (String,ProductRef) given that
rule
is not set.An example:
image = ImageProduct(description="hi") spectrum = SpectrumProduct(description="there") simple = Product(description="everyone") context=MapContext() context.refs.put("x",ProductRef(image)) context.refs.put("y",ProductRef(spectrum)) context.refs.put("z",ProductRef(simple)) print context.refs.size() # 3 print context.refs.get('x').product.description # hi print context.refs.get('y').product.description # there print context.refs.get('z').product.description # everyone
It is possible to insert a ProductRef at a specific key in the MapContext. The same insertion behaviour is followed as for a Python dict, in that if there is already an existing ProductRef for the given key, that ProductRef is replaced with the new one:
product4=SpectrumProduct(description="everybody") context.refs.put("y", ProductRef(product4)) product5=SpectrumProduct(description="here") context.refs.put("a", ProductRef(product5)) print context.refs.get('x').product.description # hi print context.refs.get('y').product.description # everybody print context.refs.get('z').product.description # everyone print context.refs.get('a').product.description # here
Note that the rules are only applied when putting an entry to the map!
Be aware that
the put() method of the map view may throw a ContextRuleException if the data added to the context violates the rules applied to the context.
the put() method of the map view may throw a ValueError if either of the arguments to the put() method are null.
- BaseProduct–Product
AbstractContext——Contex———MapContext
- addRule(rule)[source]
Add to the rule that controls the products to be added into the context. The new rule will be old rule AND added rule.
- applyRule(key, ref)[source]
returns True if the input key name and ProductRef pass rule.
Default behavior is return True if the fully qualified class name of the product which ref refers to equals
zInfo['refs'][key]
, OR the superclass.applyRule() result.Subclasses can override this method according to what is written in zInfo.
- class fdi.pal.context.RefContainer(**kwds)[source]
Bases:
ODict
A map where Rules of its owner Context are applied when put(k,v) is called, and the owner MapContext’s ID can be put to v’s parents list.
Implemented as an ODict that RefContainer has a _STID when json.loads’ed. A MapContext has a _data, which has a refs:RefContainer, which has a datap, which has a name:ProductRef. when used as context.refs.get(‘x’).product.description, the RefContainer is called with get() or __getitem__(), which calls superclass composite’s __getitem__()
- property owner
Property
- put(key, ref)
set label-ref pair after validating then add parent to the ref
- setOwner(owner)[source]
records who owns this container.
Due to reason described in __setitem__ doc, existing refs will be set again upon owner set
- string(level=0, tablefmt='grid', **kwds)
- level:
default=0,
- keyval:
default=`None`. If set to a string, ODict class name is not shown and a label of given string is shown with each key as ‘label key:
- ‘.
- tablefmt:
=’rst’, tablefmt1=’simple’, tablefmt2=’simple’, matprint=None, trans=True, heavy=True
- toString(level=0, tablefmt='grid', **kwds)[source]
- level:
default=0,
- keyval:
default=`None`. If set to a string, ODict class name is not shown and a label of given string is shown with each key as ‘label key:
- ‘.
- tablefmt:
=’rst’, tablefmt1=’simple’, tablefmt2=’simple’, matprint=None, trans=True, heavy=True
- txt(level=0, tablefmt='grid', **kwds)
- level:
default=0,
- keyval:
default=`None`. If set to a string, ODict class name is not shown and a label of given string is shown with each key as ‘label key:
- ‘.
- tablefmt:
=’rst’, tablefmt1=’simple’, tablefmt2=’simple’, matprint=None, trans=True, heavy=True
fdi.pal.definable module
- class fdi.pal.definable.Definable(**kwds)[source]
Bases:
object
for items being able to be defined with a Definition.
- getDefinition()[source]
Returns the definition associated to this definable item. mh: adopting http://herschel.esac.esa.int/hcss-doc-15.0/load/hcss_drm/api/index.html?herschel/ia/pal/ProductRef.html but parameterize definition does not seem worth it given the few numbers of implemented definitions
fdi.pal.dicthk module
- class fdi.pal.dicthk.DictHk(**kwds)[source]
Bases:
Taggable
Definition of services provided by a product storage supporting versioning.
- getTagUrnMap()[source]
Get the full tag->urn mappings.
mh: returns an iterator csdb: csdb/v1/storage/tag?tag=tag1,tag2
- getTags(urn=None, datatype=None, sn=None)[source]
Get all of the tags that map to a given URN or a pair of data type and serial number.
Get all known tags if input arenot specified. mh: returns an iterator.
If datatype and sn are given, use them and ignore urn.
- getUrn(tag)[source]
Gets the URNs corresponding to the given tag.
Returns an empty list if tag is not None and does not exist. curl -X GET “http://123.56.102.90:31702/csdb/v1/storage/info?urns=urn%3Apoolbs%3A20211018%3A1” -H “accept: /”
- getUrnObject(tag)[source]
Gets the URNobjects corresponding to the given tag. Returns an empty list if tag does not exist.
- get_missing(urn, datatype, sn, no_check=False)
make URN(s) if datatype and sn(s) are given and vice versa.
- Parameters:
no_check (bool) – Do not Check if datatype and sn are in the pool’s HK. Default is False
- Returns:
tuple
str, str, int – Refer tp parseUrn.
- Raises:
ValueError if urn not found or not from this pool. –
KeyError if datatype does not exist. –
IndexError if sn does not exist. –
- removeUrn(urn=None, datatype=None, sn=None)[source]
Remove the given urn (or a pair of data type and serial number) from the tag and urn maps.
Only changes maps in memory, not to write on disk here.
- removekey(key, thecontainer, thename, cross_ref_map, othername)[source]
Remove the given key from the map and the counterpart key in the correponding cross_referencing map.
- fdi.pal.dicthk.add_tag_datatype_sn(tag, datatype, sn, dTypes=None, dTags=None)[source]
Static function to add a tag to datatype-sn to pool fmt 2.0
- Parameters:
tag (str) – A tag. Multiple tags have to make multiple calls. None and empty tags are ignored.
datatype (str) – The class name of the data item, new or existing.
sn (int) – The serial number in integer.
dTypes (dict) – the first nested mapping of pool fmt 2.
dTags (dict) – the tag mapping of pool fmt 2.
- fdi.pal.dicthk.get_missing(self, urn, datatype, sn, no_check=False)[source]
make URN(s) if datatype and sn(s) are given and vice versa.
- Parameters:
no_check (bool) – Do not Check if datatype and sn are in the pool’s HK. Default is False
- Returns:
tuple
str, str, int – Refer tp parseUrn.
- Raises:
ValueError if urn not found or not from this pool. –
KeyError if datatype does not exist. –
IndexError if sn does not exist. –
fdi.pal.httpclientpool module
fdi.pal.httppool module
fdi.pal.localpool module
- class fdi.pal.localpool.LocalPool(makenew=True, **kwds)[source]
Bases:
ManagedPool
the pool will save all products in local computer.
- Makenew:
when the pool does not exist, make a new one (
``True`
; default) or throws PoolNotFoundError (`False`
).
creates file structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files if not exist.
- __init__(makenew=True, **kwds)[source]
creates file structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files if not exist.
- doLoad(resourcetype, index, start=None, end=None, serialize_out=False)[source]
does the action of loading.
serialize_out: if True returns contents in serialized form.
- doSave(resourcetype, index, data, tags=None, serialize_in=True, **kwds)[source]
does the media-specific saving.
index: int
- getHead(ref)[source]
Returns the latest version of a given product, belonging to the first pool where the same track id is found.
- getMetaByUrn(urn, resourcetype=None, index=None)[source]
Get all of the meta data belonging to a product of a given URN or a pair of data type and serial number.
mh: returns an iterator.
- getMetaByUrnJson(urn, resourcetype, index)[source]
like getMetaByUrn but returns un-deserialized form.
- readHK(hktype=None, serialize_out=False, all_versions=True)[source]
loads and returns the housekeeping data, or empty dict if not found.
hktype: one of the mappings listed in dicthk.HKDBS. serialize_out: if True return serialized form. Default is false.
- setMetaByUrn(start, end, urn=None, datatype=None, sn=None)[source]
Sets the location of the meta data of the specified data to the given URN or a pair of data type and serial number.
- Data:
usually serialized Product.
- setup()[source]
Sets up LocalPool interals.
Make sure that self._poolname and self._poolurl are present.
- writeJsonmmap(fp, data, serialize_in=True, serialize_out=False, close=False, check_time=False, meta_location=False, **kwds)[source]
write data in JSON from mmap file at fp.
register the file. Leave file open by default close. data: to be serialized and saved. serialize_out: if True returns contents in serialized form. :check_time: to check if file has not been written since we did last time. Default False. :meta_location: return the start and end offsets of metadata in data JSON. Default False. :return: int bytes written. If meta_location is
`True`
, adding int int start and end point offsets of metadata in seriaized data.
fdi.pal.mempool module
- class fdi.pal.mempool.MemPool(**kwds)[source]
Bases:
ManagedPool
the pool will save all products in memory.
creates data structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files if not exist.
- __init__(**kwds)[source]
creates data structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files if not exist.
- doLoad(resourcetype, index, start=0, end=0, serialize_out=False)[source]
does the action of loadProduct. note that the index is given as a string.
- doSave(resourcetype, index, data, tags=None, serialize_in=True, **kwds)[source]
does the media-specific saving
- getHead(ref)[source]
Returns the latest version of a given product, belonging to the first pool where the same track id is found.
- getMetaByUrn(urn, resourcetype=None, index=None)[source]
Get all of the meta data belonging to a product of a given URN.
- readHK(hktype=None, serialize_in=True, serialize_out=False)[source]
loads and returns the housekeeping data
hktype: one of ‘classes’, ‘tags’, ‘urns’ to return. default is None to return alldirs serialize_out: if True return serialized form. Default is false.
- setMetaByUrn(data, urn, resourcetype=None, index=None)[source]
Sets the location of the meta data of the specified data to the given URN.
- Data:
usually unserialized Product.
fdi.pal.poolmanager module
- class fdi.pal.poolmanager.PoolManager[source]
Bases:
object
This class provides the means to reference ProductPool objects without having to hard-code the type of pool. For example, it could be desired to easily switch from one pool type to another.
This is done by calling the getPool() method, which will return an existing pool or create a new one if necessary.
- PlacePaths = {'csdb': 'v1', 'file': '/tmp/httppool', 'http': '111.111.111.111:2222/fdi/v0.15', 'https': '111.111.111.111:2222/fdi/v0.15', 'mem': '/', 'server': '/tmp/httppool/data'}
- classmethod getPool(poolname=None, poolurl=None, pool=None, makenew=True, auth=None, client=None, **kwds)[source]
returns an instance of pool according to name or path of the pool.
Returns the pool object if the pool is registered. Creates the pool if it does not already exist. the same poolname-path always get the same pool. Http pools will be registered on the server side.
Pools registered are kept as long as the last reference remains. When the last is gone the pool gets :meth;`removed` d.
- poolnamestr
name of the pool.
- poolurlstr
If given the poolpath, scheme, place will be derived from it. if not given for making a new pool (i.e. when poolname is not a registered pool name. If poolname is missing it is derived from poolurl; if poolurl is also absent, ValueError will be raised.
- pool: ProductPool
If auth and client are given they will substitute those of pool. If pool is not given, those will need to be given.
- makenewbool
When the pool does not exist, make a new one (
``True`
; default) or throws PoolNotFoundError (`False`
).- authstr
For remoteRegister.
- clientdefault is None.
For remoteRegister.
- kwdsdict
Passed to pool instanciation arg-list.
- ProductPool:
The pool object.
- classmethod isLoaded(poolname)[source]
Whether an item with the given id has been loaded (cached).
- Returns:
the number of remaining week references if the pool is loaded. Returns 0 if poolname is not found in _GlobalPoolList or weakref count is 0.
- classmethod remove(poolname, ignore_error=False)[source]
Remove from list and unregister remote pools.
returns 0 for successful removal,
1
for poolname not registered or referenced, still attempted to remove.> 1
for the number of weakrefs the pool still have, and removing failed.
- fdi.pal.poolmanager.remoteRegister(pool)[source]
if registered a pool’s auth and client will be used.
Note that a new http/csdb pool gets remoteRegistered before locally registered.
Parameter
- poolHttpClientPool, PublicClientPool
Pool object to be registered on remote server and have client/session set.
- authobject
Authorization object for the client. If given will substitute that of pool, if pool has auth.
- clientflask.requests (testing), or requests.Session
The client. If given will substitute that of pool, if pool is given
fdi.pal.productpool module
- class fdi.pal.productpool.ManagedPool(**kwds)[source]
Bases:
ProductPool
,DictHk
A ProductPool that manages its internal house keeping.
Creates and initializes a productpool.
poolname: if provided will override that in poolurl.
poolurl: needed to initialize.
- doLoad(resourcetype, index, start=None, end=None, serialize_out=False)[source]
to be implemented by subclasses to do the action of loading
- doRemove(resourcetype, index, ayn=False)[source]
to be implemented by subclasses to do the action of reemoving
- doSave(resourcetype, index, data, tags=None, serialize_in=True, **kwds)[source]
to be implemented by subclasses to do the action of saving
- doSelect(query, previous=None)[source]
to be implemented by subclasses to do the action of querying.
- exists(urn, resourcetype=None, index=None)[source]
Determines the existence of a product with specified URN.
- getMetaByUrn(urn=None, resourcetype=None, index=None)[source]
Get all of the meta data belonging to a product of a given URN.
mh: returns an iterator.
- getPoolpath()[source]
Gets the poolpath of this pool.
poolpath is usually derived from poolurl received from
PoolManager
during initialization.
- getProductClasses()[source]
Returns all Product classes found in this pool. mh: returns an iterator.
- loadDescriptors(urn, resourcetype=None, index=None)[source]
Loads the descriptors belonging to specified URN.
- meta_filter(q, typename=None, reflist=None, urnlist=None, snlist=None, datatypes=None)[source]
returns filtered collection using the query.
q is a MetaQuery valid inputs: typename and ns list; productref list; urn list; datatypes dict.
- Typename:
data type (class name)
- Reflist:
list of ProductRefs
- Urnlist:
list of URNs
- Datatypes:
dict of {typename:sn_list}
- prod_filter(q, cls=None, reflist=None, urnlist=None, snlist=None, datatypes=None)[source]
returns filtered collection using the query.
q: an AbstractQuery. valid inputs: cls and ns list; productref list; urn list; datatypes dict.
- Cls:
type. data type
- Reflist:
list of ProductRefs
- Urnlist:
list of URNs
- Datatypes:
dict of {cls:sn_list}
- readHK()[source]
Subclass should overide this.
- Returns:
- Of 5 dicts that are the legacy self._classes, self._tags,
- self._urns, and
self._dTypes, self._dTags
- Return type:
tuple
- saveOne(prd, tag, geturnobjs, serialize_in, serialize_out, res, **kwds)[source]
Save one product.
# get the latest HK
- the new classes and tags.
- dTypes combines old classes and urns, is defined as::
- $class_name0:
‘currentSN’: $sn ‘sn’:
- $sn0:
tags: [$tag1, $tag2, … meta: [$start, $end] refcnt: $count
- $sn1:
tags: [$tag1, $tag2, … meta: [$start, $end] refcnt: $count
- example::
- ‘foo.bar.Bar’:
‘currentSN’: 1 0:
‘tags’: [‘cat’, ‘white’] ‘meta’: [123, 456] ‘refcnt’: 0
- 1:
‘tags’: [‘dog’, ‘white’] ‘meta’: [321, 765] ‘refcnt’: 0
- ‘foo.baz.Baz’:
‘currentSN’: 34 34:
‘tags’: [‘tree’, ‘green’] ‘meta’: [100, 654] ‘refcnt’: 1
- dTags differs from tag 1. uses dType:[sn], instead of urn, so there is no poolname anywhere, 2. simplify by removing second level dict::
- $tag_name0:
$class_name1:[$sn1, $sn2…] $class_name2:[$sn3, …]
- example::
‘cat’: { ‘foo.bar.Bar’:[0] } ‘white’: { ‘foo.bar.Bar’; [0, 1] } ‘dog’: …
- res:
list of results.
- serialize_out:
if True returns contents in serialized form.
list of the following: ProductRef. Urn if geturnobjs is set. if`serialze_out` is set for ProductRef no product metadata is stored in the returned instance. The result is also stored in the re parameter.
- schematicLoad(resourcetype, index, start=None, end=None, serialize_out=False)[source]
do the scheme-specific loading
- schematicRemove(urn=None, resourcetype=None, index=None, asyn=False, **kwds)[source]
do the scheme-specific removing
- schematicSave(products, tag=None, geturnobjs=False, serialize_in=True, serialize_out=False, asyn=False, **kwds)[source]
do the scheme-specific saving.
- Parameters:
product (BaseProduct, list) – Product or a list of them or ‘[ size1, prd, size2, prd2, …]’.
tag (str, list) – If given a tag, all products will be having this tag.
If a list tags are given to every one product then the
number of tags must not be the same to that of `product`. If
they are equal, each tag is goven to the product at the same
index in the `product` list.
serialize_out (bool) – if True returns contents in serialized form.
serialize_in (bool) – If set, product input is serialized.
- Returns:
ProductRef (Product reference.)
Urn (If geturnobjs is set.)
str (If serialze_out is set, serialized form of ProductRef or URN.)
list (list of the above of input is a list.)
- setMetaByUrn(start, end, urn=None, datatype=None, sn=None)[source]
Sets the location of the meta data of the specified data to the given URN or a pair of data type and serial number.
- Data:
usually un/serialized Product.
Return :urn: stirng :datatype: class name :sn: serial number in string.
- setup()[source]
- Sets up interal machiney of this Pool,
but only if self._poolname and self._poolurl are present, and other pre-requisits are met.
Subclasses should implement own setup(), and make sure that self._poolname and self._poolurl are present with ``
if <pre-requisit not met>: return True if super().setup(): return True
# super().setup() has done its things by now. <do setup> return False
- ``
returns: True if not both self._poolname and self._poolurl are present.
- class fdi.pal.productpool.ProductPool(poolname='', poolurl='', **kwds)[source]
Bases:
Definable
,Taggable
,Versionable
A mechanism that can store and retrieve Products.
A product pool should not be used directly by users. The general user should access data in a ProductPool through a ProductStorage instance.
When implementing a ProductPool, the following rules need to be applied:
Pools must guarantee that a Product saved via the pool saveProduct(Product) method is stored persistently, and that method returns a unique identifier (URN). If it is not possible to save a Product, an IOException shall be raised.
A saved Product can be retrieved using the loadProduct(Urn) method, using as the argument the same URN that assigned to that Product in the earlier saveProduct(Product) call. No other Product shall be retrievable by that same URN. If this is not possible, an IOException or GeneralSecurityException is raised.
Pools should not implement functionality currently implemented in the core package. Specifically, it should not address functionality provided in the Context abstract class, and it should not implement versioning/cloning support.
Creates and initializes a productpool.
poolname: if provided will override that in poolurl.
poolurl: needed to initialize.
- __init__(poolname='', poolurl='', **kwds)[source]
Creates and initializes a productpool.
poolname: if provided will override that in poolurl.
poolurl: needed to initialize.
- property count
for property getter
- getDefinition()[source]
Returns pool definition info which contains pool type and other pool specific configuration parameters
- getProductClasses()[source]
Returns all Product classes found in this pool. mh: returns an iterator.
- getUrnId()[source]
Get the identifier of this pool used to build URN, usually it’s same as id returned by getId().
- loadProduct(urn, serialize_out=False, asyn=False)[source]
Loads a Product belonging to specified URN.
serialize_out: if True returns contents in serialized form.
- property poolname
for property getter
- property poolurl
for property getter
- qm(qw, prod='BaseProduct', urns=None)[source]
short-hand method for select(qw, variable’m’, ptype=prod, previous=urns.
example: ..code: curl http://foo.edu:23456/data/pool/api/qm__m[“age”]>66 and m[“name”]==”Bob”’
- remove(urn=None, resourcetype=None, index=None, ignore_error=False, asyn=False, **kwds)[source]
Removes a Product belonging to specified URN or a pair of data type and serial number.
- removeAll(ignore_error=False, asyn=False, **kwds)[source]
Remove all pool data (self, products) and all pool meta data (self, descriptors, indices, etc.).
- saveProduct(product, tag=None, geturnobjs=False, serialize_in=True, serialize_out=False, asyn=False, **kwds)[source]
Saves specified product(s) and returns the designated ProductRefs or URNs.
Saves a product or a list of products to the pool, possibly under the supplied tag(s), and returns the reference (or a list of references if the input is a list of products), or Urns if geturnobjs is True.
See pal document for pool structure.
- Parameters:
product (BaseProduct, list) – Product or a list of them or ‘[ size1, prd, size2, prd2, …]’.
tag (str, list) – If given a tag, all products will be having this tag.
If a list tags are given to every one product then the
number of tags must not be the same to that of `product`. If
they are equal, each tag is goven to the product at the same
index in the `product` list.
serialize_out (bool) – if True returns contents in serialized form.
serialize_in (bool) – If set, product input is serialized.
- schematicLoad(resourcetype, index, start=None, end=None, serialize_out=False)[source]
to be implemented by subclasses to do the scheme-specific loading
- schematicRemove(urn=None, resourcetype=None, index=None, asyn=False, **kwds)[source]
to be implemented by subclasses to do the scheme-specific removing
- schematicSave(products, tag=None, geturnobjs=False, serialize_in=True, serialize_out=False, asyn=False, **kwds)[source]
to be implemented by subclasses to do the scheme-specific saving
- schematicSelect(query, previous=None)[source]
to be implemented by subclasses to do the scheme-specific querying.
- select(query, variable='m', ptype=<class 'fdi.dataset.product.Product'>, previous=None)[source]
Returns a list of references to products that match the specified query.
- Parameters:
query (str) – the ‘where’ query string to make a query object.
variable (str) – name of the dummy variable in the query string. if variable is ‘m’, query goes via MetaQuery(ptype, query) ; else by AbstractQuery(ptype, variable, query) .
ptype (class) – The class object whose instances are to be queried. Or fragment of the name of such classes.
previous (list or str) – of urns, possibly from previous search. or a string of comma-separated urns, e.g. ‘urn:a:foo:12,urn:b:bar:9’
- Returns:
of found URNs.
- Return type:
list
- setup()[source]
- Sets up interal machiney of this Pool,
but only if self._poolname and self._poolurl are present, and other pre-requisits are met.
Subclasses should implement own setup(), and make sure that self._poolname and self._poolurl are present with ``
if <pre-requisit not met>: return True if super().setup(): return True
# super().setup() has done its things by now. <do setup> return False
- ``
returns: True if not both self._poolname and self._poolurl are present.
- fdi.pal.productpool.makeLockpath(direc, op='w')[source]
returns the appropriate path to put lock file.
creats the path if non-existing. Set lockpath-base permission to all-modify so other fdi users can use. op: ‘r’ for readlock no-reading) ‘w’ for writelock (no-writing)
- fdi.pal.productpool.populate_pool2(tags, ptype, sn=None, dTypes=None, dTags=None)[source]
- tagslist
The tags in a list. None and empty tags are ignored.
- ptypestr
The product name / datatype / class name of the data item, new or existing.
- snstr
Serial number. If is None, use the one in dTypes.
- Returns:
dTypes and dTags with updates, and the index/serial number
- Return type:
tuple
fdi.pal.productref module
- class fdi.pal.productref.ProductRef(urn=None, poolname=None, product=None, meta=None, poolmanager=None, **kwds)[source]
Bases:
MetaDataHolder
,DeepcmpEqual
,Serializable
,Comparable
A lightweight reference to a product that is stored in a ProductPool or in memory.
Urn can be the string or URNobject.
- Parameters:
urn (str)
poolname (str) – If given overrides the pool name in urn, and causes metadata to be loaded from pool, unless this prodref points to a mempool.
product (subclass of BaseProduct) – A productref created from a single product will result in a memory pool urn, and the metadata won’t be loaded.
meta (Metadata) – If meta is given, it will be used instead of that from poolname.
poolmanager (class) – subclass od PoolManager
**kwds
- Returns:
Reference of a product.
- Return type:
- __init__(urn=None, poolname=None, product=None, meta=None, poolmanager=None, **kwds)[source]
Urn can be the string or URNobject.
- Parameters:
urn (str)
poolname (str) – If given overrides the pool name in urn, and causes metadata to be loaded from pool, unless this prodref points to a mempool.
product (subclass of BaseProduct) – A productref created from a single product will result in a memory pool urn, and the metadata won’t be loaded.
meta (Metadata) – If meta is given, it will be used instead of that from poolname.
poolmanager (class) – subclass od PoolManager
**kwds
- Returns:
Reference of a product.
- Return type:
- equals(o, verbose=False)[source]
true if o is a non-null ProductRef, with the same Product type than this one, and:
urns and products are null in both refs, or unrs are equal and products are null, or # <– mh urns are null in both refs, and their products are equal, or urns and products are equal in both refs
- getHash()[source]
Returns a code number for the product; actually its MD5 signature. This allows checking whether a product already exists in a pool or not.
- getParents()[source]
Return the in-memory parent context products of this reference.
That is, the contexts in program memory that contain this product reference object. A context that contains a different product reference object pointing to the same URN is not a parent of this product reference.
Furthermore, it should be understood that this method does not return the parent contexts of the product pointed to by this reference as stored in any underlying pool or storage.
Returns: the parents
- getPoolname()[source]
Returns the name of the product pool associated.
If not set, poolname from getUrnObj().getPoolname() is used and set to self.
- getProduct()[source]
Get the product that this reference points to.
If the product is a Context, it is kept internally, so further accesses don’t need to ask the storage for loading it again. Otherwise, the product is returned but the internal reference remains null, so every call to this method involves a request to the storage.
This way, heavy products are not kept in memory after calling this method, thus maintaining the ProductRef a lighweight reference to the target product.
In case of a Context, if it is wanted to free the reference, call unload().
Returns: the product
- getSize()[source]
Returns the estimated size(in bytes) of the product in memory.
Useful for providing this information for a user that wants to download the product from a remote site. Returns: the size in bytes
- property meta
Property
- property parents
property
- property product
- setParents(parents)[source]
Sets the in-memory parent context products of this reference.
- Parameters:
parents –
- setUrnObj(urnobj, poolname=None, meta=None)[source]
sets urn
A productref created from a single product will result in a memory pool urn, and the metadata won’t be loaded.
Parameters:
- urnobjUrn
a URN object.
- poolnamestr
if given overrides the pool name in urn, and causes metadata to be loaded from pool.
- meta: MetaData
If is given, it will be used instead of that from poolname.
- string(level=0, **kwds)
- txt(level=0, **kwds)
- unload()[source]
Clear the cached meta and frees internal reference to the product, so it can be garbage collected.
- property urn
Property
- property urnobj
Property
fdi.pal.productstorage module
- class fdi.pal.productstorage.ProductStorage(pool=None, poolurl=None, poolmanager=None, **kwds)[source]
Bases:
object
Logical store created from a pool or a poolURL.
Every instanciation with the same pool will result in a new instance of ProdStorage.
Gets the storage “control pannel” for pool with specifed name.
auth with client, can be given here to be passed to PoolManager.getPool. client how to call remote api if poolurl indicates a remote pool. Default is None for using the configured host/port/credentials. If doing a mocked server test, this needs to be set. :pool: can be given as the only pramemter. If auth and client are given they will substitute those of pool. If pool is not given, those will need to be given. :poolname: if is a string will be taken as a poolname. if is a pool object will be registered with its name, :poolurl: is sent to the PoolManager with poolname to get the pool object.
- __init__(pool=None, poolurl=None, poolmanager=None, **kwds)[source]
Gets the storage “control pannel” for pool with specifed name.
auth with client, can be given here to be passed to PoolManager.getPool. client how to call remote api if poolurl indicates a remote pool. Default is None for using the configured host/port/credentials. If doing a mocked server test, this needs to be set. :pool: can be given as the only pramemter. If auth and client are given they will substitute those of pool. If pool is not given, those will need to be given. :poolname: if is a string will be taken as a poolname. if is a pool object will be registered with its name, :poolurl: is sent to the PoolManager with poolname to get the pool object.
- getHead(ref)[source]
Returns the latest version of a given product, belonging to the first pool where the same track id is found.
- getMeta(urn)[source]
Get the metadata belonging to the writable pool that associated to a given URN. returns an ODict.
- getPools()[source]
Returns the set of ProductPools registered. mh: in a list of poolnames XXX TODO: getPoolnames
- getTags(urn)[source]
Get the tags belonging to the writable pool that associated to a given URN. returns an iterator.
- getUrnFromTag(tag)[source]
Get the URN belonging to the writable pool that is associated to a given tag.
- getWritablePool(obj=False)[source]
returns the poolname of the first pool, which is the only writeable pool.
- Obj:
(bool) return the pool objject instead of the name.
- load(urnortag)[source]
Loads a product with a URN or a list of products with a tag, from the (writeable) pool.
It always creates new ProductRefs. :return: productref if there is only one. A
`list`
of`ProductRefs`
. urnortag: urn or tag
- register(poolname=None, poolurl=None, pool=None, **kwds)[source]
Registers the given pools to the storage.
- Client:
passed to PoolManager.getPool.
- Auth:
passed to PoolManager.getPool.
- remove(urn, datatype=None, index=None, ignore_error=False)[source]
removes product of urn from the writeable pool
- save(product, tag=None, poolname=None, geturnobjs=False, asyn=False, **kwds)[source]
saves to the writable pool if it has been registered.
- Parameters:
product (BaseProduct, list) – Product or a list of them or ‘[ size1, prd, size2, prd2, …]’.
tag (str, list) – If given a tag, all products will be having this tag.
If a list tags are given to every one product then the
number of tags must not be the same to that of `product`. If
they are equal, each tag is goven to the product at the same
index in the `product` list.
serialize_out (bool) – if True returns contents in serialized form.
serialize_in (bool) – If set, product input is serialized.
poolName (str) – If the named pool is not registered, registers and saves.
geturnobjs (bool) – returns UrnObjs if geturnobjs is True.
kwds (options passed to json.dump() for subclasses.)
- Returns:
ProductRef (Product reference.)
Urn (If geturnobjs is set.)
str (If serialze_out is set, serialized form of ProductRef or URN.)
list (list of the above of input is a list.)
- select(query, variable=None, ptype=None, previous=None)[source]
Returns a list of URNs to products that match the specified query.
- Parameters:
query (the query object, or str) –
- The Query instances or
the ‘where’ query string to make a query object.
variable (str) – name of the dummy variable in the query string. if variable is ‘m’, query goes via MetaQuery(ptype, query) ; else by AbstractQuery(ptype, variable, query) .
ptype (class) – The class object whose instances are to be queried. Or fragment of the name of such classes.
previous (list or str) – of urns, possibly from previous search. or a string of comma-separated urns, e.g. ‘urn:a:foo:12,urn:b:bar:9’
- Return type:
the set of return eferences to products matching the supplied query.
fdi.pal.publicclientpool module
fdi.pal.query module
- class fdi.pal.query.AbstractQuery(product=<class 'fdi.dataset.product.Product'>, variable='p', where='', allVersions=False, **kwds)[source]
Bases:
StorageQuery
provides default implementations for the pal storage query.
creates an AbstractQuery with product variable name, query string or function.
product: type (Class or ‘Card’ name) of the products to be queried. where: the query string.
- class fdi.pal.query.MetaQuery(product=<class 'fdi.dataset.product.Product'>, where='', allVersions=False, **kwds)[source]
Bases:
AbstractQuery
Meta data query formulates a query on the meta data of a Product.
Typically this type of query is faster than a full query on the Product Access Layer.
creates an MetaQuery with a query string or function.
product: type (Class or ‘Card’ name) of the products to be queried. where: the query string. ‘where’ is a query string or function that returns True or False. In the query string variable name is ‘m’ for a MetaData type, as in
m = product.meta
.- __init__(product=<class 'fdi.dataset.product.Product'>, where='', allVersions=False, **kwds)[source]
creates an MetaQuery with a query string or function.
product: type (Class or ‘Card’ name) of the products to be queried. where: the query string. ‘where’ is a query string or function that returns True or False. In the query string variable name is ‘m’ for a MetaData type, as in
m = product.meta
.
- class fdi.pal.query.StorageQuery(**kwds)[source]
Bases:
Serializable
Query on a ProductStorage.
- property allVersions
- string(level=0, tablefmt='rst', tablefmt1='simple', tablefmt2='rst', **kwds)
- txt(level=0, tablefmt='rst', tablefmt1='simple', tablefmt2='rst', **kwds)
- property type
- property variable
- property where
fdi.pal.taggable module
- class fdi.pal.taggable.Taggable(**kwds)[source]
Bases:
object
Definition of services provided by a product storage supporting tagging.
- getTags(urn=None, datatype=None, sn=None)[source]
Get all of the tags that map to a given URN or a pair of data type and serial number.
Get all known tags if urn is not specified. mh: returns an iterator.
If datatype and sn are given, use them and ignore urn.
- getUrn(tag)[source]
Gets the URNs corresponding to the given tag. Returns an empty list if tag does not exist.
fdi.pal.urn module
- class fdi.pal.urn.Urn(urn=None, poolname=None, cls=None, index=None, poolurl=None, **kwds)[source]
Bases:
DeepcmpEqual
,Serializable
,Comparable
The object representation of the product URN string.
The memory consumed by sets of this object are much less than sets of URN strings.
Only when the class types in URN string are not in classpath, the urn object will consume equals or a little more than URN string as the object has to hold the original urn string. However this should be considered as exceptional cases.
Using this object representation also help to avoid parsing cost of URN string. URN string should be immutable.
About_URN
The Universial Resource Name (URN, https://datatracker.ietf.org/doc/html/rfc2141 ) string has this format:
urn:<poolname>:<resourcetype>:<serialnumber>
with modified rules desribed below.
- <poolname>:
Also called poolID. It consists of 1-32 characters, is case-sensitive, which deviates from rfc2141. Character allowed are
alpha
,digit
,safe
, defined in rfc1630 (https://datatracker.ietf.org/doc/html/rfc1630). These are excluded: ``, ``%
,?
,!
,*
,``’, ``"
,(
,)
,=
,/
, and what listed inmod:poolmanager:Invalid_Pool_Names
, e.g.pools
,urn
,URN
,api
.- <resourcetype>:
type name of the data item (usually class name of data products inheriting
BaseProduct
)- <serialnumber>:
internal index for a certain <resourcetype>.
The
poolname
in a URN is a label. Some examples:URNs are used to to identify data be cause URNs are location agnostic. Storage Pools (subclasses of
ProductPool
) are where data item reside. The PoolURL is used to give practical information of a pool, such as a poolname, its location, and its access scheme. PoolURL is designed to be a local set-up detail that is supposed to be hidden from pool users. Data processing software use ``URN``s to refer to products, without specifying pool location. The poolID in a URN could be aLocalPool
on the development laptop and aHTTPClientPool
on the production cloud.Creates the URN object with the urn string or components.
give urn and optional poolurl, or all poolname, cls, index arguments. if urn is given and pool, class, etc are also specified, the latter are ignored. else the URN object is constructed from them. Urn(u) will make a Urn object out of u.
All arguements are None by default.
Parameters urn: string. A URM string. poolname: string, provides pool name part of URN if URN is missing from input. cls: type. the full class name is used for datatype. index: int. poolurl: string. IF specified will provide info of the pool involved.
- __init__(urn=None, poolname=None, cls=None, index=None, poolurl=None, **kwds)[source]
Creates the URN object with the urn string or components.
give urn and optional poolurl, or all poolname, cls, index arguments. if urn is given and pool, class, etc are also specified, the latter are ignored. else the URN object is constructed from them. Urn(u) will make a Urn object out of u.
All arguements are None by default.
Parameters urn: string. A URM string. poolname: string, provides pool name part of URN if URN is missing from input. cls: type. the full class name is used for datatype. index: int. poolurl: string. IF specified will provide info of the pool involved.
- property place
- property pool
returns the poolname.
- string(level=0, **kwds)
- txt(level=0, **kwds)
- property urn
property
- class fdi.pal.urn.UrnUtils[source]
Bases:
object
- fdi.pal.urn.makeUrn(poolname, typename, index)[source]
assembles a URN or a list of URNs with infos of the pool, the resource type, and the index.
- Poolname:
str or list of them.
- Typename:
str, name of data type. or list of them.
index: int or string or list of them. If any two or more are lists, all have to be list of the same length.
- fdi.pal.urn.parseUrn(urn, int_index=True, check_poolename=None)[source]
Checks the URN string is valid in its form and splits it.
A Product URN has several segment. For example if the urn is
urn:mypool/v2:proj1.product:322
* poolname, also called poolURN or poolID, optionally path-like:mypool/v2
, * resource type (usually class) nameproj1.product
, * index number322
, If urn is None or empty returns (None,None,None)Parameter
- urnstr,list
One or a list of URN strings to be decomposed.
- int_indexbool
If True (default) returns integer index, else string index.
- check_poolenamestr
Raise ValueError is any urn has a poolname different from the value of check_poolename.
- returns:
tuple
* If urn is None or a zero-length string, returns (None, None, None).
* If urn is a non-zero-length string, returns a tuple of – :poolname: Name of the pool :resourceclass: type of resource/products :index: One or a list of (int) serial number of resourceclass in the pool.
* If urn is a list URNs and all of them have identical poolname and identical resourceclasses, returns a tuple (poolname, resourceclass, list-of-index])
* If urn is a list URNs and not all of them have identical poolname or not identical resourceclasses, returns a tuple (list_of_poolname, list_of_resourceclass, list-of-index), …)
- fdi.pal.urn.parse_poolurl(url, poolhint=None)[source]
Disassambles a pool URL.
A Pool URL is It is generated to desribe . For example:
input: * url: to be decomposed. * poolhint: A urn or a poolname (the first distinctive substring) needs to be given if the poolname has more than one level.
returns: poolpath, scheme, place, poolname. returns (None, None,None,None) if url is None or empty.
About_poolURL
The
PoolURL
format is in the form of a URL that preceeds its poolname part:<scheme>://<place><poolpath>/<poolname>
- <scheme>:
Implementation protocol including
file
forLocalPool
,mem
forMemPool
,http
,https
forHttpclientPool
.- <place>:
IP:port such as``192.168.5.6:8080`` for
http
andhttps
schemes, or an empty string forfile
andmem
schemes.- <poolname>:
same as in URN.
- <poolpath>:
The part between
place
and an optionalpoolhint
:- <username>:
- <password>:
For
file
orserver
schemes, e.g. poolpath is/c:/tmp
inhttp://localhost:9000/c:/tmp/mypool/
withpoolhint
keyword arguement ofparse_poolurl()
not given, or given asmypool
(ormyp
ormy
…).For
http
andhttps
schemes, it is e.g./0.6/tmp
inhttps://10.0.0.114:5000/v0.6/tmp/mypool
withpoolhint
keyword arguement not given, or given asmypool
(ormyp` or 'my' ...). The meaning of poolpath is subject to interpretation by the server. In the preceeding example the poolpath has an API version. :meth:`ProductPool.transformpath` is used to map it further. Note that trailing blank and ``/
are ignored, and stripped in the output.
Examples:
file:///tmp/mydata for pool
`mydata`
file:///d:/data/test2–v2 for pool
test2--v2
mem:///dummy for pool
dummy
https://10.0.0.114:5000/v0.6/obs for a httpclientpool
obs
server:///tmp/data/0.4/test for a pool
test
used on a server.