fdi.pal package

Subpackages

• fdi.pal.resources package

Submodules

fdi.pal.comparable module

class fdi.pal.comparable.Comparable(**kwds)

基类：object

compareTo(o)

fdi.pal.context module

class fdi.pal.context.AbstractContext(*args, **kwds)

基类：object

A special kind of Product that can hold references to other Products.

This abstract product introduces the lazy loading and saving of references to Products or ProductRefs that it is holding. It remembers its state. http://herschel.esac.esa.int/hcss-doc-15.0/load/hcss_drm/api/herschel/ia/pal/Context.html

Sets rule to None if zInfo does not have refs, else to zInfo['refs'].

__init__(*args, **kwds)

Sets rule to None if zInfo does not have refs, else to zInfo['refs'].

addRule(rule)

Add to the rule that controls the products to be added into the context. The new rule will be old rule AND added rule.

applyRule(*args)

returns True if the input ProductRef passes rule.

If refs is not specified in zInfo return True. If ``zInfo[‘refs’] is empty, return True.

Subclasses can override this method according to what is written in zInfo.

getAllRefs(recursive, includeContexts)

Provides a set of the unique references stored in this context.

getRefs()

Returns the reference container mapping

getRule()

Get the rule that controls the products to be added into the context.

hasDirtyReferences(storage)

Returns a logical to specify whether this context has dirty references or not.

static isContext(cls)

Yields true if specified class belongs to the family of contexts.

isValid()

Provides a mechanism to ensure whether it is valid to store this context in its current state.

readDataset(storage, table, defaultPoolId)

Reads a dataset with information within this context that is normally not accessible from the normal Product interface.

property refs

Property

refsChanged()

Indicates that the references have been changed in memory, which marks this context as dirty.

set(name, refs)

add owner to RefContainer or add named references.

Name

name of the new RefContainer. if is refs, set owner of refs to self.

setRefs(refs)

Changes/Adds the mapping container that holds references.

setRule(rule)

Set the rule that controls the products to be added into the context.

writeDataset(storage)

Creates a dataset with information within this context that is normally not accessible from the normal Product interface.

class fdi.pal.context.Context(*args, **kwds)

基类：fdi.pal.context.AbstractContext, fdi.dataset.baseproduct.BaseProduct

See docstring of AbstractContext.

__init__(*args, **kwds)

applyRule(*args)

returns True if the input ProductRef passes rule.

Default behavior is return the superclass.applyRule() result.

Subclasses can override this method according to what is written in zInfo.

getAllRefs(recursive=False, includeContexts=True, seen=None)

Provides a set of the unique references stored in this Context.

This includes references that are contexts, but not the contents of these subcontexts. This is equivalent to getAllRefs(recursive=false, includeContexts= true). recursive - if true, include references in subcontexts includeContexts - if true, include references to contexts, not including this one

hasDirtyReferences(storage)

Returns a logical to specify whether this context has dirty references or not.

exception fdi.pal.context.ContextRuleException

基类：ValueError

class fdi.pal.context.MapContext(*args, **kwds)

基类：fdi.pal.context.Context

Allows grouping Products into a map of (String, ProductRef) pairs. New entries can be added if they comply to the adding rules of this context. The default behaviour is to allow adding any (String,ProductRef) given that rule is not set.

An example:

image     = ImageProduct(description="hi")
spectrum  = SpectrumProduct(description="there")
simple    = Product(description="everyone")

context=MapContext()
context.refs.put("x",ProductRef(image))
context.refs.put("y",ProductRef(spectrum))
context.refs.put("z",ProductRef(simple))
print context.refs.size() # 3
print context.refs.get('x').product.description # hi
print context.refs.get('y').product.description # there
print context.refs.get('z').product.description # everyone

It is possible to insert a ProductRef at a specific key in the MapContext. The same insertion behaviour is followed as for a Python dict, in that if there is already an existing ProductRef for the given key, that ProductRef is replaced with the new one:

product4=SpectrumProduct(description="everybody")
context.refs.put("y", ProductRef(product4))
product5=SpectrumProduct(description="here")
context.refs.put("a", ProductRef(product5))

print context.refs.get('x').product.description # hi
print context.refs.get('y').product.description # everybody
print context.refs.get('z').product.description # everyone
print context.refs.get('a').product.description # here

Note that the rules are only applied when putting an entry to the map!

Be aware that

1. the put() method of the map view may throw a ContextRuleException if the data added to the context violates the rules applied to the context.

2. the put() method of the map view may throw a ValueError if either of the arguments to the put() method are null.

   BaseProduct–Product

AbstractContext——Contex———MapContext

__init__(*args, **kwds)

addRule(rule)

Add to the rule that controls the products to be added into the context. The new rule will be old rule AND added rule.

applyRule(key, ref)

returns True if the input key name and ProductRef pass rule.

Default behavior is return True if the fully qualified class name of the product which ref refers to equals zInfo['refs'][key], OR the superclass.applyRule() result.

Subclasses can override this method according to what is written in zInfo.

class fdi.pal.context.RefContainer(**kwds)

基类：fdi.dataset.odict.ODict

A map where Rules of its owner Context are applied when put(k,v) is called, and the owner MapContext’s ID can be put to v’s parents list.

Implemented as an ODict that RefContainer has a _STID when json.loads’ed. A MapContext has a _data, which has a refs:RefContainer, which has a datap, which has a name:ProductRef. when used as context.refs.get(‘x’).product.description, the RefContainer is called with get() or __getitem__(), which calls superclass composite’s __getitem__()

__init__(**kwds)

clear()

remove all productRefs

get(key, *args, **kwds)

getOwner()

Returns the reference container mapping

property owner

Property

put(key, ref)

set label-ref pair after validating then add parent to the ref

set(key, ref)

set label-ref pair after validating then add parent to the ref

setOwner(owner)

records who owns this container.

Due to reason described in __setitem__ doc, existing refs will be set again upon owner set

size()

string(level=0, tablefmt='grid', **kwds)

level

default=0,

keyval

default=`None`. If set to a string, ODict class name is not shown and a label of given string is shown with each key as ‘label key:

‘.

tablefmt

=’rst’, tablefmt1=’simple’, tablefmt2=’simple’, matprint=None, trans=True, heavy=True

toString(level=0, tablefmt='grid', **kwds)

level

default=0,

keyval

default=`None`. If set to a string, ODict class name is not shown and a label of given string is shown with each key as ‘label key:

‘.

tablefmt

=’rst’, tablefmt1=’simple’, tablefmt2=’simple’, matprint=None, trans=True, heavy=True

fdi.pal.definable module

class fdi.pal.definable.Definable(**kwds)

基类：object

for items being able to be defined with a Definition.

getDefinition()

Returns the definition associated to this definable item. mh: adopting http://herschel.esac.esa.int/hcss-doc-15.0/load/hcss_drm/api/index.html?herschel/ia/pal/ProductRef.html but parameterize definition does not seem worth it given the few numbers of implemented definitions

fdi.pal.dicthk module

class fdi.pal.dicthk.DictHk(**kwds)

基类：fdi.pal.taggable.Taggable

Definition of services provided by a product storage supporting versioning.

getTagUrnMap()

Get the full tag->urn mappings. mh: returns an iterator

getTags(urn=None)

Get all of the tags that map to a given URN. Get all known tags if urn is not specified. mh: returns an iterator.

getUrn(tag)

Gets the URNs corresponding to the given tag. Returns an empty list if tag does not exist.

getUrnObject(tag)

Gets the URNobjects corresponding to the given tag.

removeTag(tag)

Remove the given tag from the tag and urn maps.

removeUrn(urn)

Remove the given urn from the tag and urn maps.

Only changes maps in memory, not on disk.

removekey(key, themap, thename, othermap, othername)

Remove the given key.

setTag(tag, urn)

Sets the specified tag to the given URN.

tagExists(tag)

Tests if a tag exists.

fdi.pal.httpclientpool module

class fdi.pal.httpclientpool.HttpClientPool(**kwds)

基类：fdi.pal.productpool.ProductPool

the pool will save all products on a remote server.

Initialize connection to the remote server. creates file structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files on server if not exist.

__init__(**kwds)

Initialize connection to the remote server. creates file structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files on server if not exist.

dereference(**kwds)

Decrement the reference count of a ProductRef.

exists(**kwds)

Determines the existence of a product with specified URN.

getCacheInfo(**kwds)

getCount(**kwds)

Return the number of URNs for the product type.

getPoolpath(**kwds)

getProductClasses(**kwds)

Returns all Product classes found in this pool. mh: returns an iterator.

getReferenceCount(**kwds)

Returns the reference count of a ProductRef.

getTagUrnMap(**kwds)

Get the full tag->urn mappings. mh: returns an iterator

getTags(**kwds)

Get all of the tags that map to a given URN. Get all known tags if urn is not specified. mh: returns an iterator.

getUrn(**kwds)

Gets the URNs corresponding to the given tag. Returns an empty list if tag does not exist.

getUrnObject(**kwds)

Gets the URNobjects corresponding to the given tag.

isAlive(**kwds)

Test if the pool is capable of responding to commands.

isEmpty(**kwds)

Determines if the pool is empty.

meta(urn)

Loads the meta-data belonging to the product of specified URN.

readHK()

loads and returns the housekeeping data

reference(**kwds)

Increment the reference count of a ProductRef.

removeAll(**kwds)

Remove all pool data (self, products) and all pool meta data (self, descriptors, indices, etc.).

removeTag(**kwds)

Remove the given tag from the tag and urn maps.

removeUrn(**kwds)

Remove the given urn from the tag and urn maps.

removekey(**kwds)

Remove the given key.

schematicLoad(resourcetype, index, start=None, end=None, serialize_out=False)

does the scheme-specific part of loadProduct.

schematicRemove(urn=None, resourcetype=None, index=None)

does the scheme-specific part of removal.

urn or (resourcetype, index)

schematicSave(products, tag=None, geturnobjs=False, serialize_out=False, **kwds)

does the media-specific saving to remote server.

schematicSelect(**kwds)

to be implemented by subclasses to do the scheme-specific querying.

schematicWipe()

does the scheme-specific remove-all

setTag(**kwds)

Sets the specified tag to the given URN.

setup()

Sets up HttpPool interals.

Make sure that self._poolname and self._poolurl are present.

tagExists(**kwds)

Tests if a tag exists.

fdi.pal.httpclientpool.parseApiArgs1(all_args, serialize_out=False)

parse the command path to get positional and keywords arguments.

all_args: a list of path segments for the args list.

fdi.pal.httpclientpool.serialize_args1(*args, **kwds)

fdi.pal.httpclientpool.toServer(method_name=None)

decorator to divert local calls to server and return what comes back.

fdi.pal.httpclientpool.toserver(self, method, *args, **kwds)

fdi.pal.httpclientpool.writeJsonwithbackup(fp, data)

write data in JSON after backing up the existing one.

fdi.pal.httppool module

class fdi.pal.httppool.HttpPool(**kwds)

基类：fdi.pal.localpool.LocalPool

the pool will save all products locally on the host Http server.

creates file structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files if not exist.

__init__(**kwds)

creates file structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files if not exist.

fdi.pal.localpool module

class fdi.pal.localpool.LocalPool(makenew=True, **kwds)

基类：fdi.pal.productpool.ManagedPool

the pool will save all products in local computer.

Makenew

when the pool does not exist, make a new one (``True`; default) or throws PoolNotFoundError (`False`).

creates file structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files if not exist.

__init__(makenew=True, **kwds)

creates file structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files if not exist.

doLoad(resourcetype, index, start=None, end=None, serialize_out=False)

does the action of loading.

serialize_out: if True returns contents in serialized form.

doRemove(resourcetype, index)

does the action of removal of product from pool.

doSave(resourcetype, index, data, tag=None, serialize_in=True, **kwds)

does the media-specific saving.

index: int

doWipe()

does the action of remove-all

getCacheInfo()

getHead(ref)

Returns the latest version of a given product, belonging to the first pool where the same track id is found.

getMetaByUrn(urn, resourcetype=None, index=None)

Get all of the meta data belonging to a product of a given URN.

mh: returns an iterator.

getMetaByUrnJson(urn, resourcetype, index)

readHK(hktype=None, serialize_out=False)

loads and returns the housekeeping data

hktype: one of ‘classes’, ‘tags’, ‘urns’ to return. default is None to return alldirs serialize_out: if True return serialized form. Default is false.

readmmap(filename, start=None, end=None, close=False, check_time=False)

setMetaByUrn(start, end, urn)

Sets the location of the meta data of the specified data to the given URN.

Data

usually serialized Product.

setup()

Sets up LocalPool interals.

Make sure that self._poolname and self._poolurl are present.

writeHK(fp0=None)

save the housekeeping data to disk

writeJsonmmap(fp, data, serialize_in=True, serialize_out=False, close=False, check_time=False, meta_location=False, **kwds)

write data in JSON from mmap file at fp.

register the file. Leave file open by default close. data: to be serialized and saved. serialize_out: if True returns contents in serialized form. :check_time: to check if file has not been written since we did last time. Default False. :meta_location: return the start and end offsets of metadata in data JSON. Default False. :return: int bytes written. If meta_location is `True`, adding int int start and end point offsets of metadata in seriaized data.

fdi.pal.localpool.wipeLocal(path)

does the scheme-specific remove-all.

A new directory at path will be created.

fdi.pal.mempool module

class fdi.pal.mempool.MemPool(**kwds)

基类：fdi.pal.productpool.ManagedPool

the pool will save all products in memory.

creates data structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files if not exist.

__init__(**kwds)

creates data structure if there isn’t one. if there is, read and populate house-keeping records. create persistent files if not exist.

doLoad(resourcetype, index, start=0, end=0, serialize_out=False)

does the action of loadProduct. note that the index is given as a string.

doRemove(resourcetype, index)

does the action of removal.

doSave(resourcetype, index, data, tag=None, serialize_in=True, **kwds)

does the media-specific saving

doWipe()

does the action of remove-all

getHead(ref)

Returns the latest version of a given product, belonging to the first pool where the same track id is found.

getMetaByUrn(urn, resourcetype=None, index=None)

Get all of the meta data belonging to a product of a given URN.

getPoolSpace()

returns the map of this memory pool.

readHK(hktype=None, serialize_in=True, serialize_out=False)

loads and returns the housekeeping data

hktype: one of ‘classes’, ‘tags’, ‘urns’ to return. default is None to return alldirs serialize_out: if True return serialized form. Default is false.

setMetaByUrn(data, urn)

Sets the location of the meta data of the specified data to the given URN.

Data

usually unserialized Product.

setup()

Sets up MemPool interals.

make sure that self._poolname and self._poolurl are present.

writeHK()

save the housekeeping data to mempool

fdi.pal.poolmanager module

class fdi.pal.poolmanager.PoolManager

基类：object

This class provides the means to reference ProductPool objects without having to hard-code the type of pool. For example, it could be desired to easily switch from one pool type to another.

This is done by calling the getPool() method, which will return an existing pool or create a new one if necessary.

PlacePaths = {'file': '/tmp', 'http': '127.0.0.1:9883/fdi/v0.12', 'https': '127.0.0.1:9883/fdi/v0.12', 'mem': '/', 'server': '/tmp/data'}

classmethod getMap()

Returns a poolname - poolobject map.

classmethod getPool(poolname=None, poolurl=None, pool=None, makenew=True, **kwds)

returns an instance of pool according to name or path of the pool.

Returns the pool object if the pool is registered. Creates the pool if it does not already exist. the same poolname-path always get the same pool. Http pools will be registered on the sserver side.

Pools registered are kept as long as the last reference remains. When the last is gone the pool gets :meth;`removed` d.

poolname

name of the pool.

poolurl

if given the poolpath, scheme, place will be derived from it. if not given for making a new pool (i.e. when poolname is not a registered pool name..

If poolname is missing it is derived from poolurl; if poolurl is also absent, ValueError will be raised.

makenew

when the pool does not exist, make a new one (``True`; default) or throws PoolNotFoundError (`False`).

kwds

passed to pool instanciation arg-list.

Returns

the pool object.

classmethod getPoolurlMap()

Gives the default poolurls of PoolManager.

classmethod isLoaded(poolname)

Whether an item with the given id has been loaded (cached).

返回

the number of remaining week references if the pool is loaded. Returns 0 if poolname is not found in _GlobalPoolList or weakref count is 0.

items()

Returns map’s items

classmethod remove(poolname)

Remove from list and unregister remote pools.

returns 0 for successful removal, 1 for poolname not registered or referenced, still attempted to remove. > 1 for the number of weakrefs the pool still have, and removing failed.

classmethod removeAll()

deletes all pools from the pool list, pools not wiped

classmethod save(poolname, poolobj)

classmethod setPoolurlMap(new)

Sets the default poolurls of PoolManager.

classmethod size()

Gives the number of entries in this manager.

fdi.pal.poolmanager.remoteRegister(p, poolurl)

fdi.pal.poolmanager.remoteUnregister(poolurl)

this method does not reference pool object.

fdi.pal.productpool module

class fdi.pal.productpool.ManagedPool(**kwds)

基类：fdi.pal.productpool.ProductPool, fdi.pal.dicthk.DictHk

A ProductPool that manages its internal house keeping.

Creates and initializes a productpool.

• poolname: if provided will override that in poolurl.

• poolurl: needed to initialize.

dereference(ref)

Decrement the reference count of a ProductRef.

doLoad(resourcetype, index, start=None, end=None, serialize_out=False)

to be implemented by subclasses to do the action of loading

doRemove(resourcetype, index)

to be implemented by subclasses to do the action of reemoving

doSave(resourcetype, index, data, tag=None, serialize_in=True, **kwds)

to be implemented by subclasses to do the action of saving

doSelect(query, results=None)

to be implemented by subclasses to do the action of querying.

doWipe()

to be implemented by subclasses to do the action of wiping.

exists(urn)

Determines the existence of a product with specified URN.

getCacheInfo()

getCount(typename)

Return the number of URNs for the product type.

getMetaByUrn(urn, resourcetype=None, index=None)

Get all of the meta data belonging to a product of a given URN.

mh: returns an iterator.

getPoolpath()

Gets the poolpath of this pool.

poolpath is usually derived from poolurl received from PoolManager during initialization.

getProductClasses()

Returns all Product classes found in this pool. mh: returns an iterator.

getReferenceCount(ref)

Returns the reference count of a ProductRef.

isEmpty()

Determines if the pool is empty.

loadDescriptors(urn)

Loads the descriptors belonging to specified URN.

lockpath(op='w')

Make lock path using transformed poolname as name.

meta(urn)

Loads the meta-data info belonging to the product of specified URN.

meta_filter(q, typename=None, reflist=None, urnlist=None, snlist=None)

returns filtered collection using the query.

q is a MetaQuery valid inputs: typename and ns list; productref list; urn list

prod_filter(q, cls=None, reflist=None, urnlist=None, snlist=None)

returns filtered collection using the query.

q: an AbstractQuery. valid inputs: cls and ns list; productref list; urn list

reference(ref)

Increment the reference count of a ProductRef.

saveOne(prd, tag, geturnobjs, serialize_in, serialize_out, res, kwds)

Save one product.

Res

list of result.

Serialize_out

if True returns contents in serialized form.

schematicLoad(resourcetype, index, start=None, end=None, serialize_out=False)

do the scheme-specific loading

schematicRemove(urn=None, resourcetype=None, index=None)

do the scheme-specific removing

schematicSave(products, tag=None, geturnobjs=False, serialize_in=True, serialize_out=False, **kwds)

do the scheme-specific saving.

Serialize_out

if True returns contents in serialized form.

schematicSelect(query, results=None)

do the scheme-specific querying.

schematicWipe()

do the scheme-specific wiping

setMetaByUrn(start, end, urn)

Sets the location of the meta data of the specified data to the given URN.

Data

usually un/serialized Product.

setup()

Sets up interal machiney of this Pool,

but only if self._poolname and self._poolurl are present, and other pre-requisits are met.

Subclasses should implement own setup(), and make sure that self._poolname and self._poolurl are present with ``

if <pre-requisit not met>: return True if super().setup(): return True

# super().setup() has done its things by now. <do setup> return False

``

returns: True if not both self._poolname and self._poolurl are present.

transformpath(path)

override this to changes the output from the input one (default) to something else.

exception fdi.pal.productpool.PoolNotFoundError

基类：Exception

class fdi.pal.productpool.ProductPool(poolname='', poolurl='', **kwds)

基类：fdi.pal.definable.Definable, fdi.pal.taggable.Taggable, fdi.pal.versionable.Versionable

A mechanism that can store and retrieve Products.

A product pool should not be used directly by users. The general user should access data in a ProductPool through a ProductStorage instance.

When implementing a ProductPool, the following rules need to be applied:

1. Pools must guarantee that a Product saved via the pool saveProduct(Product) method is stored persistently, and that method returns a unique identifier (URN). If it is not possible to save a Product, an IOException shall be raised.

2. A saved Product can be retrieved using the loadProduct(Urn) method, using as the argument the same URN that assigned to that Product in the earlier saveProduct(Product) call. No other Product shall be retrievable by that same URN. If this is not possible, an IOException or GeneralSecurityException is raised.

3. Pools should not implement functionality currently implemented in the core package. Specifically, it should not address functionality provided in the Context abstract class, and it should not implement versioning/cloning support.

Creates and initializes a productpool.

• poolname: if provided will override that in poolurl.

• poolurl: needed to initialize.

exception ParametersIncommpleteError

基类：Exception

__init__(poolname='', poolurl='', **kwds)

Creates and initializes a productpool.

• poolname: if provided will override that in poolurl.

• poolurl: needed to initialize.

accept(visitor)

Hook for adding functionality to object through visitor pattern.

dereference(ref)

Decrement the reference count of a ProductRef.

exists(urn)

Determines the existence of a product with specified URN.

getCount(typename)

Return the number of URNs for the product type.

getDefinition()

Returns pool definition info which contains pool type and other pool specific configuration parameters

getId()

Gets the identifier of this pool.

getPlace()

Gets the place of this pool.

getPoolname()

Gets the poolname of this pool as an Object.

getPoolurl()

Gets the pool URL of this pool.

getProductClasses()

Returns all Product classes found in this pool. mh: returns an iterator.

getReferenceCount(ref)

Returns the reference count of a ProductRef.

getScheme()

Gets the scheme of this pool.

getUrnId()

Get the identifier of this pool used to build URN, usually it’s same as id returned by getId().

isAlive()

Test if the pool is capable of responding to commands.

isEmpty()

Determines if the pool is empty.

loadDescriptors(urn)

Loads the descriptors belonging to specified URN.

loadProduct(urn, serialize_out=False)

Loads a Product belonging to specified URN.

serialize_out: if True returns contents in serialized form.

meta(urn)

Loads the meta-data belonging to the product of specified URN.

property poolname

for property getter

property poolurl

for property getter

reference(ref)

Increment the reference count of a ProductRef.

remove(urn)

Removes a Product belonging to specified URN.

removeAll()

Remove all pool data (self, products) and all pool meta data (self, descriptors, indices, etc.).

saveDescriptors(urn, desc)

Save/Update descriptors in pool.

saveProduct(product, tag=None, geturnobjs=False, serialize_in=True, serialize_out=False, **kwds)

Saves specified product and returns the designated ProductRefs or URNs.

Saves a product or a list of products to the pool, possibly under the supplied tag, and returns the reference (or a list of references if the input is a list of products), or Urns if geturnobjs is True.

See pal document for pool structure.

serialize_out: if True returns contents in serialized form.

schematicLoad(resourcetype, index, start=None, end=None, serialize_out=False)

to be implemented by subclasses to do the scheme-specific loading

schematicRemove(urn=None, resourcetype=None, index=None)

to be implemented by subclasses to do the scheme-specific removing

schematicSave(products, tag=None, geturnobjs=False, serialize_in=True, serialize_out=False, **kwds)

to be implemented by subclasses to do the scheme-specific saving

schematicSelect(query, results=None)

to be implemented by subclasses to do the scheme-specific querying.

schematicWipe()

to be implemented by subclasses to do the scheme-specific wiping.

select(query, variable='m', ptype=<class 'fdi.dataset.product.Product'>, results=None)

Returns a list of references to products that match the specified query.

setPoolname(poolname)

Replaces the current poolname of this pool.

setPoolurl(poolurl)

Replaces the current poolurl of this pool.

setup()

Sets up interal machiney of this Pool,

but only if self._poolname and self._poolurl are present, and other pre-requisits are met.

Subclasses should implement own setup(), and make sure that self._poolname and self._poolurl are present with ``

if <pre-requisit not met>: return True if super().setup(): return True

# super().setup() has done its things by now. <do setup> return False

``

returns: True if not both self._poolname and self._poolurl are present.

fdi.pal.productpool.makeLockpath(direc, op='w')

returns the appropriate path to put lock file.

creats the path if non-existing. Set lockpath-base permission to all-modify so other fdi users can use. op: ‘r’ for readlock no-reading) ‘w’ for writelock (no-writing)

fdi.pal.productref module

class fdi.pal.productref.ProductRef(urn=None, poolname=None, product=None, meta=None, **kwds)

基类：fdi.dataset.metadataholder.MetaDataHolder, fdi.dataset.eq.DeepcmpEqual, fdi.dataset.serializable.Serializable, fdi.pal.comparable.Comparable

A lightweight reference to a product that is stored in a ProductPool or in memory.

Urn can be the string or URNobject.

Poolname if given overrides the pool name in urn, and causes metadata to be loaded from pool, unless this prodref points to a mempool. If meta is given, it will be used instead of that from poolname. A productref created from a single product will result in a memory pool urn, and the metadata won’t be loaded.

__init__(urn=None, poolname=None, product=None, meta=None, **kwds)

Urn can be the string or URNobject.

Poolname if given overrides the pool name in urn, and causes metadata to be loaded from pool, unless this prodref points to a mempool. If meta is given, it will be used instead of that from poolname. A productref created from a single product will result in a memory pool urn, and the metadata won’t be loaded.

addParent(parent)

add a parent

equals(o, verbose=False)

true if o is a non-null ProductRef, with the same Product type than this one, and:

urns and products are null in both refs, or nurs are equal and products are null, or # <– mh urns are null in both refs, and their products are equal, or urns and products are equal in both refs

getHash()

Returns a code number for the product; actually its MD5 signature. This allows checking whether a product already exists in a pool or not.

getMeta()

Returns the metadata of the product.

getParents()

Return the in-memory parent context products of this reference.

That is, the contexts in program memory that contain this product reference object. A context that contains a different product reference object pointing to the same URN is not a parent of this product reference.

Furthermore, it should be understood that this method does not return the parent contexts of the product pointed to by this reference as stored in any underlying pool or storage.

Returns: the parents

getPoolname()

Returns the name of the product pool associated.

getProduct()

Get the product that this reference points to.

If the product is a Context, it is kept internally, so further accesses don’t need to ask the storage for loading it again. Otherwise, the product is returned but the internal reference remains null, so every call to this method involves a request to the storage.

This way, heavy products are not kept in memory after calling this method, thus maintaining the ProductRef a lighweight reference to the target product.

In case of a Context, if it is wanted to free the reference, call unload().

Returns: the product

getSize()

Returns the estimated size(in bytes) of the product in memory.

Useful for providing this information for a user that wants to download the product from a remote site. Returns: the size in bytes

getStorage()

Returns the product storage associated.

getType()

Specifies the Product class to which this Product reference is pointing to.

getUrn()

Returns the Uniform Resource Name (URN) of the product.

getUrnObj()

Returns the URN as an object.

isLoaded()

Informs whether the pointed product is already loaded.

property meta

Property

property parents

property

property product

removeParent(parent)

remove a parent

参数

parent –

setParents(parents)

Sets the in-memory parent context products of this reference.

参数

parents –

setStorage(storage)

Sets the product storage associated.

setUrn(urn)

setUrnObj(urnobj, poolname=None, meta=None)

sets urn

A productref created from a single product will result in a memory pool urn, and the metadata won’t be loaded.

urnobj: Urn a URN object. poolname: str if given overrides the pool name in urn, and causes metadata to be loaded from pool. meta: MetaData If is given, it will be used instead of that from poolname.

string(level=0, **kwds)

toString(level=0, **kwds)

unload()

Clear the cached meta and frees internal reference to the product, so it can be garbage collected.

property urn

Property

property urnobj

Property

fdi.pal.productstorage module

class fdi.pal.productstorage.ProductStorage(pool=None, poolurl=None, **kwds)

基类：object

Logical store created from a pool or a poolURL.

Every instanciation with the same pool will result in a new instance of ProdStorage.

Gets the storage “control pannel” for pool with specifed name.

pool: if is a string will be taken as a poolname. if is a pool object will be registered with its name, poolurl: is sent to the PoolManager with poolname to get the pool object.

__init__(pool=None, poolurl=None, **kwds)

Gets the storage “control pannel” for pool with specifed name.

pool: if is a string will be taken as a poolname. if is a pool object will be registered with its name, poolurl: is sent to the PoolManager with poolname to get the pool object.

accept(visitor)

Hook for adding functionality to object through visitor pattern.

getAllTags()

Get all tags defined in the writable pool.

getHead(ref)

Returns the latest version of a given product, belonging to the first pool where the same track id is found.

getMeta(urn)

Get the metadata belonging to the writable pool that associated to a given URN. returns an ODict.

getPool(poolname)

mh: returns the pool object from poolname

getPools()

Returns the set of ProductPools registered. mh: in a list of poolnames

getProductClasses(poolname)

Yields all Product classes found in this pool.

getTags(urn)

Get the tags belonging to the writable pool that associated to a given URN. returns an iterator.

getUrnFromTag(tag)

Get the URN belonging to the writable pool that is associated to a given tag.

getWritablePool()

returns the poolname of the first pool, which is the only writeable pool.

load(urnortag)

Loads a product with a URN or a list of products with a tag, from the (writeable) pool.

It always creates new ProductRefs. :return: productref if there is only one. A `list` of `ProductRefs`. urnortag: urn or tag

register(poolname=None, poolurl=None, pool=None, **kwds)

Registers the given pools to the storage.

remove(urn)

removes product of urn from the writeable pool

save(product, tag=None, poolname=None, geturnobjs=False, **kwds)

saves to the writable pool if it has been registered.

product: can be one or a list of prpoducts. poolName: if the named pool is not registered, registers and saves. geturnobjs: mh: returns UrnObjs if geturnobjs is True. kwds: options passed to json.dump() for localpools. Returns: one or a list of productref with storage info.

select(query, previous=None)

Returns a list of URNs to products that match the specified query.

Parameters: query - the query object previous - results to be refined Returns: the set of return eferences to products matching the supplied query.

unregister(pool=None, **kwds)

Unregisters the given pools to the storage.

unregisterAll()

wipePool()

Clear all data and meta data of the writable pool.

fdi.pal.query module

class fdi.pal.query.AbstractQuery(product=<class 'fdi.dataset.product.Product'>, variable='p', where='', allVersions=False, **kwds)

基类：fdi.pal.query.StorageQuery

provides default implementations for the pal storage query.

creates an AbstractQuery with product variable name, query string or function.

product: type (Class or ‘Card’ name) of the products to be queried. where: the query string.

__init__(product=<class 'fdi.dataset.product.Product'>, variable='p', where='', allVersions=False, **kwds)

creates an AbstractQuery with product variable name, query string or function.

product: type (Class or ‘Card’ name) of the products to be queried. where: the query string.

class fdi.pal.query.MetaQuery(product=<class 'fdi.dataset.product.Product'>, where='', allVersions=False, **kwds)

基类：fdi.pal.query.AbstractQuery

Meta data query formulates a query on the meta data of a Product.

Typically this type of query is faster than a full query on the Product Access Layer.

creates an MetaQuery with a query string or function.

product: type (Class or ‘Card’ name) of the products to be queried. where: the query string. ‘where’ is a query string or function that returns True or False. In the query string variable name is ‘m’ for a MetaData type, as in m = product.meta.

__init__(product=<class 'fdi.dataset.product.Product'>, where='', allVersions=False, **kwds)

creates an MetaQuery with a query string or function.

product: type (Class or ‘Card’ name) of the products to be queried. where: the query string. ‘where’ is a query string or function that returns True or False. In the query string variable name is ‘m’ for a MetaData type, as in m = product.meta.

class fdi.pal.query.StorageQuery(**kwds)

基类：fdi.dataset.serializable.Serializable

Query on a ProductStorage.

accept(visitor)

Hook for adding functionality to object through visitor pattern.

property allVersions

getAllVersions()

Are all versions to be retrieved, or just the latest?

getType()

Get the class used in the query.

getVariable()

Get the variable name used in the query expression, eg “p”.

getWhere()

Get the query expression to be evaluated.

retrieveAllVersions()

Returns the allVersions related to this object.

setAllVersions(allVersions)

Sets the allVersions of this object.

setType(type)

Sets the type of this object.

setVariable(variable)

Sets the variable of this object.

setWhere(where)

Sets the where of this object.

string(level=0, tablefmt='rst', tablefmt1='simple', tablefmt2='rst', **kwds)

toString(level=0, tablefmt='rst', tablefmt1='simple', tablefmt2='rst', **kwds)

property type

property variable

property where

fdi.pal.taggable module

class fdi.pal.taggable.Taggable(**kwds)

基类：object

Definition of services provided by a product storage supporting tagging.

getTagUrnMap()

Get the full tag->urn mappings. mh: returns an iterator

getTags(urn=None)

Get all of the tags that map to a given URN. Get all known tags if urn is not specified. mh: returns an iterator.

getUrn(tag)

Gets the URNs corresponding to the given tag. Returns an empty list if tag does not exist.

getUrnObject(tag)

Gets the URNobjects corresponding to the given tag.

removeTag(tag)

Remove the given tag from the tag and urn maps.

removeUrn(urn)

Remove the given urn from the tag and urn maps.

removekey(key, themap, thename, othermap, othername)

Remove the given key.

setTag(tag, urn)

Sets the specified tag to the given URN.

tagExists(tag)

Tests if a tag exists.

fdi.pal.urn module

class fdi.pal.urn.Urn(urn=None, poolname=None, cls=None, index=None, poolurl=None, **kwds)

基类：fdi.dataset.eq.DeepcmpEqual, fdi.dataset.serializable.Serializable, fdi.pal.comparable.Comparable

The object representation of the product URN string.

The memory consumed by sets of this object are much less than sets of URN strings.

Only when the class types in URN string are not in classpath, the urn object will consume equals or a little more than URN string as the object has to hold the original urn string. However this should be considered as exceptional cases.

Using this object representation also help to avoid parsing cost of URN string. URN string should be immutable.

About_URN

The Universial Resource Name (URN) string has this format:

urn:<poolname>:<resourcetype>:<serialnumber>

where

<poolname>

Also called poolID. Its requirements is like that to a file name and a URL segment, having only printable characters excluding `` , ``%, ?, *, =, /.

<resourcetype>

class name of the data item (usually Product)

<serialnumber>

internal index for a certain <resourcetype>.

The poolname in a URN is a label. Some examples:

• urn:pool_mh:fdi.dataset.product.Product:2

• urn:20.20:svom.products.SVOMMapContext:0

Storage Pools (subclasses of ProductPool) are where data item reside. The PoolURL is used to give practical information of a pool, such as a poolname, its location, and its access scheme. It is designed to be a local set-up detail that is supposed to be hidden from pool users. Data processing software use ``URN``s to refer to products, without specifying pool location. The poolID in a URN could be a LocalPool on the development laptop and a HTTPClientPool on the production cloud.

Creates the URN object with the urn string or components.

give urn and optional poolurl, or all poolname, cls, index arguments. if urn is given and pool, class, etc are also specified, the latter are ignored. else the URN object is constructed from them. Urn(u) will make a Urn object out of u.

All arguements are None by default.

__init__(urn=None, poolname=None, cls=None, index=None, poolurl=None, **kwds)

Creates the URN object with the urn string or components.

give urn and optional poolurl, or all poolname, cls, index arguments. if urn is given and pool, class, etc are also specified, the latter are ignored. else the URN object is constructed from them. Urn(u) will make a Urn object out of u.

All arguements are None by default.

getIndex()

Returns the product index.

getPlace()

Returns the netloc in this

getPool()

Returns the pool name in this

getPoolId()

Returns the pool URN in this

getPoolpath()

returns the poolpath stored

getScheme()

Returns the urn scheme.

getType()

Returns class type of Urn

getTypeName()

Returns class type name of Urn.

getUrn()

Returns the urn in this

getUrnWithoutPoolId()

property place

property pool

returns the poolname.

setUrn(urn, poolurl=None)

parse urn to get poolname, resource, index.

string(level=0, **kwds)

toString(level=0, **kwds)

property urn

property

class fdi.pal.urn.UrnUtils

基类：object

static checkUrn(identifier)

Throw a ValueError if the identifier is not a legal URN.

static containsUrn(poolobj, urn)

Informs whether a URN belongs to the given pool.

static extractRecordIDs(urns)

Extracts product IDs (serial numbers) from a set of urns.

static getClass(urn)

Get the class contained in a URN.

static getClassName(urn)

Get the class name contained in a URN.

static getLater(urn1, urn2)

Returns the later of two urns.

static getPool(urn, pools)

Returns the pool corresponding to the pool id inside the given urn.

pools: ProductPool or subclass

static getPoolId(urn)

Returns the pool id part of the URN.

static getProductId(urn)

Returns the product id part of the URN, that is, the last token.

static isUrn(identifier)

Informs whether the given identifier corresponds to a URN.

fdi.pal.urn.makeUrn(poolname, typename, index)

assembles a URN with infos of the pool, the resource type, and the index

index: int or string

fdi.pal.urn.parseUrn(urn)

Checks the URN string is valid in its form and splits it.

A Product URN has several segment. For example if the urn is urn:mypool/v2:proj1.product:322 * poolname, also called poolURN or poolID, optionally path-like: mypool/v2, * resource type (usually class) name proj1.product, * index number 322,

returns poolname, resourceclass, index. if urn is None or empty returns (None,None,None)

fdi.pal.urn.parse_poolurl(url, poolhint=None)

Disassambles a pool URL.

A Pool URL is It is generated to desribe . For example:

input: * url: to be decomposed. * poolhint: A urn or a poolname (the first distinctive substring) needs to be given if the poolname has more than one level.

returns: poolpath, scheme, place, poolname. returns (None, None,None,None) if url is None or empty.

About_poolURL

The PoolURL format is in the form of a URL that preceeds its poolname part:

<scheme>://<place><poolpath>/<poolname>

<scheme>

Implementation protocol including file for LocalPool, mem for MemPool, http, https for HttpclientPool.

<place>

IP:port such as``192.168.5.6:8080`` for http and https schemes, or an empty string for file and mem schemes.

<poolname>

same as in URN.

<poolpath>

The part between place and an optional poolhint:

<username>

<password>

• For file or server schemes, e.g. poolpath is /c:/tmp in http://localhost:9000/c:/tmp/mypool/ with poolhint keyword arguement of parse_poolurl() not given, or given as mypool (or myp or my …).

• For http and https schemes, it is e.g. /0.6/tmp in https://10.0.0.114:5000/v0.6/tmp/mypool with poolhint keyword arguement not given, or given as mypool (or myp` or 'my' ...). The meaning of poolpath is subject to interpretation by the  server. In the preceeding example the poolpath has an API version.  :meth:`ProductPool.transformpath` is used to map it further. Note that trailing blank and ``/ are ignored, and stripped in the output.

Examples:

• file:///tmp/mydata for pool `mydata`

• file:///d:/data/test2–v2 for pool test2--v2

• mem:///dummy for pool dummy

• https://10.0.0.114:5000/v0.6/obs for a httpclientpool obs

• server:///tmp/data/0.4/test for a pool test used on a server.

fdi.pal.versionable module

class fdi.pal.versionable.Versionable(**kwds)

基类：object

for items being able to be defined with a Definition.

getLastVersion(ref)

Returns the latest version of the given ProductRef.

getVersions(ref)

Returns all the versions of the given ProductRef.

saveProductRef(ref)

Saves the product referenced and returns the designated URN.

fdi.pal.webapi module