fdi.pal package

Subpackages

fdi.pal.resources package

Submodules

fdi.pal.common module

fdi.pal.common.getObjectbyId(idn, lgbv): lgb is from deserializing caller’s globals().values() locals().values() and built-ins

fdi.pal.common.getProductObject(urn, lgb=None): Returns a product from URN. returns object.

fdi.pal.comparable module

class fdi.pal.comparable.Comparable(**kwds)

Bases: object

compareTo(o)

fdi.pal.context module

class fdi.pal.context.Context(**kwds)

Bases: Product

A special kind of Product that can hold references to other Products.

This abstract product introduces the lazy loading and saving of references to Products or ProductRefs that it is holding. It remembers its state. http://herschel.esac.esa.int/hcss-doc-15.0/load/hcss_drm/api/herschel/ia/pal/Context.html

getAllRefs(recursive, includeContexts): Provides a set of the unique references stored in this context.

hasDirtyReferences(storage): Returns a logical to specify whether this context has dirty references or not.

static isContext(cls): Yields true if specified class belongs to the family of contexts.

isValid(): Provides a mechanism to ensure whether it is valid to store this context in its current state.

readDataset(storage, table, defaultPoolId): Reads a dataset with information within this context that is normally not accessible from the normal Product interface.

refsChanged(): Indicates that the references have been changed in memory, which marks this context as dirty.

writeDataset(storage): Creates a dataset with information within this context that is normally not accessible from the normal Product interface.

exception fdi.pal.context.ContextRuleException: Bases: ValueError

class fdi.pal.context.MapContext(**kwds)

Bases: Context

Allows grouping Products into a map of (String, ProductRef) pairs. New entries can be added if they comply to the adding rules of this context. The default behaviour is to allow adding any (String,ProductRef).

An example:

image     = ImageProduct(description="hi")
spectrum  = SpectrumProduct(description="there")
simple    = Product(description="everyone")

context=MapContext()
context.refs.put("x",ProductRef(image))
context.refs.put("y",ProductRef(spectrum))
context.refs.put("z",ProductRef(simple))
print context.refs.size() # 3
print context.refs.get('x').product.description # hi
print context.refs.get('y').product.description # there
print context.refs.get('z').product.description # everyone

It is possible to insert a ProductRef at a specific key in the MapContext. The same insertion behaviour is followed as for a Python dict, in that if there is already an existing ProductRef for the given key, that ProductRef is replaced with the new one:

product4=SpectrumProduct(description="everybody")
context.refs.put("y", ProductRef(product4))
product5=SpectrumProduct(description="here")
context.refs.put("a", ProductRef(product5))

print context.refs.get('x').product.description # hi
print context.refs.get('y').product.description # everybody
print context.refs.get('z').product.description # everyone
print context.refs.get('a').product.description # here

Note that the rules are only applied when putting an entry to the map!

Be aware that

the put() method of the map view may throw a ContextRuleException if the data added to the context violates the rules applied to the context.
the put() method of the map view may throw a ValueError if either of the arguments to the put() method are null.

addRule(rule): Add to the rule that controls the products to be added into the context. The new rule will be old rule AND added rule.

getAllRefs(recursive=False, includeContexts=True, seen=None): Provides a set of the unique references stored in this context. This includes references that are contexts, but not the contents of these subcontexts. This is equivalent to getAllRefs(recursive=false, includeContexts= true). recursive - if true, include references in subcontexts includeContexts - if true, include references to contexts, not including this one

getRefs(): Returns the reference container mapping

getRule(): Get the rule that controls the products to be added into the context.

hasDirtyReferences(storage): Returns a logical to specify whether this context has dirty references or not.

static isContext(cls): Yields true if specified class belongs to the family of contexts.

isValid(): Provides a mechanism to ensure whether it is valid to store this context in its current state.

readDataset(storage, table, defaultPoolId): Reads a dataset with information within this context that is normally not accessible from the normal Product interface.

property refs: Property

set(name, refs): add owner to RefContainer

setRefs(refs): Changes/Adds the mapping container that holds references.

setRule(rule): Set the rule that controls the products to be added into the context.

writeDataset(storage): Creates a dataset with information within this context that is normally not accessible from the normal Product interface.

class fdi.pal.context.RefContainer(**kwds)

Bases: Serializable, ODict

A map where Rules of a Context are applied when put(k,v) is called, and the owner MapContext’s ID can be put to v’s parents list.

Implemwnted using dataset.Composite so that RefContainer has a ClassID when json.loads’ed. A MapContext has a _sets, which has a refs:RefContainer, which has a _sets, which has a name:ProductRef. when used as context.refs.get(‘x’).product.description, the RefContainer is called with get() or __getitem__(), which calls superclass composite’s _set’s __getitem__()

clear(): remove all productRefs

get(key)

put(key, ref): set label-ref pair after validating then add parent to the ref

serializable(): Can be encoded with serializableEncoder

set(key, ref)

setOwner(owner): records who owns this container

size()

fdi.pal.context.applyrules(key, ref, rules)

fdi.pal.definable module

class fdi.pal.definable.Definable(**kwds)

Bases: object

for items being able to be defined with a Definition.

getDefinition(): Returns the definition associated to this definable item. mh: adopting http://herschel.esac.esa.int/hcss-doc-15.0/load/hcss_drm/api/index.html?herschel/ia/pal/ProductRef.html but parameterize definition does not seem worth it given the few numbers of implemented definitions

fdi.pal.httpclientpool module

class fdi.pal.httpclientpool.HttpClientPool(**kwds)

Bases: ProductPool

the pool will save all products on a remote server.

getHead(ref): Returns the latest version of a given product, belonging to the first pool where the same track id is found.

readHK(): loads and returns the housekeeping data

schematicLoadProduct(resourcename, indexstr, urn): does the scheme-specific part of loadProduct.

schematicRemove(typename, serialnum, urn): does the scheme-specific part of removal.

schematicSave(typename, serialnum, data, urn, tag=None): does the media-specific saving to remote server save metadata at localpool

schematicWipe(): does the scheme-specific remove-all

transformpath(path): override this to changes the output from the input one (default) to something else.

writeHK(fp0): save the housekeeping data to disk

fdi.pal.httpclientpool.writeJsonwithbackup(fp, data): write data in JSON after backing up the existing one.

fdi.pal.localpool module

class fdi.pal.localpool.LocalPool(**kwds)

Bases: ProductPool

the pool will save all products in local computer.

getHead(ref): Returns the latest version of a given product, belonging to the first pool where the same track id is found.

readHK(): loads and returns the housekeeping data

schematicLoadProduct(resourcename, indexstr, urn=None): does the scheme-specific part of loadProduct.

schematicRemove(typename, serialnum): does the scheme-specific part of removal.

schematicSave(typename, serialnum, data, urn=None, tag=None): does the media-specific saving

schematicWipe(): does the scheme-specific remove-all

transformpath(path): override this to changes the output from the input one (default) to something else.

writeHK(fp0): save the housekeeping data to disk

fdi.pal.localpool.writeJsonwithbackup(fp, data): write data in JSON after backing up the existing one.

fdi.pal.mempool module

class fdi.pal.mempool.MemPool(**kwds)

Bases: ProductPool

the pool will save all products in memory.

getHead(ref): Returns the latest version of a given product, belonging to the first pool where the same track id is found.

getPoolSpace(): returns the map of this memory pool.

readHK(): loads and returns the housekeeping data

schematicLoadProduct(resourcename, indexstr): does the scheme-specific part of loadProduct. note that the index is given as a string.

schematicRemove(typename, serialnum): does the scheme-specific part of removal.

schematicSave(typename, serialnum, data): does the media-specific saving

schematicWipe(): does the scheme-specific remove-all

writeHK(fp0): save the housekeeping data to mempool

fdi.pal.pnspoolserver module

fdi.pal.pnspoolserver.bad_request(error)

fdi.pal.pnspoolserver.calc(d): generates result product directly using data on PNS.

fdi.pal.pnspoolserver.calcresult(cmd, ops='')

fdi.pal.pnspoolserver.checkpath(path)

fdi.pal.pnspoolserver.cleanPTS(d): Removing traces of past runnings the Processing Task Software.

fdi.pal.pnspoolserver.cleanup(cmd): DELETE is used to clean up the Processing Task Software (PST) to its initial configured state.

fdi.pal.pnspoolserver.configPNS(d=None): Configure the PNS itself by replacing the pnsconfig var

fdi.pal.pnspoolserver.configPTS(d=None): Configure the Processing Task Software by running the config script. Ref init PTS.

fdi.pal.pnspoolserver.defaultprocessinput(data): puts all undecoded json to every files.

fdi.pal.pnspoolserver.defaultprocessoutput(filemode): reads each of the files and returns the contents in a filename indexed dict.

fdi.pal.pnspoolserver.dosleep(indata, ops): run ‘sleep [ops]’ in the OS. ops is 3 if not given.

fdi.pal.pnspoolserver.filesin(dir): returns names and contents of all files in the dir, ‘None’ if dir not existing.

fdi.pal.pnspoolserver.genposttestprod(d): generate post test product. put the 1st input (see maketestdata in test_pns.py) parameter to metadata and 2nd to the product’s dataset

fdi.pal.pnspoolserver.get_apis()

fdi.pal.pnspoolserver.getinfo(cmd): returns init, config, run input, run output.

fdi.pal.pnspoolserver.initPTS(d=None): Initialize the Processing Task Software by running the init script defined in the config. Execution on the server host is in the pnshome directory and run result and status are returned. If input/output directories cannot be created with serveruser as owner, Error401 will be given.

fdi.pal.pnspoolserver.makepublicAPI(ops)

fdi.pal.pnspoolserver.not_found(error)

fdi.pal.pnspoolserver.run(d, processinput=None, processoutput=None): Generates a product by running script defined in the config under ‘run’. Execution on the server host is in the pnshome directory and run result and status are returned.

fdi.pal.pnspoolserver.setup(cmd, ops=''): PUT is used to initialize or configure the Processing Task Software (PST).

fdi.pal.pnspoolserver.testinit(d=None): Renames the ‘init’ ‘config’ ‘run’ ‘clean’ scripts to ‘.save’ and points it to the ‘.ori’ scripts.

fdi.pal.pnspoolserver.testrun(d)

fdi.pal.pnspoolserver.unauthorized(error)

fdi.pal.pnspoolserver.uploadScript(op, d=None)

fdi.pal.pnspoolserver.verify(username, password): This function is called to check if a username / password combination is valid.

fdi.pal.poolmanager module

class fdi.pal.poolmanager.PoolManager

Bases: object

This class provides the means to reference ProductPool objects without having to hard-code the type of pool. For example, it could be desired to easily switch from one pool type to another.

This is done by calling the getPool(String) method, which will return an existing pool or create a new one if necessary.

classmethod getMap()

classmethod getPool(poolurn)

returns an instance of pool according to urn.

create the pool if it does not already exist. the same pool-URN always get the same pool.

classmethod isLoaded(poolurn): Whether an item with the given id has been loaded (cached).

classmethod removeAll(): deletes all pools from the pool list, pools unwiped

classmethod save(poolurn, poolobj)

classmethod size(): Gives the number of entries in this manager.

fdi.pal.productpool module

class fdi.pal.productpool.ProductPool(poolurn=None, use_basepoolpath=True, **kwds)

Bases: Definable, Taggable, Versionable

A mechanism that can store and retrieve Products.

A product pool should not be used directly by users. The general user should access data in a ProductPool through a ProductStorage instance.

When implementing a ProductPool, the following rules need to be applied:

Pools must guarantee that a Product saved via the pool saveProduct(Product) method is stored persistently, and that method returns a unique identifier (URN). If it is not possible to save a Product, an IOException shall be raised.

A saved Product can be retrieved using the loadProduct(Urn) method, using as the argument the same URN that assigned to that Product in the earlier saveProduct(Product) call. No other Product shall be retrievable by that same URN. If this is not possible, an IOException or GeneralSecurityException is raised.

Pools should not implement functionality currently implemented in the core paclage. Specifically, it should not address functionality provided in the Context abstract class, and it should not implement versioning/cloning support.

accept(visitor): Hook for adding functionality to object through visitor pattern.

dereference(ref): Decrement the reference count of a ProductRef.

exists(urn): Determines the existence of a product with specified URN.

getDefinition(): Returns pool definition info which contains pool type and other pool specific configuration parameters

getId(): Gets the identifier of this pool.

getProductClasses(): Returns all Product classes found in this pool. mh: returns an iterator.

getReferenceCount(ref): Returns the reference count of a ProductRef.

getUrnId(): Get the identifier of this pool used to build URN, usually it’s same as id returned by getId().

isAlive(): Test if the pool is capable of responding to commands.

isEmpty(): Determines if the pool is empty.

loadDescriptors(urn): Loads the descriptors belonging to specified URN.

loadProduct(urn): Loads a Product belonging to specified URN.

lockpath()

meta(urn): Loads the meta-data belonging to the product of specified URN.

mfilter(q, cls=None, reflist=None, urnlist=None, snlist=None)

returns filtered collection using the query.

q is a MetaQuery valid inputs: cls and ns list; productref list; urn list

pfilter(q, cls=None, reflist=None, urnlist=None, snlist=None)

returns filtered collection using the query.

q is a AbstractQuery. valid inputs: cls and ns list; productref list; urn list

reference(ref): Increment the reference count of a ProductRef.

remove(urn): Removes a Product belonging to specified URN.

removeAll(): Remove all pool data (self, products) and all pool meta data (self, descriptors, indices, etc.).

saveDescriptors(urn, desc): Save/Update descriptors in pool.

saveProduct(product, tag=None, geturnobjs=False)

Saves specified product and returns the designated ProductRefs or URNs. Saves a product or a list of products to the pool, possibly under the supplied tag, and return the reference (or a list of references is the input is a list of products), or Urns if geturnobjs is True.

Pool:!!dict

_classes:!!odict

$product0_class_name:!!dict

currentSN:!!int $the serial number of the latest added prod to the pool

sn:!!list

$serial number of a prod
$serial number of a prod
…

$product1_class_name:!!dict …

_urns:!!odict

$URN0:!!odict

meta:!!MetaData $prod.meta tags:!!list

$tag

$tag

…

_tags:!!odict

urns:!!list

$urn
$urn
…

$urn:!!$serialized product

schematicLoadProduct(resourcename, indexstr): to be implemented by subclasses to do the scheme-specific loading

schematicRemove(typename, serialnum): to be implemented by subclasses to do the scheme-specific removing

schematicSave(typename, serialnum, data): to be implemented by subclasses to do the scheme-specific saving

select(query, results=None): Returns a list of references to products that match the specified query.

fdi.pal.productref module

class fdi.pal.productref.ProductRef(urn=None, poolurn=None, product=None, meta=None, **kwds)

Bases: MetaDataHolder, Serializable, Comparable

A lightweight reference to a product that is stored in a ProductPool or in memory.

addParent(parent): add a parent

equals(o)

true if o is a non-null ProductRef, with the same Product type than this one, and:

urns and products are null in both refs, or nurs are equal and products are null, or # <– mh urns are null in both refs, and their products are equal, or urns and products are equal in both refs

getHash(): Returns a code number for the product; actually its MD5 signature. This allows checking whether a product already exists in a pool or not.

getMeta(): Returns the metadata of the product.

getParents()

Return the in-memory parent context products of this reference. That is, the contexts in program memory that contain this product reference object. A context that contains a different product reference object pointing to the same URN is not a parent of this product reference.

Furthermore, it should be understood that this method does not return the parent contexts of the product pointed to by this reference as stored in any underlying pool or storage.

Returns: the parents

getProduct()

Get the product that this reference points to.

If the product is a Context, it is kept internally, so further accesses don’t need to ask the storage for loading it again. Otherwise, the product is returned but the internal reference remains null, so every call to this method involves a request to the storage.

This way, heavy products are not kept in memory after calling this method, thus maintaining the ProductRef a lighweight reference to the target product.

In case of a Context, if it is wanted to free the reference, call unload().

Returns: the product

getSize(): Returns the estimated size(in bytes) of the product in memory. Useful for providing this information for a user that wants to download the product from a remote site. Returns: the size in bytes

getStorage(): Returns the product storage associated.

getType(): Specifies the Product class to which this Product reference is pointing to.

getUrn(): Returns the Uniform Resource Name (URN) of the product.

getUrnObj(): Returns the URN as an object.

isLoaded(): Informs whether the pointed product is already loaded.

property meta: Property

property parents: property

property product

removeParent(parent): remove a parent

serializable(): Can be encoded with serializableEncoder

setParents(parents): Sets the in-memory parent context products of this reference.

setStorage(storage): Sets the product storage associated.

setUrn(urn)

setUrnObj(urnobj, poolurn=None, meta=None): sets urn Poolurn if given overrides the pool URN in urn, and causes metadata to be loaded from pool. A productref created from a single product will result in a memory pool urn, and the metadata won’t be loaded. If meta is given, it will be used instead of that from poolurn.

toString(matprint=None, trans=True)

unload(): Clear the cached meta and frees internal reference to the product, so it can be garbage collected.

property urn: Property

property urnobj: Property

fdi.pal.productstorage module

class fdi.pal.productstorage.ProductStorage(pool=None, **kwds)

Bases: object

Logical store created from a pool or a poolURN.

Every instanciation with the same pool will result in a new instance of ProdStorage.

accept(visitor): Hook for adding functionality to object through visitor pattern.

getAllTags(): Get all tags defined in the writable pool.

getHead(ref): Returns the latest version of a given product, belonging to the first pool where the same track id is found.

getMeta(urn): Get the metadata belonging to the writable pool that associated to a given URN. returns an ODict.

getPool(poolurn): mh: returns the pool object from poolurn

getPools(): Returns the set of ProductPools registered. mh: in a list of poolurns

getProductClasses(poolurn): Yields all Product classes found in this pool.

getTags(urn): Get the tags belonging to the writable pool that associated to a given URN. returns an iterator.

getUrnFromTag(tag): Get the URN belonging to the writable pool that is associated to a given tag.

getWritablePool(): returns the poolurn of the first pool, which is the only writeable pool.

load(urnortag): Loads a product with a URN or a list of products with a tag, from the (writeable) pool. It always creates new ProductRefs. returns productref(s). urnortag: urn or tag

register(pool): Registers the given pools to the storage.

remove(urn): removes product of urn from the writeable pool

save(product, tag=None, poolurn=None, geturnobjs=False): saves to the writable pool if it has been registered. if not, registers and saves. product can be one or a list of prpoducts. Returns: one or a list of productref with storage info. mh: or UrnObjs if geturnobjs is True.

select(query, previous=None)

Returns a list of URNs to products that match the specified query.

Parameters: query - the query object previous - results to be refined Returns: the set of return eferences to products matching the supplied query.

wipePool(poolurn)

fdi.pal.query module

class fdi.pal.query.AbstractQuery(product=<class 'fdi.dataset.product.Product'>, variable='p', where='', allVersions=False, **kwds)

Bases: StorageQuery

provides default implementations for the pal storage query.

hashCode()

toString()

class fdi.pal.query.MetaQuery(product=<class 'fdi.dataset.product.Product'>, where='', allVersions=False, **kwds)

Bases: AbstractQuery

Meta data query formulates a query on the meta data of a Product.

Typically this type of query is faster than a full query on the Product Access Layer.

class fdi.pal.query.StorageQuery(**kwds)

Bases: Serializable

Query on a ProductStorage.

accept(visitor): Hook for adding functionality to object through visitor pattern.

getType(): Get the class used in the query.

getVariable(): Get the variable name used in the query expression, eg “p”.

getWhere(): Get the query expression to be evaluated.

retrieveAllVersions(): Are all versions to be retrieved, or just the latest?

fdi.pal.runpnsserver module

fdi.pal.taggable module

class fdi.pal.taggable.Taggable(**kwds)

Bases: object

Definition of services provided by a product storage supporting versioning.

getTagUrnMap(): Get the full tag->urn mappings. mh: returns an iterator

getTags(urn=None): Get all of the tags that map to a given URN. Get all known tags if urn is not specified. mh: returns an iterator.

getUrn(tag): Gets the URNs corresponding to the given tag.

getUrnObject(tag): Gets the URNobjects corresponding to the given tag.

removeTag(tag): Remove the given tag from the tag and urn maps.

removeUrn(urn): Remove the given urn from the tag and urn maps.

removekey(key, themap, thename, othermap, othername): Remove the given key.

setTag(tag, urn): Sets the specified tag to the given URN.

tagExists(tag): Tests if a tag exists.

fdi.pal.urn module

class fdi.pal.urn.Urn(urn=None, pool=None, cls=None, index=None, **kwds)

Bases: DeepEqual, Serializable, Comparable

The object representation of the product URN string. The memory consumed by sets of this object are much less than sets of URN strings.

Only when the class types in URN string are not in classpath, the urn object will consume equals or a little more than URN string as the object has to hold the original urn string. However this should be considered as exceptional cases.

Using this object representation also help to avoid parsing cost of URN string. mh: URN format:

urn:poolname:resourceclass:serialnumber

where

Resourceclass

(fully qualified) class name of the resource (product)

Poolname

scheme + :// + place + directory

Scheme

file, mem, http … etc

Place

192.168.5.6:8080, c:, an empty string … etc

Directory

A label for the pool that is by default used as the full path where the pool is stored. ProductPool.transformpath() can used to change the directory here to other meaning.

for file scheme: / + name + / + name + … + / + name
for mem scheme: / + name

Serialnumber

internal index. str(int).

URN is immutable.

a__eq__(o): mh: compare urn string after the first 4 letters.

static getFullPath(urn): returns the place+poolname+resource directory of the urn

getIndex(): Returns the product index.

getPlace(): Returns the netloc in this

getPool(): Returns the pool name in this

getPoolId(): Returns the pool URN in this

getScheme(): Returns the urn scheme.

getType(): Returns class type of Urn

getTypeName(): Returns class type name of Urn.

getUrn(): Returns the urn in this

getUrnWithoutPoolId()

hasData(): Returns whether this data wrapper has data.

property place

property pool: returns the pool URN.

serializable(): Can be encoded with serializableEncoder

setUrn(urn): parse urn to get scheme, place, pool, resource, index.

toString()

property urn: property

fdi.pal.urn.makeUrn(poolname, typename, index): assembles a URN with infos of the pool, the resource type, and the index

fdi.pal.urn.parseUrn(urn): Checks the URN string is valid in its form and splits it. Pool URN is in the form of a URL that does not have ‘:’ in its path part. Product URNs are more complicated. For example if the urn is urn:file://c:/tmp/mypool/proj1.product:322 into poolname file://c:/tmp/mypool, resource type (usually class) name proj1.product, serial number in string '322', scheme file, place c: (with ip and port if given), and poolpath c:/tmp/mypool Poolname is also called poolURN or poolID.

fdi.pal.versionable module

class fdi.pal.versionable.Versionable(**kwds)

Bases: object

for items being able to be defined with a Definition.

getLastVersion(ref): Returns the latest version of the given ProductRef.

getVersions(ref): Returns all the versions of the given ProductRef.

saveProductRef(ref): Saves the product referenced and returns the designated URN.

fdi.pal package

Subpackages

Submodules

fdi.pal.common module

fdi.pal.comparable module

fdi.pal.context module

fdi.pal.definable module

fdi.pal.httpclientpool module

fdi.pal.localpool module

fdi.pal.mempool module

fdi.pal.pnspoolserver module

fdi.pal.poolmanager module

fdi.pal.productpool module

fdi.pal.productref module

fdi.pal.productstorage module

fdi.pal.query module

fdi.pal.runpnsserver module

fdi.pal.taggable module

fdi.pal.urn module

fdi.pal.versionable module

fdi

Navigation

Related Topics