fdi.utils package

Submodules

fdi.utils.checkjson module

fdi.utils.checkjson.checkjson(obj, dbg=0, int_key=True, **kwds)

seriaizes the given object and deserialize. check equality.

fdi.utils.checkjson.getyaml()

fdi.utils.common module

exception fdi.utils.common.UserOrGroupNotFoundError

Bases: BaseException

fdi.utils.common.attrstr(p, v, missingval='', ftime=False, state=True, width=1, **kwds)

generic string representation of an attribute of a parameter or dataset.

p: parameter object. v: name of parameter attribute. ‘_valid’, ‘_type’, ‘_default’, ‘_value’ (for Parameter) or ‘_data’ (dataset) missingval: string used when the parameter does not have the attribute. ftime: True means that attribute value will be FineTime if _type is ‘finetime’. state: The state validity of the parameter is returned in place of value, if the state is not in Ommitted_Valid_Rule_Names – ‘valid’, ‘range’, ‘’ or ‘default’.

fdi.utils.common.attrstr1(p, v, missingval='', ftime=False, state=True, width=1, **kwds)

generic string representation of an attribute of a parameter or dataset.

p: parameter object. v: name of parameter attribute. ‘_valid’, ‘_type’, ‘_default’, ‘_value’ (for Parameter) or ‘_data’ (dataset) missingval: string used when the parameter does not have the attribute. ftime: True means that attribute value will be FineTime if _type is ‘finetime’. state: The state validity of the parameter is returned in place of value, if the state is not in Ommitted_Valid_Rule_Names – ‘valid’, ‘range’, ‘’ or ‘default’.

fdi.utils.common.binhexstring(val, typ_, width=0, v=None, p=None, level=0, **kwds)

returns val in binary, hex, or string according to typ_.

val; list of validity descriptor entries. typ_: parameter type in DataTypes.

fdi.utils.common.bstr(x, length=0, tostr=True, quote="'", yaml=False, html=False, **kwds)

returns the best string representation.

if the object is a string, return single-quoted; if has toString(), use it; else returns str(). Length limited by lls(lls)

fdi.utils.common.exprstrs(param, v='_value', extra=False, **kwds)

Generates a set of strings for param.toString().

Param:

Parameter or xDstaset.

Extra:

Whether to include less often used attributes such as `fits_keyword`.

fdi.utils.common.findShape(data, element_seq=<class 'str'>)

Shape of list/dict of list/dict.

Element_seq:

treat elements of these sequence types as scalars.

fdi.utils.common.find_all_files(datadir, verbose=False, include=None, exclude=None, not_if=None, absdir=False)

returns a list of names of all files in datadir.

Name:

of starting directory or a list of file name strings to filter.

Include:

only if a file name has any of these sub-strings. format is as if used in glob(include).

Exclude:

only if a file name has not any of these sub-strings. Empty strings are removed.

Not_if:

a function that returns true if given a name of unwanted file. default is None, (which excludes directories when datadir is a string, and disabled if datadir is a list.

Absdir:

Set to True to return absolute_paths.

fdi.utils.common.fullname(obj)

full class name with module name.

https://stackoverflow.com/a/2020083/13472124

fdi.utils.common.getObjectbyId(idn, lgbv)

lgb is from deserializing caller’s globals().values() locals().values() and built-ins

fdi.utils.common.getUidGid(username)

returns the UID and GID of the named user.

return: -1 if not available

fdi.utils.common.grouper(iterable, n, fillvalue=None)

Collect data into fixed-length chunks or blocks

fdi.utils.common.l2t(v)

convert lists to tuples in nested data structures

fdi.utils.common.ld2tk(v)

convert lists, to tuples and dicts to frozensets in nested data structures array.array is converted to (typecode, itemsize, size, ld2tk(0th element))

fdi.utils.common.lls(s, length=80)

length-limited string.

Returns the str if len <= length or length <=3. Returns ‘begin…end’ if not.

fdi.utils.common.mstr(obj, level=0, excpt=None, indent=4, depth=0, **kwds)

Makes a presentation string at a detail level.

‘tablefmt’ is needed to be passed in recursive calls under some conditions it is used.

fdi.utils.common.pathjoin(*p)

join path segments with given separater (default ‘/’). Useful when ‘' is needed.

fdi.utils.common.str2md5(string)

fdi.utils.common.t2l(v)

convert tuples to lists in nested data structures

fdi.utils.common.trbk(e)

trace back

fdi.utils.common.trbk2(e)

fdi.utils.common.wls(st, width=15, fill=None, linebreak='\n', unprintable='#')

generates a string comtaining width-limited strings separated with ‘ ‘.

Identifies Line-breaks with str.splitlines https://docs.python.org/3.6/library/stdtypes.html#str.splitlines Removes trailing line-breaks.

st:

input string. If not a string, `str(st)` is used.

width:

if > 0 returns the str with linebreak inserted every width chars max. Default width is 15. A CJK characters occupies 2 in widths.

linebreak:

line-break character. default `

`

unprintable:

substitute unprintable characters with this, only active if wide or unprintable characters are found. default is ‘#’.

fdi.utils.fetch module

fdi.utils.fetch.fetch(paths, nested, re='', sep='/', exe=['is'], not_quoted=True)

Use paths to access values of internal elements of a nested python object.

Paths:

1). If given as a string, the string will be splitted with sep into a list of strings, then go on to 2); 2). If given as a list of strings, its 0th member is to match one of the first level of nested attributes, keys, or method names. If the list was made in 1) the 0th member will be converted to an integer if possible.

• if the 0th member is a string and can be parsed by deserialize_args(), the result to used as te named method and its arguments.

• if that fails, it will be taken as a string and check if there is a match in keys (members);

• else search in attributes.

Nested:

a live nested data structure.

Re:

datapath representation for nested. Can be applied to reproduce the result.

Exe:

1. A list of patterns which if found in the name of a method/function the matching method/function is allowed to run. 2) If one of the pattern is ‘*’, all methods/functions are allowed to run. 3) If a pattern starts with a ‘-’ then the matching method/function to the pattern (‘-’ removed) is not allowed to run (overriding previous rules.

Not_quoted:

the method-args string is not encoded with quote.

Result A found object and a string of python code e.g. ‘.meta[“speed”].isValid()’

fdi.utils.fits module

fdi.utils.fits.add_header(meta, header)

fdi.utils.fits.fits_dataset(hdul, ima_list)

fdi.utils.fits.fits_header()

fdi.utils.fits.main()

fdi.utils.fits.test_fits_kw(h)

fdi.utils.fits_kw module

fdi.utils.fits_kw.getFitsKw(name, ndigits=-1, extra=None)

Returns the FITS keyword for a name.

If name ends with a digit, split name to a digital part consists if all digits on the right, and a “non-digital part” on the left. Take ndigits continuoud digits, counting from right, to form the “numeric-part”. The “non-digital part”, or the name if not endibg with digits, get the pre-translation according to:

1. Look up in the Param_Names table (inverse FITS_KEYWORDS table, if fails,

2. try the extra dictionary if provided. if fails,

3. take key value

If name ends with a digit, append the “numeric-part”, 0-padded or truncated to ndigits, to the first 8 - ndigits (maximum) of characters from pre-transition, uppercased, to form the result; Iff ndigits is -1, it is deactivated.

else take a maximum of 8 characters from pre-transition. uppercased, to form the resukt.

name:

the name of e.g. a parameter.

ndigits:

how many digits (right to left) to take maximum if name ends with digits. default -1 (deactivated). Raises ValueError if more than 7.

extra:

tuple of (fits_kw, parameter_name) tuples to provide more look-up dictionary that override values from the default one.

returns:

FITS keyword.

fdi.utils.getconfig module

fdi.utils.getconfig.Config_Look_Up = {'pns': Lazy_Loading_ChainMap({'scheme': 'http', 'host': '111.111.111.111', 'port': 2222, 'baseurl': '/fdi/v0.15', 'base_local_poolpath': '/tmp/httppool', 'server_local_poolpath': '/tmp/httppool/data', 'cloud_api_version': 'v1'}, {'server_scheme': 'server', 'logger_level': 20, 'logger_level_extras': 30, 'api_version': 'v0.15', 'defaultpool': 'default', 'requests_timeout': (3.3, 909), 'username': 'foo', 'password': 'bar', 'self_host': '0.0.0.0', 'self_port': 9876, 'self_username': 'fdi', 'self_password': 'ONLY_IF_NEEDED', 'rw_user': 'foo', 'rw_pass': 'pbkdf2:sha256:260000$V1hXW8OVUKekaSHP$85b21f4fb0a3c6f0eef73165538d7aab7881ce8acc48c4af59fd33edd8bf13f2', 'ro_user': 'bar', 'ro_pass': 'pbkdf2:sha256:260000$8vrAxZeeJJhTrZLQ$70fd3819d62bb46fe89fc1cd933fb8052e83da75d66624b6146f105288be0bfd', 'userclasses': '', 'docker_version': '', 'server_version': '', 'cloud_token': '/tmp/.cloud_token', 'cloud_username': 'mh', 'cloud_password': '', 'cloud_host': '', 'cloud_port': 31702, 'cloud_scheme': 'csdb', 'cloud_api_base': '/csdb', 'mq_host': '172.17.0.1', 'mq_port': 9876, 'mq_user': '', 'mq_pass': '', 'pipeline_host': '172.17.0.1', 'pipeline_port': 9876, 'pipeline_user': '', 'pipeline_pass': '', 'oss_access_key_id': None, 'oss_access_key_secret': None, 'oss_bucket_name': None, 'oss_endpoint': None, 'oss_prefix': None})}

`Lazy_Loading_ChainMap`s for getting value with a key from Configuration dictionaries in files and in OS Environment..

fdi.utils.getconfig.Config_NameSpace = {'pns': <class 'fdi.utils.getconfig.getConfClass.<locals>.Conf'>}

NameSpace managers for configuration key-value pairs.

fdi.utils.getconfig.XXXcget(name, conf_name='pns', builtin=None, force=False)

Find value for configured variables.

Look up in various places and check if poolurl is needed. For order of places to look-up see check_env().

Parameters:

• name (str) – variable name. If starting with ‘poolurl:’ the rest of the name is used as the poolname part of a default-spec poolurl.

• conf_name (str) – name of configuration. Default ‘pns’.

• builtin (dict) – a built-in dict given as the basis.

• force (bool) – updating source maps, no matter what other parameters are.

Returns:

value of the variable.

Return type:

str

fdi.utils.getconfig.XXXcheck_env(prefixed_config, config, conf_name, osenviron, name)

look up variable name in Environment or configure file.

Parameters:

• prefixed_config (dict) – configuration map of variables with prefix in Environment.

• config (dict) – configuration map of variables without prefix in Environment.

• conf_name (str) – The prefix. e.g. ‘pns’.

• osenviron (dict) – Environment variables.

• name (str) – Variable name.

Returns:

• str – Variable value.

• Exception

• ———

• LookupError – Variable not found.

fdi.utils.getconfig.cget(name, conf_name='pns', builtin=None, force=False)

Find value for configured variables.

Look up in various places and check if poolurl is needed. The look-up order is:

1. with prefix in the Environment, e.g. PNS_HOST.

2. in prefixed_config, e.g. prefixed_config[‘host’].

3. without prefix in the Environment, e.g. HOST.

4. in config, e.g. config[‘host’].

Parameters:

• name (str) – variable name. If starting with ‘poolurl:’ the rest of the name is used as the poolname part of a default-spec poolurl.

• conf_name (str) – name of configuration. Default ‘pns’.

• builtin (dict) – a built-in dict given as the basis.

• force (bool) – Re-make the Config_NameSpace entry named conf_name, no matter what other parameters are.

Returns:

value of the variable.

Return type:

str

fdi.utils.getconfig.getConfClass(conf_name, builtin)

fdi.utils.getconfig.getConfig(name=None, conf='pns', builtin={'api_version': 'v0.15', 'base_local_poolpath': '/tmp/httppool', 'baseurl': '/fdi/v0.15', 'cloud_api_base': '/csdb', 'cloud_api_version': 'v1', 'cloud_host': '', 'cloud_password': '', 'cloud_port': 31702, 'cloud_scheme': 'csdb', 'cloud_token': '/tmp/.cloud_token', 'cloud_username': 'mh', 'defaultpool': 'default', 'docker_version': '', 'host': '111.111.111.111', 'logger_level': 20, 'logger_level_extras': 30, 'mq_host': '172.17.0.1', 'mq_pass': '', 'mq_port': 9876, 'mq_user': '', 'oss_access_key_id': None, 'oss_access_key_secret': None, 'oss_bucket_name': None, 'oss_endpoint': None, 'oss_prefix': None, 'password': 'bar', 'pipeline_host': '172.17.0.1', 'pipeline_pass': '', 'pipeline_port': 9876, 'pipeline_user': '', 'port': 2222, 'requests_timeout': (3.3, 909), 'ro_pass': 'pbkdf2:sha256:260000$8vrAxZeeJJhTrZLQ$70fd3819d62bb46fe89fc1cd933fb8052e83da75d66624b6146f105288be0bfd', 'ro_user': 'bar', 'rw_pass': 'pbkdf2:sha256:260000$V1hXW8OVUKekaSHP$85b21f4fb0a3c6f0eef73165538d7aab7881ce8acc48c4af59fd33edd8bf13f2', 'rw_user': 'foo', 'scheme': 'http', 'self_host': '0.0.0.0', 'self_password': 'ONLY_IF_NEEDED', 'self_port': 9876, 'self_username': 'fdi', 'server_local_poolpath': '/tmp/httppool/data', 'server_scheme': 'server', 'server_version': '', 'userclasses': '', 'username': 'foo'}, force=False)

Imports a dict named [conf]config.

The contents of configuration are the key-value pairs of a dict variable fdi.pns.config::<conf>config by default.

The configuration is updated by contents of a configuration file in the same format as fdi.pns.config:pnsconfig. Name of the configuration file is in the form of <conf>local.py where <conf> is the value of the conf parameter of this function, ‘pns’ by default.

The config file directory is the process owner’s ~/.config/ by default. It can be modified by the environment variable <uppercased conf>_CONF_DIR, e.g. PNS_CONF_DIR..

An exisiting configuration value can be overridden by that of an environment variable. The env var is named <uppercased <name> for name variables in the config dict; but named <uppercased <conf>_<name> for name variables in the pre-fixed dict. For example configuration of host in pnsconfig dict is overridden by the value of envirionment variable PNS_HOST. kc_type in config dict is overridden by KC_TYPE.

Parameters:

• name (str) – Identifier of the configured item whose value will be returned. If started with `poolurl:`, construct a poolurl with `scheme` and `node` with `/{name}` at the end. Default `None`, a mapping of all configured items corrected with envirionment variables is returned. Prefixed names in ENV but not in config files are allowed.

• conf (str) – File <conf>local.py` defines configuration key-value pairs in dict named <conf>config. Default ‘pns’, so the file is ‘pnslocal.py’, and the variable is `pnsconfig.

• builtin (dict. To be updated by <conf>local. default is fdi.pns.config.)

• force (bool) – Always ‘False’ reload from file instead of cache for all `conf`s cached.

Returns:

configured value.

Return type:

obj

fdi.utils.getconfig.getMappings(conf_name, builtin)

return dictionaries of variable name in Environment or configure file.

The look-up order is:

1. with prefix in the Environment, e.g. PNS_HOST.

2. in prefixed_config, e.g. prefixed_config[‘host’].

3. without prefix in the Environment, e.g. HOST.

4. in config, e.g. config[‘host’].

Parameters:

• prefixed_config (dict) – configuration map of variables with prefix in Environment.

• config (dict) – configuration map of variables without prefix in Environment.

• conf_name (str) – The prefix. e.g. ‘pns’.

• osenviron (dict) – Environment variables.

• name (str) – Variable name.

Returns:

• str – Variable value.

• Exception

• ———

• LookupError – Variable not found.

fdi.utils.getconfig.get_file_conf(conf_name)

figure iut config file name and returns the contents.

Conf_name:

str, ‘pns’ etc.

Returns: dict for pre-fixed variables and for not prefixed variables.

fdi.utils.getconfig.get_mqtt_config()

Get configured MQTT info from project configuration file.

Overrideable by uppercased environment variables. Note that there is a ‘PNS_’ in the beginning environment variable name, e.g. `PNS_MQ_HOST` for `pc['mq_host']` ref fdi.utils.getConfig and your local `~/.config/pnslocal.py`

fdi.utils.getconfig.loader(key, mapping, remove=True, exclude=None, ignore_error=False)

fdi.utils.getconfig.make_pool(pool, conf='pns', auth=None, wipe=False)

Return a ProductStorage with given pool name or poolURL.

Name:

PoolURL, or pool name (has no “://”), in which case a pool URL is made based on the result of getConfig(name=pool, conf=conf). Default is ‘’.

Auth:

if is None will be set to HTTPBasicAuth using the config.

Conf:

passed to getconfig to determine which configuration. Default `pns`.

Wipe:

whether to delete everything in the pool first.

Exception ConnectionError

fdi.utils.images module

fdi.utils.images.generate_png(buf, width, height, greyscale=True, bitdepth=8, compression=9)

generate a png file.

from https://stackoverflow.com/a/19174800/13472124 data: bigendian array of sequences

fdi.utils.images.longrainbowl(n=8)

Short rainbowl color map modified to have 5 times colors.

ref. https://www.particleincell.com/2014/colormap/ return a dictionary that translates 0 - 2**n-1 to (R, G, B) where R, G, B are color from 0 - 255.

fdi.utils.images.shortrainbowl(n=8)

Short rainbowl color map modified to have 4 times colors.

ref. https://www.particleincell.com/2014/colormap/ return a dictionary that translates 0 - 2**n-1 to (R, G, B) where R, G, B are color from 0 - 255.

fdi.utils.images.toPng(adset, grey=False, compression=0, cspace=8, cmap=None, verbose=False)

Make a PNG an image from an ArrayDataset.

adset: ArrayDataset whose data is a Sequence of Sequence of 2byte data. grey: Grey scale for pixel values clipped to [median-3stdev, median+3stdev] then mapped to full grey range, if True (default) else RGB color of sorted unique pixel values scaled to full color space.. compression: 0-9 for how much to compress. default to 0 for no compression cspace; 1-16 for bits per channel. 16 for grey 8 for color. cmap: an ordered dictionary that gives (R,G,B) for a pixel value. default is longrainbowl.

fdi.utils.json_to_yaml module

fdi.utils.jsonpath module

fdi.utils.jsonpath.flatten_compact(roots, num=False, style='short', sep='.', path_list=False)

Colapse datapaths of elements of an object.

Duplicated paths will only have the last one. For example:

for n, c in {‘group1.val’: a10[‘col1’], ‘group1.err’: a10[‘col2’],

‘no-group.val’: a10[‘col1’], ‘group2.val’: a10[‘col1’], ‘group2.err’: a10[‘col2’], ‘group2.seq’: a10[‘col2’], ‘group2.wgt’: Column(data=[‘’, 0.32, -9876543210], unit=’g’), }.items():

v.addColumn(n, c)

will make one-level datapath:

col1
col2
group1.val
group1.err
no-group.val
group2.val
group2.err
group2.seq
group2.wgt

shown in a tree:

|__ 'col1'                                     <Column> (3,)
|__ 'col2'                                     <Column> (3,)
|__ 'group1.val'                               <Column> (3,)
|__ 'group1.err'                               <Column> (3,)
|__ 'no-group.val'                             <Column> (3,)
|__ 'group2.val'                               <Column> (3,)
|__ 'group2.err'                               <Column> (3,)
|__ 'group2.seq'                               <Column> (3,)
\__ 'group2.wgt'                               <Column> (3,)

Flatten_compact `print(' 'self.join(flatten_compact([v]).keys()))`:

col1
col2
g.val
g.err
n.val
g.seq
g.wgt

The ‘group’ part look good in a table header:

+-----------------+-----------------+--------+----------------------------------------+
|                 |     group1      |  no-   |                 group2                 |
|                 |                 | group  |                                        |
+--------+--------+--------+--------+--------+--------+--------+--------+-------------+
|   col1 |   col2 |    val |    err |    val |    val |    err |    seq | wgt         |
|   (eV) |  (cnt) |   (eV) |  (cnt) |   (eV) |   (eV) |  (cnt) |  (cnt) | (g)         |
+========+========+========+========+========+========+========+========+=============+
|    1   |    0   |    1   |    0   |    1   |    1   |    0   |    0   |             |
+--------+--------+--------+--------+--------+--------+--------+--------+-------------+
|    4.4 |   43.2 |    4.4 |   43.2 |    4.4 |    4.4 |   43.2 |   43.2 | 0.32        |
+--------+--------+--------+--------+--------+--------+--------+--------+-------------+
| 5400   | 2000   | 5400   | 2000   | 5400   | 5400   | 2000   | 2000   | -9876543210 |
+--------+--------+--------+--------+--------+--------+--------+--------+-------------+


:roots: where. A list of maps.
:num: preceed keys with sequence numbers.
:style: for keys, `short`: use shortened path e.g. ```abc.def.hgi``` ```a.d.hgi```. `last2`: use the right-most 2 segments, `full` for untrated paths, anything else to use only the last one.
:sep: separater used in output. Default is '.'.
:path_list: a list of path segments in place of value. Defalut `False`.
:return: dict(flatten_compact_path:(list of path|val)

fdi.utils.jsonpath.getCacheInfo()

fdi.utils.jsonpath.jexp(expr, *args, **kwds)

fdi.utils.jsonpath.jsonPath(data, expr, val='simple', sep='/', indent=None, *args, **kwds)

Make a JSONPath query on the data.

Expr:

JSONPath expression. Ref ‘jsonpath_ng’

Sep:

‘’ or None for keeping jsonpath_ng format (e.g. a.b.[3].d; other string for substituting ‘.’ to the given string, with ‘[’ and ‘]’ removed. Default is ‘/’.

Val:

‘context’ for returning the list of DatumInContext of find; ‘simple’ (default) for list of simple types of values and summarizing list and dict values; other for a list of un-treated `DatumInContext.value`s; ‘paths’ for a list of paths only.

Indent:

for json.dumps.

Returns:

• If val is `context`, return the list of DatumInContext of jsonpath_ng.ext.parse().find().

• Else return a list of full_path-value pairs from the output of find().

• * If val is `simple`, only node values of simple types are kept, list and dict types will show as ‘<list> length’ and ‘<dict> [keys [… [length]]]’, respectively.

• * If val is `full`, the values of returned `list`s are un-treated `DatumInContext.value`s.

fdi.utils.leapseconds module

Get TAI-UTC difference in seconds for a given time using tzdata.

i.e., find the number of seconds that must be added to UTC to compute TAI for any timestamp at or after the given time[1].

>>> from datetime import datetime
>>> import leapseconds
>>> leapseconds.dTAI_UTC_from_utc(datetime(2005, 1, 1, tzinfo = timezone.utc))
datetime.timedelta(0, 32)
>>> leapseconds.utc_to_tai(datetime(2015, 7, 1, tzinfo = timezone.utc))
datetime.datetime(2015, 7, 1, 0, 0, 36, tzinfo=datetime.timezone.utc)
>>> leapseconds.tai_to_utc(datetime(2015, 7, 1, 0, 0, 36, tzinfo = timezone.utc))
datetime.datetime(2015, 7, 1, 0, 0, tzinfo=datetime.timezone.utc)
>>> leapseconds.tai_to_utc(datetime(2015, 7, 1, 0, 0, 35, tzinfo = timezone.utc)) # leap second
datetime.datetime(2015, 7, 1, 0, 0, tzinfo=datetime.timezone.utc)
>>> leapseconds.tai_to_utc(datetime(2015, 7, 1, 0, 0, 34, tzinfo = timezone.utc))
datetime.datetime(2015, 6, 30, 23, 59, 59, tzinfo=datetime.timezone.utc)

Python 2.6+, Python 3, Jython, Pypy support.

[1]: https://github.com/eggert/tz/blob/master/leap-seconds.list [2]: https://github.com/eggert/tz/blob/master/tzfile.h [3]: https://github.com/eggert/tz/blob/master/zic.c [4]: http://datacenter.iers.org/eop/-/somos/5Rgv/latest/16

class fdi.utils.leapseconds.LeapSecond(utc, dTAI_UTC)

Bases: tuple

Create new instance of LeapSecond(utc, dTAI_UTC)

dTAI_UTC

Alias for field number 1

utc

Alias for field number 0

fdi.utils.leapseconds.dTAI_UTC_from_tai(tai_time)

UTC time = tai_time - dTAI_UTC_from_tai(tai_time).

fdi.utils.leapseconds.dTAI_UTC_from_utc(utc_time)

TAI time = utc_time + dTAI_UTC_from_utc(utc_time).

fdi.utils.leapseconds.gps_to_tai(gps_time)

Convert GPS time given as datetime object to TAI time.

fdi.utils.leapseconds.gps_to_utc(gps_time)

Convert GPS time given as datetime object to UTC time.

fdi.utils.leapseconds.leapseconds(tzfiles=['/usr/share/zoneinfo/right/UTC', '/usr/lib/zoneinfo/right/UTC'], use_fallback=True)

Extract leap seconds from tzfiles.

fdi.utils.leapseconds.tai_to_gps(tai_time)

Convert TAI time given as datetime object to GPS time.

fdi.utils.leapseconds.tai_to_utc(tai_time)

Convert TAI time given as datetime object to UTC time.

fdi.utils.leapseconds.utc_to_gps(utc_time)

Convert UTC time given as datetime object to GPS time.

fdi.utils.leapseconds.utc_to_tai(utc_time)

Convert UTC time given as datetime object to TAI time.

fdi.utils.loadfiles module

fdi.utils.loadfiles.loadMedia(filename, content_type='image/png')

fdi.utils.loadfiles.loadcsv(filepath, delimiter=',', header=0, return_dict=False, pad=None)

Loads the contents of a CSV file into a list of tuples.

Header:

the first header linea are taken as column headers if `header > 0`. If no column header given (default 0), `colN` where N = 1, 2, 3… are returned.

the second header linea are also recorded (usually units) if header ` > 1. :return_dict: if ```True`` returns `dict[colhd]=(col, unit). Default is ```False`.

Returns:

Default is a list of (colhd, column, unit) tuplees.

fdi.utils.masked module

fdi.utils.masked.masked(value, mask)

Returns the masked part of the value, high bit number of mask, how wide the mask is..

e.g. value=0b00101100 mask=0b00011100, the result is 0b011, 5, 3

fdi.utils.moduleloader module

https://stackoverflow.com/a/43573798/13472124

class fdi.utils.moduleloader.MyLoader(filename)

Bases: SourceFileLoader

Cache the module name and the path to the file found by the finder.

create_module(spec)

Use default semantics for module creation.

exec_module(module)

Execute the module.

get_data(path)

Return the data from path as raw bytes.

get_filename(fullname)

Return the path to the source file as found by the finder.

class fdi.utils.moduleloader.MyMetaFinder

Bases: MetaPathFinder

find_spec(fullname, path, target=None)

class fdi.utils.moduleloader.SelectiveMetaFinder

Bases: MetaPathFinder

Raise ExcludedModule exception if loading modules whose names have any of cls.exclude.

exception ExcludedModule

Bases: Exception

exclude = []

find_spec(fullname, path, target=None)

fdi.utils.moduleloader.install()

Inserts the finder into the import machinery

fdi.utils.moduleloader.installSelectiveMetaFinder()

Inserts the finder into the import machinery

fdi.utils.moduleloader.main(ipath='.')

fdi.utils.monokaimod module

class fdi.utils.monokaimod.MonokaiMod

Bases: MonokaiStyle

styles = {Token: '#f8f8f2', Token.Comment: 'bold italic #af2', Token.Comment.Hashbang: '', Token.Comment.Multiline: '', Token.Comment.Preproc: '', Token.Comment.PreprocFile: '', Token.Comment.Single: '', Token.Comment.Special: '', Token.Error: '#960050 bg:#1e0010', Token.Escape: '', Token.Generic: '', Token.Generic.Deleted: '#f92672', Token.Generic.Emph: 'italic', Token.Generic.Error: '', Token.Generic.Heading: '', Token.Generic.Inserted: '#a6e22e', Token.Generic.Output: 'bg:#022 ansibrightcyan', Token.Generic.Prompt: 'bold #f92672', Token.Generic.Strong: 'bold', Token.Generic.Subheading: '#75715e', Token.Generic.Traceback: '', Token.Keyword: '#66d9ef', Token.Keyword.Constant: '', Token.Keyword.Declaration: '', Token.Keyword.Namespace: '#f92672', Token.Keyword.Pseudo: '', Token.Keyword.Reserved: '', Token.Keyword.Type: '', Token.Literal: '#ae81ff', Token.Literal.Date: '#e6db74', Token.Literal.Number: '#ae81ff', Token.Literal.Number.Bin: '', Token.Literal.Number.Float: '', Token.Literal.Number.Hex: '', Token.Literal.Number.Integer: '', Token.Literal.Number.Integer.Long: '', Token.Literal.Number.Oct: '', Token.Literal.String: '#e6db74', Token.Literal.String.Affix: '', Token.Literal.String.Backtick: '', Token.Literal.String.Char: '', Token.Literal.String.Delimiter: '', Token.Literal.String.Doc: 'bold italic #af2', Token.Literal.String.Double: '', Token.Literal.String.Escape: '#ae81ff', Token.Literal.String.Heredoc: '', Token.Literal.String.Interpol: '', Token.Literal.String.Other: '', Token.Literal.String.Regex: '', Token.Literal.String.Single: '', Token.Literal.String.Symbol: '', Token.Name: '#f8f8f2', Token.Name.Attribute: '#a6e22e', Token.Name.Builtin: '', Token.Name.Builtin.Pseudo: '', Token.Name.Class: '#a6e22e', Token.Name.Constant: '#66d9ef', Token.Name.Decorator: '#a6e22e', Token.Name.Entity: '', Token.Name.Exception: '#a6e22e', Token.Name.Function: '#a6e22e', Token.Name.Function.Magic: '', Token.Name.Label: '', Token.Name.Namespace: '', Token.Name.Other: '#a6e22e', Token.Name.Property: '', Token.Name.Tag: '#f92672', Token.Name.Variable: '', Token.Name.Variable.Class: '', Token.Name.Variable.Global: '', Token.Name.Variable.Instance: '', Token.Name.Variable.Magic: '', Token.Operator: '#f92672', Token.Operator.Word: '', Token.Other: '', Token.Punctuation: '#f8f8f2', Token.Punctuation.Marker: '', Token.Text: 'bg:#022 ansibrightcyan', Token.Text.Whitespace: ''}

Style definitions for individual token types.

fdi.utils.options module

fdi.utils.options.opt(ops, argv=None)

Example: ops = [ {‘long’:’help’, ‘char’:’h’, ‘default’: false, ‘description’:’print help’}, {‘long’:’verbose’, ‘char’:’v’, ‘default’: false, ‘description’:’print info’}, {‘long’:’username=’, ‘char’:’u’, ‘default’: ‘foo’, ‘description’:’non-empty user name/ID’}, {‘long’:’password=’, ‘char’:’p’, ‘default’: ‘bar’, ‘description’:’password’}, {‘long’:’host=’, ‘char’:’i’, ‘default’: ‘0.0.0.0’, ‘description’:’host IP/name’}, {‘long’:’port=’, ‘char’:’o’, ‘default’: 5000, ‘description’:’port number’} ]

Optionly use ‘OPTSTART’ in command line to start options an have ppreceeding ones ignored. Useful when application is invoked with other options, e.g. by pytest

argv: if provided will be used in plsce of sys.argv.

fdi.utils.queueworks module

class fdi.utils.queueworks.queuework2(topics, host=None, port=None, username=None, passwd=None, client_id=None, callback=None, qos=1, userdata=None, clean_session=None)

Bases: object

get_message()

get_size()

init_args(topics, host=None, port=None, username=None, passwd=None, client_id=None, on_msg_callback=None, qos=1, clean_session=None, userdata=None)

init_client(force=False)

init_logger()

keepalive = 60

logger = None

on_connect(client, userdata, flags, rc)

on_disconnect(client, userdata, rc)

on_message(client, userdata, msg)

on_publish(client, userdata, mid)

on_subscribe(client, userdata, mid, granted_qos)

process_mid(mid, where)

if mid is already in mwa, check it; if not put it in

send(topics, text, conn=True)

start_receive(loop='no')

start_send()

stop_receive()

stop_send()

subclient = None

wait_for(client, msgType, period=0.1)

Waiting for subscribe and publish to clear.

http://www.steves-internet-guide.com/subscribing-topics-mqtt-client/ return: 0 if successful.

fdi.utils.run_proc module

fdi.utils.run_proc.kill_processes(pid)

Kills parent and children processess

https://codeigo.com/python/kill-subprocess

fdi.utils.run_proc.run_proc(cmd, as_user, pwdir, timeout, pty=-1)

Execute a shell command and return status.

Parameters:

• cmd (list) – A list of command line strings. e.g. from shelex.split("ls -t")

• as_user (str) – execute as user. overrides env var USER and pwd.getpwnam().

• pwdir (str) – overides what is in pwd.getpwnam().

• timeout (float) – sets popen timeout. If triggers will get the processed killed.

• pty (int) – passed to stdin, stdout, stderr in popen.

• pid_file (str2) – Name of the file that has the cmd process ID. Needed to kill by grand parents.

Returns:

• dict – If try block fails when starting, a dictionary of:

• returncode: the process return code. -9 if killed aftet timeout.

• message: Error message from exception. – If starting successfully, a dictionary of:

• command: The cmd from input.

• returncode: the process returncode. -9 if killed aftet timeout.

• stdout:, stderr: from communicate() of the process.

• message: Announce success and why may have been killed.

fdi.utils.tofits module

fdi.utils.tofits.add_header(meta, header)

Populate header with keyword lines extracted from MetaData.

fdi.utils.tofits.fits_dataset(hdul, dataset_list, name_list=None, level=0)

Fill an HDU list with dataset data.

Hdul:

list of HDUs.

Dataset_list:

list of dataset subclasses.

Name_list:

fdi.utils.tofits.fits_header()

fdi.utils.tofits.main()

fdi.utils.tofits.test_fits_kw(h)

fdi.utils.tofits.toFits(data, file='', **kwds)

convert dataset to FITS.

Data:

a list of Dataset or a BaseProduct (or its subclass).

File:

‘’ for returning fits stream. string for file name. default ‘’.

fdi.utils.totalsize module

https://code.activestate.com/recipes/577504/

fdi.utils.totalsize.getsize_gc(obj)

sum size of object & members.

fdi.utils.totalsize.getsize_rec(obj_0)

Recursively iterate to sum size of object & members.

fdi.utils.totalsize.total_size(o, handlers={}, verbose=False)

Returns the approximate memory footprint an object and all of its contents.

Automatically finds the contents of the following builtin containers and their subclasses: tuple, list, deque, dict, set and frozenset. To search other containers, add handlers to iterate over their contents:

handlers = {SomeContainerClass: iter,

OtherContainerClass: OtherContainerClass.get_elements}

fdi.utils.tree module

fdi.utils.tree.logger = <Logger fdi.utils.tree (DEBUG)>

Aaron Hall https://stackoverflow.com/a/59109706

fdi.utils.tree.tree(data_object, level=0, style='line', prefix='', seen=None)

A recursive generator, given an object object will yield a visual tree structure line by line with each line prefixed by the same characters.

Style:

can be `line` pr `ascii`.

fdi.utils.ydump module

class fdi.utils.ydump.MyRepresenter(default_style: Any = None, default_flow_style: Any = None, dumper: Any = None)

Bases: RoundTripRepresenter

yaml_representers: Dict[Any, Any] = {<class 'NoneType'>: <function RoundTripRepresenter.represent_none>, <class 'str'>: <function SafeRepresenter.represent_str>, <class 'bytes'>: <function SafeRepresenter.represent_binary>, <class 'bool'>: <function SafeRepresenter.represent_bool>, <class 'int'>: <function SafeRepresenter.represent_int>, <class 'float'>: <function SafeRepresenter.represent_float>, <class 'list'>: <function SafeRepresenter.represent_list>, <class 'tuple'>: <function SafeRepresenter.represent_list>, <class 'dict'>: <function SafeRepresenter.represent_dict>, <class 'set'>: <function SafeRepresenter.represent_set>, <class 'ruamel.yaml.compat.ordereddict'>: <function SafeRepresenter.represent_ordereddict>, <class 'collections.OrderedDict'>: <function RoundTripRepresenter.represent_dict>, <class 'datetime.date'>: <function SafeRepresenter.represent_date>, <class 'datetime.datetime'>: <function SafeRepresenter.represent_datetime>, None: <function SafeRepresenter.represent_undefined>, <class 'ruamel.yaml.scalarstring.LiteralScalarString'>: <function RoundTripRepresenter.represent_literal_scalarstring>, <class 'ruamel.yaml.scalarstring.FoldedScalarString'>: <function RoundTripRepresenter.represent_folded_scalarstring>, <class 'ruamel.yaml.scalarstring.SingleQuotedScalarString'>: <function RoundTripRepresenter.represent_single_quoted_scalarstring>, <class 'ruamel.yaml.scalarstring.DoubleQuotedScalarString'>: <function RoundTripRepresenter.represent_double_quoted_scalarstring>, <class 'ruamel.yaml.scalarstring.PlainScalarString'>: <function RoundTripRepresenter.represent_plain_scalarstring>, <class 'ruamel.yaml.scalarint.ScalarInt'>: <function RoundTripRepresenter.represent_scalar_int>, <class 'ruamel.yaml.scalarint.BinaryInt'>: <function RoundTripRepresenter.represent_binary_int>, <class 'ruamel.yaml.scalarint.OctalInt'>: <function RoundTripRepresenter.represent_octal_int>, <class 'ruamel.yaml.scalarint.HexInt'>: <function RoundTripRepresenter.represent_hex_int>, <class 'ruamel.yaml.scalarint.HexCapsInt'>: <function RoundTripRepresenter.represent_hex_caps_int>, <class 'ruamel.yaml.scalarfloat.ScalarFloat'>: <function RoundTripRepresenter.represent_scalar_float>, <class 'ruamel.yaml.scalarbool.ScalarBoolean'>: <function RoundTripRepresenter.represent_scalar_bool>, <class 'ruamel.yaml.comments.CommentedSeq'>: <function RoundTripRepresenter.represent_list>, <class 'ruamel.yaml.comments.CommentedMap'>: <function RoundTripRepresenter.represent_dict>, <class 'ruamel.yaml.comments.CommentedOrderedMap'>: <function SafeRepresenter.represent_ordereddict>, <class 'ruamel.yaml.comments.CommentedSet'>: <function RoundTripRepresenter.represent_set>, <class 'ruamel.yaml.comments.TaggedScalar'>: <function RoundTripRepresenter.represent_tagged_scalar>, <class 'ruamel.yaml.timestamp.TimeStamp'>: <function RoundTripRepresenter.represent_datetime>}

class fdi.utils.ydump.MyYAML(_kw: Any = <object object>, typ: Optional[Text] = None, pure: Any = False, output: Any = None, plug_ins: Any = None)

Bases: YAML

_kw: not used, forces keyword arguments in 2.7 (in 3 you can do (*, safe_load=..) typ: ‘rt’/None -> RoundTripLoader/RoundTripDumper, (default)

‘safe’ -> SafeLoader/SafeDumper, ‘unsafe’ -> normal/unsafe Loader/Dumper ‘base’ -> baseloader

pure: if True only use Python modules input/output: needed to work as context manager plug_ins: a list of plug-in files

dump(data, stream=None, default_flow_style=False, width=80, allow_unicode=True, **kw)

fdi.utils.ydump.ydump(od, stream=None, register=None, **kwds)

YAML dump that outputs OrderedDict like dict.

fdi.utils.ydump.yinit(mapping=4, sequence=4, offset=2, register=None)

Initializes YAML.