gssa package

Subpackages

• gssa.comparator package
  • Submodules
  • gssa.comparator.comparator module
  • gssa.comparator.parse module
  • gssa.comparator.simulation_definition module
  • Module contents
• gssa.families package
  • Submodules
  • gssa.families.docker module
  • gssa.families.elmer_libnuma module
  • gssa.families.elmer_libnuma_legacy module
  • gssa.families.fenics module
  • gssa.families.g module
  • gssa.families.gssf_arguments module
  • gssa.families.mesher_gssf module
  • Module contents

Submodules

gssa.client module

class gssa.client.GoSmartSimulationClientComponent(x, gssa_file, subdirectory, output_files, tmp_transferrer='/tmp', input_files=None, definition_files=None, skip_clean=False, server=None)

Bases: autobahn.asyncio.wamp.ApplicationSession

finalize(guid)

make_call(suffix)

Make a call com.gosmartsimulation.DEST.suffix.

If this object has a default server set, address it, otherwise we call whichever one got the full namespace.

onComplete(*args, **kwargs)

onFail(*args, **kwargs)

onJoin(details)

onLeave(*args, **kwargs)

onStatus(*args, **kwargs)

shutdown()

Start the exiting steps.

gssa.client.wrapped_coroutine(f)

gssa.config module

gssa.config.get(key, default=None)

Get a value from the global configuration.

If the config file is not loaded, it will be loaded when this is called.

Parameters:

• key (str) – A dot-separated hierarchical key to find in the YAML configuration file.
• default (Optional) – Fall-back to return if the key is not found. Defaults to None.

gssa.config.get_api_version()

Retrieve the API version for this server.

gssa.config.get_config_file()

Get the location of the main configuration file.

gssa.config.init_config(config_file=None)

Load the configuration file.

Parameters:config_file (Optional[str]) – Load the YAML configuration from this file. Defaults to [etc_location]/glossia.yml. In a system-wide install, this might be /usr/local/etc/glossia/glossia.yml

gssa.config.init_logger()

Set up logging.

Uses standard Python logging module, although will set up txaio if logging.txaio is True in the configuration.

gssa.database module

class gssa.database.SQLiteSimulationDatabase(database)

Bases: object

active_count()

Check how many simulations are still marked IN_PROGRESS.

addOrUpdate(simulation)

Update the simulation row or add a new one if not already here.

all()

create()

Set up the database.

delete(simulation, soft=True)

Remove a simulation from the database.

Parameters:

• simulation (str) – GUID of the simulation.
• soft (Optional[bool]) – do a soft delete.

getStatusAndValidation(guid)

Return both status and validation.

getValidation(guid)

Return just the validation XML string for a simulation.

markAllOld()

Mark simulations as exited if still appearing to run - usually on server start-up.

retrieve(guid)

Get a simulation by the client’s GUID.

search(guid_start)

setStatus(guid, exit_code, status, percentage, timestamp)

Update the status of a simulation in the database.

updateValidation(guid, validation_xml)

Add the validation XML string to the simulation row.

gssa.definition module

class gssa.definition.GoSmartSimulationDefinition(guid, xml_string, tmpdir, translator, finalized=False, ignore_development=False, update_status_callback=None)

Bases: object

Routines for working with a single specific simulation.

cancel()

Send the cancel request to the model builder (family).

clean()

Clean out the working directory.

create_xml_from_string(xml)

Turn the string XML into an ElementTree object.

Parameters:xml (str) – string-version of GSSA-XML.

finalize()

Trigger the heavy lifting of interpreting the GSSA-XML.

finalized()

gather_diagnostic()

Create a diagnostic archive.

gather_results()

Create a results archive.

get_dir()

Return working directory.

get_exit_status()

get_files()

get_guid()

Return the simulation’s GUID.

get_remote_dir()

Get remote directory location.

This directory indicates where on the client’s system we should be pulling/pushing from/to.

init_percentage_socket_server()

Start up the status server.

logs(only=None)

Send the cancel request to the model builder (family).

push_files(files, transferrer=None)

Send back the results.

set_exit_status(success, message=None)

Set the status to be recorded in the DB.

set_remote_dir(remote_dir)

simulate()

Tell the family to start simulating.

summary()

Provide a handy synopsis of this definition.

update_files(files)

Wraps the file transferrer.

validation()

DEPRECATED: run validation (requires third-party tool).

gssa.docker module

class gssa.docker.OutputHandler(lock, loop=None, **kwargs)

Bases: hachiko.hachiko.AIOEventHandler, watchdog.events.FileSystemEventHandler

on_moved(event)

class gssa.docker.Submitter

Bases: object

Utility to manage communication with dockerlaunch.

Submit the request to run an instance to the dockerlaunch daemon, also handling some of the standard bits and pieces of interface.

add_input(input_file)

Set additional input.

Make a note that this file should be transferred to the Docker volume for the simulation

cancel()

Break the output wait and instruct dockerlaunch to destroy simulation.

copy_output(requested, target)

Retrieve output.

Copy output files from the Docker volume to the simulation ‘working directory’

destroy(wait_for_response=True)

Destroy the simulation/bridge containers.

finalize()

Close dockerlaunch connection.

logs(only=None)

Retrieve logs from the simulation.

notify_output(filename)

Print a message to highlight a new output file from Docker.

output(requested, exists_only=False)

Return the content of the file requested

Parameters:

• requested (str) – file name to request
• exists_only (Optional[bool]) – only check existence, don’t load

Returns:

file content unless exists_only set, then existence True/False.

Return type:

str|bool

reader = None

receive_response(reader)

Handle JSON reply from dockerlaunch and format neatly.

run_script(loop, working_directory, image, files_required=[], magic_script=None)

Run the simulation.

Parameters:

• loop (asyncio.BaseEventLoop) – look to use for asynchronous calls.
• working_directory (str) – location of simulation configuration/definition.
• image (str) – a container image name. This must be known to dockerlaunch or it will refuse to run.
• files_required (Optional[array(str) – files to return as output.
• magic_script (Optional[str]) – DEPRECATED: script to upload to simulation container after all else in place as a trigger (default off).

send_command(writer, command, arguments=None)

Send a command to the dockerlaunch daemon.

set_update_socket(socket_location)

Set the socket that the bridge container should use to communicate.

writer = None

gssa.error module

class gssa.error.Error

Bases: enum.Enum

Go-Smart error families.

These are the canonical errors for client use - wherever possible, the most applicable of these should be returned. They indicate where the fault is likely to lie.

exception gssa.error.ErrorException(ref, message, *args, **kwargs)

Bases: RuntimeError

An exception wrapping Error.

get_error()

class gssa.error.ErrorMessage

Bases: dict

A trivial subclass of dict marking an ErrorMessage.

gssa.error.makeError(ref, message)

A full error message (as returned via WAMP).

Parameters:

• ref (gssa.error.Error|str) – a enum (or enum name) to categorize the error.
• message (str) – free-form error information string.

Returns:

a dictionary with entries id (int), code (str) and message. The former two are representations of the Error enum, the integer (ordinal) value and string name respectively.

Return type:

ErrorMessage

gssa.family module

class gssa.family.Family

Bases: object

Essential routines for Families.

cancel()

family_name = None

get_needle_parameter(needle_index, key, try_json=True)

Retrieve a parameter for a given needle.

Needle index can be either needle index (as given in XML input) or an integer n indicating the nth needle in the order of the needles XML block.

get_parameter(key, try_json=True, parameters=None)

Retrieve a parameter from the global (not needle) list.

load_core_definition(xml, parameters, algorithms)

Certain bits of the numerical model should always be present.

logs(only=None)

validation(working_directory=None)

class gssa.family.FamilyType(clsname, bases, dct)

Bases: type

Metaclass for families.

This metaclass ensures that any new Family class gets registered for access from GSSA-XML.

gssa.http_transferrer module

class gssa.http_transferrer.HTTPTransferrer

Bases: object

configure_from_xml(xml)

The XML should indicate the source/destination URL.

connect()

disconnect()

downloadFile(sourceUrlStr, destinationStr)

Downloads a file from the source URL to the destination (typically a folder).

This just grabs using urllib GET.

Parameters:

• sourceUrlStr – Url of the file which will be downloaded
• destinationStr – FullPath to the destination

pull_files(files, root, remote_root)

Uses downloadFile to pull.

push_files(files, root, remote_root)

Push using HTTP (unless we are told to use tmp).

uploadFile(sourcePath, destinationUrl)

Uploads the file located in sourcePath to the destinationUrl.

This uploads with a POST.

Parameters:

• sourcePath – fullpath to the file which will be uploaded
• destinationUrl – Url where the file is send to

gssa.parameters module

gssa.parameters.convert_parameter(parameter, typ=None, try_json=True)

Turn a parameter value into a Python object.

Normally a string, but casting is intended to be idempotent.

gssa.parameters.read_parameters(element)

Turn an XML node containing parameter definitions into a dictionary of parameters.

gssa.server module

class gssa.server.GoSmartSimulationServerComponent(server_id, database, publish_cb, ignore_development=False, use_observant=True, simdata_path='/tmp')

Bases: object

This subclasses ApplicationSession, which runs inside an Autobahn WAMP session

client = None

current = None

doApi()

com.gosmartsimulation.api

Find the current API version in use. The API version only needs to be bumped when backward-incompatible changes occur on either side

doCancel(guid)

com.gosmartsimulation.cancel

Prematurely stop the running simulation.

doClean(guid)

com.gosmartsimulation.clean

Remove anything in simulation working directory, for instance

doCompare(this_xml, that_xml)

com.gosmartsimulation.compare

Check whether two GSSA-XML files match and, if not, what their differences are.

doFinalize(guid, client_directory_prefix)

com.gosmartsimulation.finalize

Do any remaining preparation before the simulation can start.

doInit(guid)

com.gosmartsimulation.init

Dummy call for the moment.

doLogs(guid, only=None)

com.gosmartsimulation.logs

Retrieve the container logs for a simulation.

doProperties(guid)

com.gosmartsimulation.properties

Return important server-side simulation properties.

doRequestDiagnostic(guid, target)

com.gosmartsimulation.request_diagnostic

Push a bundle of diagnostic files through the transferrer. If target is None, assume gateway is running a temporary HTTP on default client port

FIXME: this should be made asynchronous!

doRequestFiles(guid, files)

com.gosmartsimulation.request_files

Push the requested output files through the transferrer and return the list that was sent.

doRequestResults(guid, target)

com.gosmartsimulation.request_results

Push a bundle of output files through the transferrer. If target is None, assume gateway is running a temporary HTTP on default client port

FIXME: this should be made asynchronous!

doRetrieveStatus(guid, allow_resync=True)

com.gosmartsimulation.retrieve_status

Get the latest status for a simulation. allow_resync permits the server to update the DB if it finds an inconsistency.

doSearch(guid, limit=None)

com.gosmartsimulation.search

Check for matching definitions

doSimulate(guid)

Start the simulation

This occurs in a separately scheduled coro from the RPC call so it will almost certainly have returned by time we do. This does not have an API endpoint as doStart() is responsible for launching it asynchronously.

doStart(guid)

com.gosmartsimulation.start

Execute the simulation in a coro.

doTmpValidation(guid, directory)

doUpdateFiles(guid, files)

com.gosmartsimulation.update_files

Add the passed files to the simulation’s reference dictionary of required input files (available to be requested later)

doUpdateSettingsXml(guid, xml)

com.gosmartsimulation.update_settings_xml

Set the GSSA-XML for a given simulation

eventComplete(guid)

Called when simulation completes

Publishes a completion event: com.gosmartsimulation.complete

eventFail(guid, message)

Called when simulation fails

Publishes a failure event: com.gosmartsimulation.failed

getProperties(guid)

Server-specific properties for this simulation

At present, just working directory location.

onRequestAnnounce()

com.gosmartsimulation.request_announce

Release a status report on each simulation in the database

TODO: this gets unwieldy, perhaps it should have an earliest simulation timestamp argument?

onRequestIdentify()

com.gosmartsimulation.request_identify

Publish basic server information.

$$ score = #cores - #active_simulations $$

This gives an availability estimate, the higher the better

setDatabase(database)

Update database backend.

Mainly for start-up. Mark everything in-progress in the DB as not-in-progress/unfinished.

setStatus(id, key, message, percentage, timestamp)

Record a status change in the database and on the filesystem.

Note that, for both those reasons, this could be slow and so should always be run with call_soon_threadsafe

FIXME: we need some sort of rate limiting here, or producer-consumer pattern with ability to skip once getting behind

updateStatus(guid, percentage, message)

Update the status.

Sets up a callback for asyncio to update database.

gssa.session module

class gssa.session.GoSmartSimulationServerSession(x, server_id, database, ignore_development=False, simdata_path='/tmp')

Bases: autobahn.asyncio.wamp.ApplicationSession

This subclasses ApplicationSession, which runs inside an Autobahn WAMP session.

doApi()

com.gosmartsimulation.api

gssa.server.GoSmartSimulationServerComponent.doApi()

Find the current API version in use. The API version only needs to be bumped when backward-incompatible changes occur on either side.

doCancel(guid)

com.gosmartsimulation.cancel

gssa.server.GoSmartSimulationServerComponent.doCancel()

Prematurely stop a running simulation but bear in mind that no request for confirmation exists!

doClean(guid)

com.gosmartsimulation.clean

gssa.server.GoSmartSimulationServerComponent.doClean()

Remove anything in simulation directory, for instance.

doCompare(this_xml, that_xml)

com.gosmartsimulation.compare

gssa.server.GoSmartSimulationServerComponent.doCompare()

Check whether two GSSA-XML files match and, if not, what their differences are.

doFinalize(guid, client_directory_prefix)

com.gosmartsimulation.finalize

gssa.server.GoSmartSimulationServerComponent.doFinalize()

Do any remaining preparation before the simulation can start.

doInit(guid)

com.gosmartsimulation.init

gssa.server.GoSmartSimulationServerComponent.doInit()

Dummy call for the moment.

doLogs(guid, only=None)

com.gosmartsimulation.logs

gssa.server.GoSmartSimulationServerComponent.doLogs()

Retrieve the container logs for a simulation.

doProperties(guid)

com.gosmartsimulation.properties

gssa.server.GoSmartSimulationServerComponent.doProperties()

Return important server-side simulation properties.

doRequestDiagnostic(guid, target)

com.gosmartsimulation.request_diagnostic

gssa.server.GoSmartSimulationServerComponent.doRequestDiagnostic()

Push a bundle of diagnostic files through the transferrer.

doRequestFiles(guid, files)

com.gosmartsimulation.request_files

gssa.server.GoSmartSimulationServerComponent.doRequestFiles()

Push the requested output files through the transferrer and return the list that was sent.

doRequestResults(guid, target)

com.gosmartsimulation.request_results

gssa.server.GoSmartSimulationServerComponent.doRequestResults()

Push a bundle of result files through the transferrer.

doRetrieveStatus(guid)

com.gosmartsimulation.retrieve_status

gssa.server.GoSmartSimulationServerComponent.doRetrieveStatus()

Get the latest status for a simulation.

doSearch(guid, limit=None)

com.gosmartsimulation.search

gssa.server.GoSmartSimulationServerComponent.doSearch()

Check for matching definitions.

doStart(guid)

com.gosmartsimulation.start

gssa.server.GoSmartSimulationServerComponent.doStart()

Execute the simulation in a coro.

doTmpValidation(guid, directory)

doUpdateFiles(guid, files)

com.gosmartsimulation.update_files

gssa.server.GoSmartSimulationServerComponent.doUpdateFiles()

Add the passed files to simulation’s reference dictionary of required input files (available to be requested later).

doUpdateSettingsXml(guid, xml)

com.gosmartsimulation.update_settings_xml

gssa.server.GoSmartSimulationServerComponent.doUpdateSettingsXml()

Set the GSSA-XML for a given simulation.

onJoin(details)

Register methods and subscribes.

Fired when we first join the router - this gives us a chance to register everything.

onRequestAnnounce()

com.gosmartsimulation.request_announce

gssa.server.GoSmartSimulationServerComponent.onRequestAnnounce()

Release a status report on each simulation in the database. TODO: this gets unwieldy, perhaps it should have an earliest simulation timestamp argument?.

onRequestIdentify()

com.gosmartsimulation.request_identify

gssa.server.GoSmartSimulationServerComponent.onRequestIdentify()

Publish basic server information.

gssa.sftp_transferrer module

class gssa.sftp_transferrer.SFTPTransferrer

Bases: object

configure_from_xml(xml)

connect()

disconnect()

pull_files(files, root, remote_root)

push_files(files, root, remote_root)

gssa.shadow_watcher module

gssa.shadow_watcher.observe(guid, transferrer, update_callback)

DEPRECATED

gssa.tmp_transferrer module

class gssa.tmp_transferrer.TmpTransferrer

Bases: object

configure_from_xml(xml)

connect()

disconnect()

pull_files(files, root, remote_root)

push_files(files, root, remote_root)

gssa.transferrer module

gssa.translator module

class gssa.translator.GoSmartSimulationTranslator

Bases: object

This extracts basic information, common to all families, from GSSA-XML.

get_files_required()

Return a list of files that should be sent back to the client.

This may be supplemented by the family when examining the model.

translate(xml)

Extract basic information from GSSA-XML.

Returns:

• lxml.etree.Element: numerical model node
• lxml.etree.Element: node full of global parameters
• lxml.etree.Element: node full of algorithms

Return type:tuple

gssa.utils module

gssa.utils.get_default_gateway()

Returns the default gateway of this machine (container).

This method is primarily intended for use communicating with ad-hoc Glot HTTP servers for file transfer (not an intended application of WAMP, so an alternative approach must be used).

Module contents