Module ciocore.data¶
A singleton containing all the data from Conductor endpoints that's needed for writing submission tools. Specifiically, it provides projects, instance types, and software package data.
It's also possible to use cached fixtures for development purposes.
Functions¶
init¶
Source
def init(*products, **kwargs):
global __products__
if products:
if kwargs.get("product"):
raise ValueError("Arguments: `products` and `product` specified. Please don't use both together. The `product` arg is deprecated.")
__products__ = list(products)
elif kwargs.get("product"):
if kwargs.get("product") == "all":
__products__ = []
else:
__products__ = [kwargs.get("product")]
else:
__products__ = []
init(*products, **kwargs)
-
Initialize and let the module know what host products to provide in the
software
property.Arguments:
products
-- Provide a list of products as a filter. If no products are given, the software data structure is built using all products from the packages endpoint. If you provide more than one product, they must all be host level products.
Keyword Arguments:
product
-- (DEPRECATED) You can provide one product with the product keyword, or you can provide the string"all"
. This is ignored if any product args are present. -- Defaults toNone
.
Example
from ciocore import data as coredata coredata.init() # OR coredata.init("maya-io") # OR LEGACY coredata.init(product="all") # OR coredata.init(product="maya-io")
data¶
Source
def data(force=False):
global __data__
global __products__
global __fixtures_dir__
global __platforms__
if __products__ == None:
raise ValueError(
'Data must be initialized before use, e.g. data.init("maya-io") or data.init().'
)
if force:
clear()
if __data__ == {}:
# PROJECTS
projects_json = _get_json_fixture("projects")
if projects_json:
__data__["projects"] = projects_json
else:
__data__["projects"] = sorted(api_client.request_projects())
# INST_TYPES
instance_types = _get_json_fixture("instance_types")
if not instance_types:
instance_types = api_client.request_instance_types()
it_platforms = set([it["operating_system"] for it in instance_types])
# SOFTWARE
software = _get_json_fixture("software")
if not software:
software = api_client.request_software_packages()
kwargs = {"platforms":it_platforms}
# If there's only one product, it's possible to initialize the software tree with a plugin.
# So we set the product kwarg. Otherwise, we set the host_products kwarg
host_products = __products__
if len(__products__) == 1:
host_products = []
kwargs["product"] = __products__[0]
software_tree = PackageTree(software, *host_products, **kwargs)
if software_tree:
__data__["software"] = software_tree
# Revisit instance types to filter out any that are not needed for any software package.
sw_platforms = software_tree.platforms()
instance_types = [it for it in instance_types if it["operating_system"] in sw_platforms]
__data__["instance_types"] = sorted(instance_types, key=lambda k: (k["cores"], k["memory"]))
__platforms__ = set([it["operating_system"] for it in __data__["instance_types"]])
return __data__
data(force=False)
-
Provides projects, instance types, and software package data.
If data has been fetched already, then return it, otherwise make requests to fetch it. The user may have to authentiicate.
The set of instance types and software are each potentially pruned to match the available platforms represented by the other. If the instance types come from an orchestrator that provides both windows and linux machines, and the software product(s) are available on both platforms, then no pruning occurs. However, if there are no windows machines, then any windows software is removed from the package tree. Likewise, if a product is chosen that only runs on Windows, then Linux instance types will be culled from the list of available hardware.
Keyword Arguments:
force
-- IfTrue
, then fetch fresh data -- Defaults toFalse
.
Raises:
ValueError
-- Module was not initialized with the init() method.
Returns:
-
A dictionary with 3 keys:
projects
,instance_types
,software
. -
projects
is a list of project names for the authenticated account. instance_types
is a list of machine specifications available.software
is a PackageTree object containing either all software available at conductor, or a subset according to the product specified on initialization.
Example
from ciocore import data as coredata coredata.init(product="maya-io") coredata.data()["software"] # Result: # <ciocore.package_tree.PackageTree object at 0x10e9a4040> coredata.data()["projects"][0] # Result: # ATestForScott coredata.data()["instance_types"][-1]["description"] # Result: # 160 core, 3844GB Mem
valid¶
Source
def valid():
global __data__
if not __data__.get("projects"):
return False
if not __data__.get("instance_types"):
return False
if not __data__.get("software"):
return False
return True
valid()
-
Check validity of the data.
Returns:
- True if
projects
,instance_types
, andsoftware
exists.
Example
from ciocore import data as coredata coredata.valid() # Result: # True
- True if
clear¶
Source
def clear():
clear()
-
Clear out the data.
valid() returns False after clear().
products¶
Source
def products():
global __products__
return __products__
products()
-
Returns:
- The product names. An empty array signifies all products.
set_fixtures_dir¶
Source
def set_fixtures_dir(path):
global __fixtures_dir__
__fixtures_dir__ = path or ""
set_fixtures_dir(path)
-
Specify a directory in which to find JSON files representing the three sets of data to provide. The individual filenames are:
- projects.json
- instance_types.json
- software.json
These files could be used in an environment where machines can't access the internet. They are also useful as a cache for developers who need to reload often as it avoids waiting for the network.
In order to get the content for the fixtures files, use the following Example
Example
from ciocore import api_client projects = api_client.request_projects() instance_types = api_client.request_instance_types() software = api_client.request_software_packages() # Write that data as JSON to the filenames listed above.
Arguments:
path
-- Directory in which to find the above files.
platforms¶
Source
def platforms():
global __platforms__
return __platforms__
platforms()
-
The set of platforms that are found in both software and instance types.
Returns:
- A set containing platforms: windows and/or linux or neither.