RESTful

As described in the overview, the SKA-DLM is exposing all the managers and their relevant functions through REST interfaces served through a FastAPI server. The REST calls are protected through a AAA API gateway connecting to an organisation wide AAA system. Both are external services in operations, but we have implemented a gateway and are using a Keycloak AAA system for testing and evaluation of the system. Since we are using FastAPI, the documentation of the available REST calls is autogenerated at run-time and available on the same URLs as the managers’ URLs. For the local evaluation setup these are:

For an example, see our DP Cluster guide.

As an example the Request Manager REST page looks like this:

../_images/ska-dlm-rest.png

Request

GET /request/query_data_item

Query Data Item

Query a data_item.

params or item_name/oid/uid is required.

Query Parameters:

  • item_name (string) – Could be empty, in which case the first 1000 items are returned

  • oid (string) – Return data_items referred to by the OID provided.

  • uid (string) – Return data_item referred to by the UID provided.

  • storage_id (string) – Return data_item referred to by a given storage_id.

  • params ({'null', 'string'}) – specify the query parameters

Status Codes:
PATCH /request/update_data_item

Update Data Item

Update fields of an existing data_item.

This is mostly used by the other convenience functions. In general when specifying an OID or an item_name, multiple entries will be updated at the same time.

Query Parameters:

  • item_name (string) – the name of the data_items to be updated

  • oid (string) – the OID of the data_items to be updated

  • uid (string) – the UID of the data_item to be updated

Status Codes:
PATCH /request/set_uri

Set Uri

Set the URI field of the uid data_item.

Query Parameters:

  • uid (string) – the uid of the data_item to be updated (Required)

  • uri (string) – the access URI for the data_item (Required)

  • storage_id (string) – the storage_id associated with the URI (Required)

Status Codes:
PATCH /request/set_metadata

Set Metadata

Populate the metadata column for a data_item with the metadata.

Query Parameters:

  • uid (string) – the UID of the data_item to be updated (Required)

Status Codes:
PATCH /request/set_state

Set State

Set the state field of the uid data_item.

Query Parameters:

  • uid (string) – the uid of the data_item to be updated (Required)

  • state (string) – the new state for the data_item (Required)

Status Codes:
PATCH /request/set_oid_expiration

Set Oid Expiration

Set the oid_expiration field of the data_items with the given OID.

Query Parameters:

  • oid (string) – the oid of the data_item to be updated (Required)

  • expiration (string) – the expiration date for the data_item (Required)

Status Codes:
PATCH /request/set_uid_expiration

Set Uid Expiration

Set the uid_expiration field of the data_item with the given UID.

Query Parameters:

  • uid (string) – the UID of the data_item to be updated (Required)

  • expiration (string) – the expiration date for the data_item (Required)

Status Codes:
PATCH /request/set_user

Set User

Set the user field of the data_item(s) with the given OID or UID.

Query Parameters:

  • oid (string) – the OID of the data_item to be updated

  • uid (string) – the UID of the data_item to be updated

  • user (string) – the user for the data_item

Status Codes:
PATCH /request/set_group

Set Group

Set the user field of the data_item(s) with the given OID or UID.

Query Parameters:

  • oid (string) – the OID of the data_item to be updated

  • uid (string) – the UID of the data_item to be updated

  • group (string) – the group for the data_item

Status Codes:
PATCH /request/set_acl

Set Acl

Set the user field of the data_item(s) with the given OID or UID.

Query Parameters:

  • oid (string) – the OID of the data_item to be updated

  • uid (string) – the UID of the data_item to be updated

  • acl (string) – the acl dict for the data_item

Status Codes:
PATCH /request/set_phase

Set Phase

Set the phase field of the data_item(s) with given UID.

Query Parameters:

  • uid (string) – the UID of the data_item to be updated (Required)

  • phase (string) – the phase for the data_item (Required)

Status Codes:
PATCH /request/update_item_tags

Update Item Tags

Update/set the item_tags field of a data_item with given item_name/OID.

This will update all records for a data_item at the same time. Updating a single UID does not make sense.

Query Parameters:

  • item_name (string) – the name of the data_item

  • oid (string) – the OID of the data_item to be updated

Status Codes:
GET /request/query_expired

Query Expired

Query for all expired data_items using the uid_expiration timestamp.

Query Parameters:

  • offset ({'null', 'duration'}) – optional offset for the query

Status Codes:
GET /request/query_deleted

Query Deleted

Query for all deleted data_items using the deleted state.

Query Parameters:

  • uid (string) – The UID to be checked, optional.

Status Codes:
GET /request/query_new

Query New

Query for all data_items newer than the date provided.

Query Parameters:

  • check_date (string) – the UTC starting date (exclusive) (Required)

  • uid (string) – The UID to be checked, optional.

Status Codes:
GET /request/query_exists

Query Exists

Query to check for existence of a data_item.

Query Parameters:

  • item_name (string) – optional item_name

  • oid (string) – the oid to be searched for

  • uid (string) – this returns only one storage_id

  • ready (boolean) – whether the item must be in READY state.

Status Codes:
GET /request/query_exist_and_ready

Query Exists And Ready

Check whether a data_item exists and is in READY state.

Query Parameters:

  • item_name (string) – optional item_name

  • oid (string) – the oid to be searched for

  • uid (string) – this returns only one storage_id

Status Codes:
GET /request/query_item_storage

Query Item Storage

Query for the storage_ids of all backends holding a copy of a data_item.

Either an item_name or a OID have to be provided.

Query Parameters:

  • item_name (string) – optional item_name

  • oid (string) – the oid to be searched for

  • uid (string) – this returns only one storage_id

Status Codes:

Ingest

POST /ingest/init_data_item

Init Data Item

Initialise a new data_item.

item_name or json_data is required.

Query Parameters:

  • item_name ({'null', 'string'}) – the item_name, can be empty, but then json_data has to be specified.

  • phase (string) – the phase this item is set to (usually inherited from the storage)

Status Codes:
Request Headers:

  • authorization – Validated Bearer token with UserInfo

POST /ingest/register_data_item

Register Data Item

Ingest a data_item (register function is an alias).

This high level function is a combination of init_data_item, set_uri and set_state(READY). It also checks whether a data_item is already registered on the requested storage.

  1. check whether requested storage is known and accessible

  2. check, if required, whether item is accessible/exists on that storage

  3. check whether item is already registered on that storage

  4. initialise the item on the storage

  5. set the access path to the payload

  6. set state to READY

  7. save metadata in the data_item table

Query Parameters:

  • item_name (string) – item name to register with. Does not need to be unique. (Required)

  • uri (string) – the relative access path to the payload. (Required)

  • item_type (string)

  • storage_name (string) – the name of the configured storage volume (name or ID required)

  • storage_id (string) – the ID of the configured storage.

  • parents ({'null', 'string'}) – uuid of parent item

  • do_storage_access_check (boolean) – perform check_storage_access() against provided storage and uri

Status Codes:
Request Headers:

  • authorization – Validated Bearer token with UserInfo

Storage

GET /storage/query_location

Query Location

Query a location.

Query Parameters:

  • location_name (string) – could be empty, in which case the first 1000 items are returned

  • location_id (string) – Return locations referred to by the location_id provided.

Status Codes:
POST /storage/init_storage

Init Storage

Initialise a new storage.

Query Parameters:

  • storage_name (string) – An organisation or owner name for the storage. (Required)

  • storage_type (string) – high level type of the storage: ‘filesystem’, ‘objectstore’ or ‘tape’ (Required)

  • storage_interface (string) – storage interface for rclone access, e.g. “posix”, “s3” (Required)

  • root_directory (string) – data directory as an absolute path on the remote storage endpoint (Required)

  • location_id ({'null', 'string'}) – a dlm registered location id

  • location_name ({'null', 'string'}) – a dlm registered location name

  • storage_capacity (integer) – reserved storage capacity in bytes

  • storage_phase (string) – one of “GAS”, “LIQUID”, “SOLID”

Status Codes:
POST /storage/create_storage_config

Create Storage Config

Create a new record in the storage_config table for a given storage_id.

Query Parameters:

  • storage_id (string) – the storage_id for which to create the entry.

  • storage_name (string) – the name of the storage for which the config is provided.

  • config_type (string) – default is ‘rclone’. Alternative enums from ConfigType.

Status Codes:
GET /storage/get_storage_config

Get Storage Config

Get the storage configuration entry for a particular storage backend.

Query Parameters:

  • storage_id (string) – the storage id, by default “”

  • storage_name (string) – the name of the storage volume, by default “”

  • config_type (string) – query only the specified type, by default “rclone”

Status Codes:
POST /storage/rclone_config

Create Rclone Config

Create a new rclone backend configuration entry on the rclone server.

Status Codes:
POST /storage/init_location

Init Location

Initialise a new storage location.

Query Parameters:

  • location_name (string) – the orgization or owner’s name managing the storage location. (Required)

  • location_type (string) – the location type, e.g. “low-operational” (Required)

  • location_country (string) – the location country name (AU, ZA or UK)

  • location_city (string) – the location city name

  • location_facility (string) – the location facility name

Status Codes:
GET /storage/query_storage

Query Storage

Query storage locations.

Query Parameters:

  • storage_name (string) – Name of the storage to query. If not provided, the first 1000 locations are returned.

  • storage_id (string) – ID of the storage to query. Ignored if storage_name is provided.

Status Codes:

Migration

GET /migration/query_migrations

Query Migrations

Query for all migrations by a given user, with optional filters.

Query Parameters:

  • start_date ({'null', 'string'}) – Filter migrations that started after this date (YYYY-MM-DD or YYYYMMDD)

  • end_date ({'null', 'string'}) – Filter migrations that ended before this date (YYYY-MM-DD or YYYYMMDD)

  • storage_id ({'null', 'string'}) – Filter migrations by a specific storage location

Status Codes:
Request Headers:

  • authorization – Validated Bearer token with UserInfo

GET /migration/get_migration

Get Migration Record

Query for a specific migration.

Query Parameters:

  • migration_id (integer) – Migration id of migration (Required)

Status Codes:
POST /migration/copy_data_item

Copy Data Item

Copy a data_item from source to destination.

Steps (1) get the current storage_id(s) of the item (2) convert one (first) storage_id to a configured rclone backend (3) initialise the new item with the same OID on the new storage (4) use the rclone copy command to copy it to the new location (5) set the access path to the payload (6) set state to READY (7) save metadata in the data_item table

Query Parameters:

  • item_name (string) – data item name, when empty the first 1000 items are returned, by default “”

  • oid (string) – object id, Return data_items referred to by the OID provided, by default “”

  • uid (string) – Return data_item referred to by the UID provided, by default “”

  • destination_name (string) – the name of the destination storage volume, by default “”

  • destination_id (string) – the destination storage, by default “”

  • path (string) – the destination path relative to storage root, by default “”

Status Codes:
Request Headers:

  • authorization – Validated Bearer token with UserInfo