Usage API¶
The Usage API client¶
To communicate with the AMIE REST Usage API, create a UsageClient object. You’ll need to provide your site name and your API key.
- class amieclient.client.UsageClient(site_name, api_key, usage_url='https://usage.xsede.org/api/v1')¶
AMIE Usage Client.
- Parameters
site_name (str) – Name of the client site.
api_key (str) – API key secret
usage_url (str) – Base URL for the XSEDE Usage api
Examples
>>> psc_client = amieclient.UsageClient(site_name='PSC', api_key=some_secrets_store['amie_api_key'])
You can also override the amie_url and usage_url parameters, if you’re doing local development or testing out a new version.
>>> psc_alt_base_client = amieclient.UsageClient(site_name='PSC', api_key='test_api_key', usage_url='https://amieclient.xsede.org/v0.20_beta/)
- clear_failed_records(failed_records_or_ids)¶
Tells the server to clear the failed records given
- Parameters
failed_records_or_ids ([FailedUsageRecord], [int]) – A list of FailedUsageRecords, or plain FailedRecordIds, to unmark as failed
- get_failed_records()¶
Gets all failed records
Takes no arguments
- send(usage_packets)¶
Sends a usage update to the Usage API host. This function accepts individual UsageMessages, lists of UsageRecords, or even a single UsageRecord. Returns a list of UsageResponses
The API currently has a request size limit of 1024KiB. We get ample room for overhead that may be added by intermediate layers (reverse proxies, etc) by capping the size of the request we send to 768KiB. This happens automatically, no need to chunk your usage packets yourself. But this potential chunking means that we may get more than one response, so for the sake of consistency this method will return a list of responses.
- Parameters
usage_packets (UsageMessage, [UsageRecord], UsageRecord) – A UsageMessage object, list of UsageRecords, or a single UsageRecord to send.
- Returns
list of responses
- status(from_time=None, to_time=None)¶
Gets the status of records processed from the queue in the provided interval.
- Parameters
from_date (Datetime) – Start date and time
to_date (Datetime) – End date and time
- summary()¶
Gets a usage summary
Not implemented yet
Usage Records¶
The Usage API currently handles three kinds of resources: Compute, Storage, and Adjustment. Each resource has a corresponding Record type.
Compute¶
- class amieclient.usage.record.ComputeUsageRecord(*, parent_record_id=None, queue=None, cpu_core_count=None, job_name=None, memory=None, local_reference=None, charge, end_time, local_project_id, local_record_id, resource, start_time, submit_time, username, node_count)¶
A ComputeUsageRecord. A Record of Compute Usage.
(TODO: a better description)
- as_dict()¶
Returns a dictionary version of this record
- classmethod from_dict(input_dict)¶
Returns a ComputeUsageRecord from a provided dictionary
Storage¶
- class amieclient.usage.record.StorageUsageRecord(charge, collection_time, local_project_id, local_record_id, resource, username, bytes_read=None, bytes_stored=None, bytes_written=None, collection_interval=None, file_count=None, files_read=None, files_written=None, media_type=None, system_copies=None, user_copies=None, local_reference=None)¶
- record_type = 'storage'¶
A usage record for storage usage.
- Parameters
charge (str) – The amount of allocation units that should be deducted from the project allocation for this job. For storage this is usually gigabytes stored
collection_time (str) – Time the storage use was collected
local_project_id (str) – The Site Project ID for the job. This must match the ProjectID provided by the site for the project with AMIE
local_record_id (str) – Site Record ID. Use to make the record identifiable to you locally
local_reference (str) – An optional key to help you identify this record locally. Included when the record load fails in the initial post or in the errors reported by the GET usage/status end point. This could be a primary key value from a local accounting system
resource (str) – Resource the job ran on. Must match the resource name used in AMIE
username (str) – The local username of the user who ran the job. Must match the username used in AMIE
bytes_read (str) – Number of bytes Read
bytes_stored (str) – Number of bytes stored
bytes_written (str) – Number of bytes written
collection_interval (str) – How often the storage use will be calculated in days
file_count (str) – Number of files stored
files_read (str) – Number of files read
files_written (str) – Number of files written
media_type (str) – Type of the storage (Tape, Disk, SSD, etc)
system_copies (str) – Number of copies of the data the system keeps
user_copies (str) – Number of copies of the data the user has chosen to keep
Adjustment¶
- class amieclient.usage.record.AdjustmentUsageRecord(adjustment_type, charge, start_time, local_project_id, local_record_id, resource, username, comment=None, local_reference=None)¶
Usage record for an Adjustment :param adjustment_type: Which type of allocation adjustment is this?
Valid values are ‘credit’, ‘refund’, ‘storage-credit’, ‘debit’, ‘reservation’, ‘storage-debit’
- Parameters
charge (str) – The amount of allocation units that should be deducted from the project allocation for this job. For storage this is usually gigabytes stored.
start_time (str) – Time for which this adjustment should be applied. For example a the time for a job refund should be the same as the job submit time to ensure the correct allocation is credited.
local_project_id (str) – The Site Project ID for the job. This must match the ProjectID provided by the site for the project with.
local_record_id (str) – AMIE Site Record ID. Use to make this record identifiable to you locally. Must be unique for the resource.
local_reference (str) – An optional key to help you identify this record locally. Included when the record load fails in the initial post or in the errors reported by the GET usage/status end point. This could be a primary key value from a local accounting system
resource (str) – Resource the job ran on. Must match the resource name used in AMIE
username (str) – The local username of the user who ran the job. Must match the username used in AMIE
comment (str) – Comment to explain reason for adjustment
Usage Messages¶
Usage Records are sent and received from the Usage API in the form of Usage Messages. A Usage Message is, essentially, a list of one particular type of Usage Record – so, for example, you cannot send a UsageMessage with a record of both Compute and Storage usage. The send_usage() method of the Usage Client will make a UsageMessage for you automatically if you give it a list of UsageRecords of the same kind, or if you give it a single UsageRecord.
- class amieclient.usage.message.UsageMessage(records)¶
- as_dict()¶
Returns a dictionary version of this record
- classmethod from_dict(input_dict)¶
Returns a UsageMessage from a provided dictionary
- json(**json_kwargs)¶
Returns a json version of this message
- pretty_print()¶
prints() a pretty version of the JSON of this packet
Responses¶
The Usage API backend processes received records asynchronously. This means that when you send along a UsageRecord, your data is validated, but not processed immediately. You will receive a quick response from the API, letting you know how many records were validated successfully and put on the stack to be processed. Any records that fail validation will be returned to you, so that you can correct whatever horrible mistake you have inflicted on the API. You monster.
Checking status¶
To see what the current status of your submitted records are, there is the usage_status() method. It takes two optional paramters – from_date and to_date – that define a date and time range. The response includes a list of resources. For each resource, the number of successful and failed records is sent. If there are any records that failed processing, they are sent as well, grouped by the error that occured.