RSE Classes

Client

class rucio.client.rseclient.RSEClient(rucio_host=None, auth_host=None, account=None, ca_cert=None, auth_type=None, creds=None, timeout=600, user_agent='rucio-clients')

Bases: rucio.client.baseclient.BaseClient

RSE client class for working with rucio RSEs

RSE_BASEURL = 'rses'
add_distance(source, destination, parameters)

Add a src-dest distance.

Parameters:
  • source – The source.
  • destination – The destination.
  • parameters – A dictionnary with property.
add_protocol(rse, params)

Sends the request to create a new protocol for the given RSE.

Parameters:
  • rse – the name of the rse.
  • scheme – identifier of this protocol
  • params – Attributes of the protocol. Supported are: hostname: hostname for this protocol (default = localhost) port: port for this protocol (default = 0) prefix: string used as a prfeix for this protocol when generating the PFN (default = None) impl: qualified name of the implementation class for this protocol (mandatory) read: integer representing the priority of this procotol for read operations (default = -1) write: integer representing the priority of this procotol for write operations (default = -1) delete: integer representing the priority of this procotol for delete operations (default = -1) extended_attributes: miscellaneous protocol specific information e.g. spacetoken for SRM (default = None)
Returns:

True if protocol was created successfully else False.

Raises:
  • Duplicate – if protocol with same hostname, port and protocol identifier already exists for the given RSE.
  • RSENotFound – if the RSE doesn’t exist.
  • KeyNotFound – if params is missing manadtory attributes to create the protocol.
  • AccessDenied – if not authorized.
add_rse(rse, **kwargs)

Sends the request to create a new RSE.

Parameters:
  • rse – the name of the rse.
  • deterministic – Boolean to know if the pfn is generated deterministically.
  • volatile – Boolean for RSE cache.
  • city – City for the RSE.
  • region_code – The region code for the RSE.
  • country_name – The country.
  • continent – The continent.
  • time_zone – Timezone.
  • staging_area – Staging area.
  • ISP – Internet service provider.
Returns:

True if location was created successfully else False.

Raises:

Duplicate – if rse already exists.

add_rse_attribute(rse, key, value)

Sends the request to add a RSE attribute.

Parameters:
  • rse – the name of the rse.
  • key – the attribute key.
  • value – the attribute value.
Returns:

True if RSE attribute was created successfully else False.

Raises:

Duplicate – if RSE attribute already exists.

delete_protocols(rse, scheme, hostname=None, port=None)

Deletes matching protocols from RSE. Protocols using the same identifier can be distinguished by hostname and port.

Parameters:
  • rse – the RSE name.
  • scheme – identifier of the protocol.
  • hostname – hostname of the protocol.
  • port – port of the protocol.
Returns:

True if success.

Raises:
delete_rse(rse)

Sends the request to delete a rse.

Parameters:rse – the name of the rse.
Returns:True if location was created successfully else False.
delete_rse_attribute(rse, key)

Sends the request to delete a RSE attribute.

Parameters:
  • rse – the RSE name.
  • key – the attribute key.
Returns:

True if RSE attribute was deleted successfully else False.

get_distance(source, destination)

Get distances between rses.

Parameters:
  • source – The source RSE.
  • destination – The destination RSE.
Returns distance:
 

List of dictionaries.

get_protocols(rse, protocol_domain='ALL', operation=None, default=False, scheme=None)

Returns protocol information. Parameter comibantions are: (operation OR default) XOR protocol.

Parameters:
  • rse – the RSE name.
  • protocol_domain – The scope of the protocol. Supported are ‘LAN’, ‘WAN’, and ‘ALL’ (as default).
  • operation – The name of the requested operation (read, write, or delete). If None, all operations are queried.
  • default – Indicates if only the default operations should be returned.
  • scheme – The identifier of the requested protocol.
Returns:

A list with details about each matching protocol.

Raises:
get_rse(rse)

Returns details about the referred RSE.

Parameters:rse – Name of the referred RSE
Returns:A dict containing all attributes of the referred RSE
Raises:RSENotFound – if the referred RSE was not found in the database
get_rse_limits(rse)

Get RSE limits.

Parameters:rse – The RSE name.
Returns:True if successful, otherwise false.
get_rse_usage(rse, filters=None)

Get RSE usage information.

Parameters:
  • rse – the RSE name.
  • filters – dictionary of attributes by which the results should be filtered
Returns:

True if successful, otherwise false.

lfns2pfns(rse, lfns, protocol_domain='ALL', operation=None, scheme=None)

Returns PFNs that should be used at a RSE, corresponding to requested LFNs. The PFNs are generated for the RSE regardless of whether a replica exists for the LFN.

Parameters:
  • rse – the RSE name
  • lfns – A list of LFN strings to translate to PFNs.
  • protocol_domain – The scope of the protocol. Supported are ‘LAN’, ‘WAN’, and ‘ALL’ (as default).
  • operation – The name of the requested operation (read, write, or delete). If None, all operations are queried.
  • scheme – The identifier of the requested protocol (gsiftp, https, davs, etc).
Returns:

A dictionary of LFN / PFN pairs.

Raises:
list_rse_attributes(rse)

Sends the request to get RSE attributes.

Parameters:rse – The RSE name.
Returns:True if RSE attribute was created successfully else False.
list_rse_usage_history(rse, filters=None)

List RSE usage history information.

Parameters:
  • rse – The RSE name.
  • filters – dictionary of attributes by which the results should be filtered.
Returns:

list of dictionnaries.

list_rses(rse_expression=None)

Sends the request to list all rucio locations(RSEs).

Rse_expression:RSE Expression to use as filter.
Returns:a list containing the names of all rucio locations.
set_rse_limits(rse, name, value)

Set RSE limit information.

Parameters:
  • rse – The RSE name.
  • name – The name of the limit.
  • value – The feature value. Set to -1 to remove the limit.
Returns:

True if successful, otherwise false.

set_rse_usage(rse, source, used, free)

Set RSE usage information.

Parameters:
  • rse – the RSE name.
  • source – the information source, e.g. srm.
  • used – the used space in bytes.
  • free – the free in bytes.
Returns:

True if successful, otherwise false.

swap_protocols(rse, domain, operation, scheme_a, scheme_b)

Swaps the priorities of the provided operation.

Parameters:
  • rse – the RSE name.
  • domain – the domain in which priorities should be swapped i.e. wan or lan.
  • operation – the operation that should be swapped i.e. read, write, or delete.
  • scheme_a – the scheme of one of the two protocols to be swapped, e.g. srm.
  • scheme_b – the scheme of the other of the two protocols to be swapped, e.g. http.
Returns:

True if success.

Raises:
update_distance(source, destination, parameters)

Update distances with the given RSE ids.

Parameters:
  • source – The source.
  • destination – The destination.
  • parameters – A dictionnary with property.
update_protocols(rse, scheme, data, hostname=None, port=None)

Updates matching protocols from RSE. Protocol using the same identifier can be distinguished by hostname and port.

Parameters:
  • rse – the RSE name.
  • scheme – identifier of the protocol.
  • data – A dict providing the new values of the protocol attibutes. Keys must match column names in database.
  • hostname – hostname of the protocol.
  • port – port of the protocol.
Returns:

True if success.

Raises:
update_rse(rse, parameters)

Update RSE properties like availability or name.

Parameters:
  • rse – the name of the new rse.
  • parameters – A dictionnary with property (name, read, write, delete as keys).

Manager

rucio.rse.rsemanager.create_protocol(rse_settings, operation, scheme=None, domain='wan')

Instanciates the protocol defined for the given operation.

Parameters:
  • rse_attr – RSE attributes
  • operation – Intended operation for this protocol
  • scheme – Optional filter if no specific protocol is defined in rse_setting for the provided operation
  • domain – Optional specification of the domain
Returns:

An instance of the requested protocol

rucio.rse.rsemanager.delete(rse_settings, lfns)

Delete a file from the connected storage. Providing a list indicates the bulk mode.

Parameters:

lfns – a single dict or a list with dicts containing ‘scope’ and ‘name’. E.g. [{‘name’: ‘1_rse_remote_delete.raw’, ‘scope’: ‘user.jdoe’}, {‘name’: ‘2_rse_remote_delete.raw’, ‘scope’: ‘user.jdoe’}]

Returns:

True/False for a single file or a dict object with ‘scope:name’ as keys and True or the exception as value for each file in bulk mode

Raises:
rucio.rse.rsemanager.download(rse_settings, files, dest_dir=None, force_scheme=None, ignore_checksum=False, printstatements=False, domain='wan', transfer_timeout=None)

Copy a file from the connected storage to the local file system. Providing a list indicates the bulk mode.

Parameters:
  • rse_settings – RSE to use
  • files – a single dict or a list with dicts containing ‘scope’ and ‘name’ if LFNs are provided and additional ‘pfn’ if PFNs are provided. Examples: [ {‘name’: ‘2_rse_remote_get.raw’, ‘scope’: ‘user.jdoe’}, {‘name’:‘3_rse_remote_get.raw’, ‘scope’: ‘user.jdoe’, ‘pfn’: ‘user/jdoe/5a/98/3_rse_remote_get.raw’} ]
  • dest_dir – path to the directory where the downloaded files will be stored. If not given, each scope is represented by its own directory.
  • force_scheme – normally the scheme is dictated by the RSE object, when specifying the PFN it must be forced to the one specified in the PFN, overruling the RSE description.
  • ignore_checksum – do not verify the checksum - caution: should only be used for rucio download –pfn
  • transfer_timeout – set this timeout (in seconds) for the transfers, for protocols that support it
Returns:

True/False for a single file or a dict object with ‘scope:name’ for LFNs or ‘name’ for PFNs as keys and True or the exception as value for each file in bulk mode

Raises:
rucio.rse.rsemanager.exists(rse_settings, files)

Checks if a file is present at the connected storage. Providing a list indicates the bulk mode.

Parameters:files – a single dict or a list with dicts containing ‘scope’ and ‘name’ if LFNs are used and only ‘name’ if PFNs are used. E.g. {‘name’: ‘2_rse_remote_get.raw’, ‘scope’: ‘user.jdoe’}, {‘name’: ‘user/jdoe/5a/98/3_rse_remote_get.raw’}
Returns:True/False for a single file or a dict object with ‘scope:name’ for LFNs or ‘name’ for PFNs as keys and True or the exception as value for each file in bulk mode
Raises:RSENotConnected – no connection to a specific storage has been established
rucio.rse.rsemanager.find_matching_scheme(rse_settings_dest, rse_settings_src, operation_src, operation_dest, domain='wan', scheme=None)

Find the best matching scheme between two RSEs

Parameters:
  • rse_settings_dest – RSE settings for the destination RSE.
  • rse_settings_src – RSE settings for the src RSE.
  • operation_src – Source Operation such as read, write.
  • operation_dest – Dest Operation such as read, write.
  • domain – Domain such as lan, wan.
  • scheme – List of supported schemes.
Returns:

Tuple of matching schemes (dest_scheme, src_scheme).

rucio.rse.rsemanager.get_protocols_ordered(rse_settings, operation, scheme=None, domain='wan')
rucio.rse.rsemanager.get_rse_info(rse, session=None)

Returns all protocol related RSE attributes.

Parameters:
  • rse – Name of the requested RSE
  • session – The eventual database session.
Returns:

a dict object with the following attributes: id … an internal identifier rse … the name of the RSE as string type … the storage type odf the RSE e.g. DISK volatile … boolean indictaing if the RSE is volatile verify_checksum … boolean indicating whether RSE supports requests for checksums deteministic … boolean indicating of the nameing of the files follows the defined determinism domain … indictaing the domain that should be assumed for transfers. Values are ‘ALL’, ‘LAN’, or ‘WAN’ protocols … all supported protocol in form of a list of dict objects with the followig structure - scheme … protocol scheme e.g. http, srm, … - hostname … hostname of the site - prefix … path to the folder where the files are stored - port … port used for this protocol - impl … naming the python class of the protocol implementation - extended_attributes … additional information for the protocol - domains … a dict naming each domain and the priority of the protocol for each operation (lower is better, zero is not upported)

Raises:

RSENotFound – if the provided RSE coud not be found in the database.

rucio.rse.rsemanager.get_space_usage(rse_settings, scheme=None)

Get RSE space usage information.

Parameters:scheme – optional filter to select which protocol to be used.
Returns:a list with dict containing ‘totalsize’ and ‘unusedsize’
Raises:ServiceUnavailable – if some generic error occured in the library.
rucio.rse.rsemanager.lfns2pfns(rse_settings, lfns, operation='write', scheme=None, domain='wan')

Convert the lfn to a pfn

Parameters:
  • lfns – logical file names as a dict containing ‘scope’ and ‘name’ as keys. For bulk a list of dicts can be provided
  • protocol – instance of the protocol to be used to create the PFN
Returns:

a dict with scope:name as key and the PFN as value

rucio.rse.rsemanager.parse_pfns(rse_settings, pfns, operation='read', domain='wan')

Checks if a PFN is feasible for a given RSE. If so it splits the pfn in its various components.

Parameters:
  • pfns – list of PFNs
  • protocol – instance of the protocol to be used to create the PFN
Returns:

A dict with the parts known by the selected protocol e.g. scheme, hostname, prefix, path, name

Raises:
rucio.rse.rsemanager.rename(rse_settings, files)

Rename files stored on the connected storage. Providing a list indicates the bulk mode.

Parameters:

files – a single dict or a list with dicts containing ‘scope’, ‘name’, ‘new_scope’ and ‘new_name’ if LFNs are used or only ‘name’ and ‘new_name’ if PFNs are used. If ‘new_scope’ or ‘new_name’ are not provided, the current one is used. Examples: [ {‘name’: ‘3_rse_remote_rename.raw’, ‘scope’: ‘user.jdoe’, ‘new_name’: ‘3_rse_new.raw’, ‘new_scope’: ‘user.jdoe’}, {‘name’: ‘user/jdoe/d9/cb/9_rse_remote_rename.raw’, ‘new_name’: ‘user/jdoe/c6/4a/9_rse_new.raw’} ]

Returns:

True/False for a single file or a dict object with LFN (key) and True/False (value) in bulk mode

Raises:
rucio.rse.rsemanager.select_protocol(rse_settings, operation, scheme=None, domain='wan')
rucio.rse.rsemanager.upload(rse_settings, lfns, source_dir=None, force_pfn=None, force_scheme=None, transfer_timeout=None)

Uploads a file to the connected storage. Providing a list indicates the bulk mode.

Parameters:
  • lfns – a single dict or a list with dicts containing ‘scope’ and ‘name’. Examples: [ {‘name’: ‘1_rse_local_put.raw’, ‘scope’: ‘user.jdoe’, ‘filesize’: 42, ‘adler32’: ‘87HS3J968JSNWID’}, {‘name’: ‘2_rse_local_put.raw’, ‘scope’: ‘user.jdoe’, ‘filesize’: 4711, ‘adler32’: ‘RSSMICETHMISBA837464F’} ] If the ‘filename’ key is present, it will be used by Rucio as the actual name of the file on disk (separate from the Rucio ‘name’).
  • source_dir – path to the local directory including the source files
  • force_pfn – use the given PFN – can lead to dark data, use sparingly
  • force_scheme – use the given protocol scheme, overriding the protocol priority in the RSE description
  • transfer_timeout – set this timeout (in seconds) for the transfers, for protocols that support it
Returns:

True/False for a single file or a dict object with ‘scope:name’ as keys and True or the exception as value for each file in bulk mode

Raises:

Storage

class rucio.rse.protocols.protocol.RSEProtocol(protocol_attr, rse_settings)

Bases: object

This class is virtual and acts as a base to inherit new protocols from. It further provides some common functionality which applies for the amjority of the protocols.

close()

Closes the connection to RSE.

connect()

Establishes the actual connection to the referred RSE.

Raises:RSEAccessDenied – if no connection could be established.
delete(path)

Deletes a file from the connected RSE.

Parameters:

path – path to the to be deleted file

Raises:
exists(path)

Checks if the requested file is known by the referred RSE.

Parameters:path – Physical file name
Returns:True if the file exists, False if it doesn’t
Raises:SourceNotFound – if the source file was not found on the referred storage.
get(path, dest, transfer_timeout=None)

Provides access to files stored inside connected the RSE.

Parameters:
  • path – Physical file name of requested file
  • dest – Name and path of the files when stored at the client
  • transfer_timeout – Transfer timeout (in seconds)
Raises:
get_space_usage()

Get RSE space usage information.

Returns:a list with dict containing ‘totalsize’ and ‘unusedsize’
Raises:ServiceUnavailable – if some generic error occured in the library.
lfns2pfns(lfns)

Retruns a fully qualified PFN for the file referred by path.

Parameters:path – The path to the file.
Returns:Fully qualified PFN.
parse_pfns(pfns)

Splits the given PFN into the parts known by the protocol. It is also checked if the provided protocol supportes the given PFNs.

Parameters:pfns – a list of a fully qualified PFNs
Returns:dic with PFN as key and a dict with path and name as value
Raises:RSEFileNameNotSupported – if the provided PFN doesn’t match with the protocol settings
put(source, target, source_dir, transfer_timeout=None)

Allows to store files inside the referred RSE.

Parameters:
  • source – path to the source file on the client file system
  • target – path to the destination file on the storage
  • source_dir – Path where the to be transferred files are stored in the local file system
  • transfer_timeout – Transfer timeout (in seconds)
Raises:
rename(path, new_path)

Allows to rename a file stored inside the connected RSE.

Parameters:
  • path – path to the current file on the storage
  • new_path – path to the new file on the storage
Raises:
stat(path)

Returns the stats of a file.

Parameters:

path – path to file

Raises:
Returns:

a dict with two keys, filesize and adler32 of the file provided in path.