Generate access URLs on server side

[ponyisi]

File delivery is currently tightly tied to the assumption that we use S3, and with a specific bucket structure. This leads to potential security issues and makes it difficult to move to other storage backends; in particular the frontend uses an S3 client and the relevant keys need to be delivered to it.

Since the object store is read-only from the client side, it would make more sense to directly give the client a URL for each file ("signed" in the case of S3, with bearer token in the case of macaroon-based systems like xrootd). This would completely abstract the backend storage system from the client; it would also mean that the clients would only get credentials sufficient to access their specific files. This requires the server to have a new/modified endpoint that returns an access URL + token given a file.

[ponyisi]

Concrete proposal for endpoint API @BenGalewsky @MattShirley :

Client sends a JSON request with the following Python-equivalent schema

transform_id: str
file_list: List[str]  # avoid forcing one request per file
scheme: Optional[str]  # allow client to request http:// or root:// (default will be http)

if backend does not support a requested scheme, there will be an error
otherwise, return

file_uri: Mapping[str, AccessInfo]

where AccessInfo is a tuple [uri: str, headers: Mapping[str, str], expires: int] (to accommodate both the cases of S3 signed URLs, which are just a URL, and of macaroons, where we would have a bearer token)