Open Cloud Mesh

Early Specifications

 DRAFT protocol definition v.0.004 released

http://lukasreschke.github.io/OpenCloudMeshSpecification/

https://github.com/GEANT/OpenCloudMeshSpecification

Final Specification GitHub

API Reference Document

https://github.com/GEANT/OCM-API

https://rawgit.com/GEANT/OCM-API/v1/docs.html

Jakub Moscicki

Massimo Lamanna

Benedikt Wegmann

Ralph Krimmel

Holger Angenent

Sciebo / Uni Münster

Contribute to the specifications and development

R, C

Committed

Christoph Herzog

Simon Leinen

SWITCH

Contribute to the specifications and development

R, C

Committed

Guido Aben

David Jericho

AARNet

Contribute to the specifications and development

R, C

Committed

David Antoš

CESNET

Contribute to the specifications and development

R, C

Committed

Frederik Orellana

DeIC

Contribute to the specifications and development

R, C

Kurt Bauer

ACOnet

Contribute to the specifications and development

R, C

Jari Miettinen

CSC / Funet / EUDAT

Contribute to the specifications and development

R, C

Hrachya Astsatryan

Andranik Hayrapetyan

Wahi

Charles du Jeu

David Gillard

Initiation

Pick up the results on Phase II.

See Section 5. of Phase II. below...

• Find a professional protocol designer who will describe OCM in a more formalized way (using Swagger Framework or similar). 
• Establish a reference infrastructure and test environment where the implementations can be validated against.

Meeting with Apiwise on 12/07/2016

The OCM partners have been approached to source the vendor-agnostic, design-first formalization of the OCM API using standard frameworks and proper documentation. 

The following 8 partners offered voluntary contributions in the range of EUR 1,000 - 2,500 installments:

AARNet, Nextcloud, Sciebo, ownCloud, Uni Vienna, CESNET, GWDG and CERN

We have collected sufficient amount of money to contract with the third-party who will deliver the results. 

VERY MUCH APPRECIATED TO ALL CONTRIBUTING PARTNERS!!!

All contributing partners need to be invoiced. An MoU and Order Form have been developed to be signed, if needed by the financial offices.

All contributions received except one. Solution will be put in place by January 2017.

All DONE. Very much appreciated!

The Dutch company called APIWISE made an offer for the work that has been accepted by the contributing partners.

An amendment have been added to define how GÉANT will evaluate the results and what our priorities are.

Contract is finalized and to be signed by GÉANT CEO.

Signed!

Dimitri van Hees - Apiwise

Peter Szegedi

The former OCM specifications are now completely (i.e. all modules) translated to https://docs.apiwise.nl/ocm/index.html

There was a discussion about priorities, what should be covered by the new specs and what not. The functional modules are:

1. sharing and federated sharing (this is the CORE)
2. synchronisation and accessing the file using WebDAV or other file transfer protocol (this could be only a starting point for discussion)
3. service/user discovery (next thing)
4. what to do with the files/folders after sharing... (outside of scope)

Third party view of APIWise would be appreciated on the overall architecture.

A lot of historical decisions had been made that can be revised by APIWise. The sharing part doesn't have to comply with the old Open Collaboration Services specification.

How security works: This must be included in the core stuff. How to trust the other services. Grant and revoke privileges to access the API by other partners. This will be rolled out in next OC/NC versions.

Some vendors don't use WebDAV. Pydio doesn't like WebDAV that is old schools, however supports it. Use it because all the other solutions are proprietary but think about new ways.

Based on HTTP and stateless (rest) protocols is good if only domains are talking but if others (e.g. mobile) want to build apps on top of the API that's a different story.

APIWise can suggest on the new file transfer protocol part as a separate module exchanging WebDAV.

The guiding rule for now should be to make it easy as possible to be implemented by other sync&share vendors.

Lookup is an other big topic. This could be the focus after file transfer. Exchanging address books and listing remote servers will be implemented in OC/NC 9.0.

We have to make sure that the API is versioned!

Stick to WebDAV for now, but there are several interpretations of the specific parts. It is not just so easy to rely on WebDAV server and expect it to work. Special attributes ITags fileIDs, etc. are optionally added in OC. Pydio doesn't have them.

Don't try perfect in one step. Staging, modules, versioning.

Document the interaction between remote servers. It is not documented yet. Pydio had identified specification holes. Need to be fixed.

SUMMARY

1. Core speccs: federating sharing and sharing API. Modernize it in a vendor neutral way. Don't stick to legacy. This must be completed first. Must be done by the CS3 workshop (preferably mid December). APIwise has some ideas on how to modernize the federated sharing module to be shared in a written document.

2. Think about file transferring. Won't be completed but start thinking about it. There's less freedom here. Start with WebDAV and the missing documentation. Looking into extensions that OC implemented that are optional to others. By the CS3 Workshop we should at least have a good understanding.

Next meeting in 3-4 weeks.

16 December 2016

1) Open issues and clarifications

We discussed 3 open issues where we needed agreements in order to finalize the work. The full list of issues are at
https://github.com/GEANT/OCM-API/issues

- Issue #24 and #25 Sharing proposal: federated sharing w/o acceptance from consumer

We agreed that notification about the acceptance of a share (before access) SHOULD be done, but it's not mandatory by the protocol specifications. It is not functional part of the access flow but SHOULD be implemented as a separate "friendly" notification flow at the application layer.

- Issue #23 Security proposal: trusted services and shared secrets / request signing

We agreed that creating a trusted circle of services is recommended but not functional part of the protocol. The Authentication of the service that offers a share MUST be part of the protocol but the Authorization mechanism is out of scope. The implementation of a potential "whitelisting" authorization mechanism is up to the actual service owner.

- Issue #26 How does a provider know where to send the invitation to?

We agreed that service domain of a particular users is an a-priori knowledge. It is up to the actual implementation whether the user offering the share must specify the service domain explicitly or it can be automatically looked up e.g, via an IdP as an attribute of the user.

Organise a call with the GÉANT AAI experts to understand what user attributes can be requested from the IdPs via federations and eduGAIN.

- Discussion about the actual share.

We agreed that currently the share is a WebDAV target (URL of the resource, plus access token, permissions and other metadata). Agreed to include an attribute that can only be WebDAV today but allows the possibility to negotiate on other protocols in the future.

2) Roadmap

- Results considered to be final will be distributed by APIWISE on 16 January 2017.

- Partners can start reviewing, commenting and discussing via email/ticket and at the CS3 Workshop on 1 February 2017.

- Based on the comments, if any, results must be finalized and made available by APIWISE on 10 February 2017.

- Contract closed.

3) CS3 Workshop

The OCM session is scheduled on 1 February 2017
between 11.00 - 13.00 CET. Will be chaired by Peter Szegedi, GÉANT. This is to present the results of the OCM work to the broader audience including vendors.

4) Next call

If needed, we can organise a call on the week of 16 January TBC.

GitHUB

https://github.com/GEANT/OCM-API

API reference documentation

https://rawgit.com/GEANT/OCM-API/v1/docs.html

Final results to be submitted by APIWISE.

Close of contract. PAID!

Initiation

Pick up the results on Phase I.

See section 4.2 Deliverable of Phase I. below...

Create the new structure of two WGs:

1. Strategic WG ocm-admin@lists.geant.org
2. Technical WG ocm-tech@lists.geant.org

Assign mailing lists above

All: ocm-all@lists.geant.org

• [ACTION] Christian to send a questionnaire to all known ownCloud instance admins.
  
  
• [ACTION] Holger to prepare simple admin instructions on how to check usage and status.

• Repeat the Zurich demo but with more domains and more interesting use cases.
• Demonstrate the interest and if possible the active participation of vendors other than ownCloud Inc.

ownCloud v.8.2 or higher: Uni Münster, Uni Vienna, AARnet, SWITCH, CESNET, GWDG

Others very close: SURFnet, DVLA, CERN

Pydio v.6.4: ASNET-AM, University of Lausanne (TBC)

Locate other vendors and identify user cases, universities with two products and cross-sharing needs.

18-02-2015

News from Christian:

Pydio will most likely join OCM. They are actually aiming for test implementation by end of March.

16-03-2016

Charles du Jeu and David Gillard from Pydio (https://pydio.com/) joined the OCM project.

Charles, the CEO/CTO of Pydio, confirmed that that are preparing the release of an important dev version, that actually contains the Federated Sharing API implementation. See https://pydio.com/en/community/releases/pydio-core/pydio-core-631-development-release for more info. Pydio is making its best to have this release transformed to a stable one by the end of this month.

Press release

https://pydio.com/en/blog/federated-sharing-pydio-connects-owncloud-and-joins-opencloudmesh-initiative

16-05-2016

Russell Albert from Zettabox signaled his interest in OCM. Discussion is on-going...

10-06-2016

Christian Sprajc from PowerFolder signaled his interest in the OCM project and reported this

PowerFolder Federated Clouds feature now avnow availableailable: Testers needed!

15-06-2016

Frank Karlitschek from Nextcloud (ownCloud fork) joined the OCM project.

Christian Schmitz

Peter Szegedi

Charles du Jeu

David Gillard

Frank and Lukas

Multi-vendor OCM validation by the Community

ownCloud - Pydio

1) AARNet (Guido) initiated some initial testing with ASNET-AM (Hrachya)

AARNet (Australia) uses ownCloud and ASNET-AM (Armenia) uses Pydio.

• Some initial issues out of the box. Pydio developers are involved...
• Results to be seen...

2) More Pydio users and test scenarios to be defined (see 3.1)

• RENATER and University of Lausanne to be contacted...

Guido Aben

Hrachya Astsatryan

Andranik Hayrapetyan

Wahi

Christian Schmitz

Guido Aben

Charles du Jeu

This vision is that the OCM spec should be:

• compliant: with standard practices of the http world (error codes, conventions,…)
• described: using industry-strength documentation/testing system (e.g. swagger.io)
• neutral: should not have any artifacts or assumptions stemming directly from particular implementation or implementation language
• modular: allow providers to implement minimal functionality and add optional components of the spec as they please (or not)
• minimal: offload as much as possible of additional functionality to existing mechanisms in the network, especially for optional modules (e.g. lookup)
• secure: compliant with modern security frameworks (e.g. OAuth2, JWT, …) For the modules, I would consider at first: - auth/autz negotiation - sharing of files - synchronization of files - user discovery (optional)
• robust: implementations should continue to deliver their  service even when interacting with a failed implementation/service or malicious intended attempts at federation as attack vector

PREFERRED MODULES:

• auth/autz negotiation
• sharing of files
• synchronization of files
• user discovery (optional)

Latest OCM protocol implementation on GitHub: https://github.com/cernbox/OpenCloudMeshSpecification

News item about the OCM demo at TNC'16:

http://www.geant.org/News_and_Events/Pages/Worlds-first-multi-vendor-server-to-server-cloud-sharing-interoperability-demonstration-powered-by-OpenCloudMesh-.aspx

Next steps:

• We'll find a professional protocol designer who will describe OCM in a more formalized way (using Swagger Framework or similar). 
• We'll establish a reference infrastructure and test environment where the implementations can be validated against. 

Start collecting organizations and people interested in joining the initiative.

Peter Szegedi

Christian Schmitz

ownCloud to release the first version of the API code and documentation.

DELAYED

Code v.0.002 has been released on 27 July 2015 by ownCloud Inc.

Comments have been provided by CERN.

VC for coordination on 24 August 2015.

• Concluded in the Communique GSec(15)015.

Pre-launch meeting organized by ownCloud on 28 August 2015 in Berlin. Peter (GÉANT), Guido (AARnet) and Kuba (CERN) and others.

• Code v.0.002 released and commented
• Christian as an interim leader
• GÉANT to provide the project framework
  • Find the neutral project lead, co-chairs from the community.
  • Approach other vendors: PowerFolder, W3C, Pydio, Cozy, fixithere
    
  • Put it in the GÉANT procurement requirements (the support for the API)

NIF/PID to be submitted and mailing list migration to be done.

Kick-off meeting: 22 October, 2015 in Vienna, Austria

The slides of the event can be found here.
password edu221015

Peter Szegedi

Christian Schmitz

DRAFT protocol definition v.0.004 released

OpenCloudMesh = ownCloudMesh

Holger Angenent

Andreas Wilmer

Ron Trompert

David Jericho

Guido Aben

Christian Kracher

Simon Leinen

19 January 2016

Jakub Moscicki

Simon Leinen

All

Open API v.1.0 documented and released at least in BETA version with the intention to come up v.2.0 vendor agnostic version (IETF WG)

Peter Szegedi

Christian Schmitz

Kuba Moscicki

Simon Leinen

Set of recommendations:

Integration with Macaroons

• authz/authn: Google macaroons (http://research.google.com/pubs/pub41892.html)

Cloud user lookup service

• Need @ a @ simple @ username @ structure
• endpoint discovery: DNS

David:

• Rather than trying to implement a meta-data or negotiation server, why don't we just use the NAPTR/SRV DNS RR method I proposed late last year? Multiple records, one for each protocol, makes discovery and announcement very easy, and allows the sites to use different endpoints for different protocols as suited.
  Something akin to:

  _standard._opencloudmesh._tcp.cloudstor.aarnet.edu.au. SRV 0 0 5009 ocm.cloudstor.aarnet.edu.au.

  _quic._opencloudmesh._udp.cloudstor.aarnet.edu.au.     SRV 0 0 5009 quic.cloudstor.aarnet.edu.au.

  _ftp._opencloudmesh._tcp.cloudstor.aarnet.edu.au.      SRV 0 0 5011 ocm.cloudstor.aarnet.edu.au.

  ...and so on. Discovery and compatibility is known by simple DNS query, and dynamically configurable using existing standards and tools. Supporting new protocols becomes a server side issue, rather than having to make any additional software aware.

Protocol negotiation handshake

Holger:

• What would you think of a signaling mechanism, comparable to the handshake when handling the method for an encrypted connection? We think of something like: Server A asks server B for a connection and offers WebDAV and CMIS and asks server B which of these it likes. Server B speaks only WebDAV so they agree to use this. Of course it is clear that now it could happen that both servers do not offer a matching method. But the acceptance threshold to participate at OCM could be lowered since only the signaling would be mandatory.

David:

• At least if the mandate was WebDAV for the most basic method, that would be the lowest barrier to entry. WebDAV has been around since 1996, and it really doesn't extend existing HTTP that far, there's no real excuse for not implementing it. A technical person is able to talk to a service using telnet/netcat/socat and do sharing functions manually, as well as making it simple to do these things programmatically using curl. Metadata against the query is also trivial, because HTTP/(1.1|2) support headers that can be used to carry all the relevant information.

• Caching should be a secondary consideration, because it's a non-trivial activity to get right (consider HTTP caching and HTTP 304 return codes). Our primary concern is fast transfer, reliable transfer, and having people join in. In the case of many small files, many alternate protocols allow streaming, batching or the like, akin to the old days of asking a ftp server for a tar of a directory.

Security and Trust

• Trusted curcle of servers vs. open internet policy
• Verification of architectures and deployments up to a community standard. DeiC's Smashbox idea
• More admin features are in 9.0 also better addressing and web popup. Shibboteh ticket.

• pass-through: data is pumped from server B to the client A via server A (as currently available)
• server-cache: as mentioned by Holger (comes with potential consistency&refresh mechanism issues to be addressed)
• client redirect: as mentioned by Paul Millar (would affect the current usability model of ownCloud)

Pick a preferred home for standardization

• IETF??? - Simon to give a thought

All the recommendations have been fed into the Phase II initiation phase.

The OCM team agreed to continue and split into two working groups (WGs):

1. Strategic policy and standardization oversee WG
2. Technical protocol definition and implementation WG

The key objective of the Strategic WG will be to reach the ultimate "Open Cloud Mesh standard" and to oversee (not overdrive) the Technical WG making sure that the open API design principles are properly addressed. The key objective of the Technical WG will be to deliver one or more working prototypes of OCM and to provide (not force) input to the Strategic WG making sure that their assumptions are realistic.

The detailed objectives and key results of the Phase II will be determined in the initiation phase.

Phase I is considered to be CLOSED.

News item: http://www.geant.org/News_and_Events/Pages/OpenCloudMesh.aspx