Consider creating a microservices key

[libremente]

Current situation

We are experiencing some issues with projects containing different repos inside.
Let's make an example.

PROJECT A
--> microservice 1
--> microservice 2
...
--> microservice n

Right now our solution is to tell PAs to create a new repo, let's call it metarepo, in order to store the information regarding the whole project. This metarepo will contain the publiccode.yml file which will be indexed in the catalog.
However, this solution has some flaws:
a) if the metarepo does not have a clear README file, the information regarding the other repositories (containing the n microservices) will be rather vague or lost. This is suboptimal.
b) the vitality index is calculated exclusively based upon the information regarding the metarepo and this does not make sense.

Proposal

We should add a key, called for example components or microservices, containing the list of repositories "related" to the project. As such, the metarepo will still contain the publiccode.yml file inside but each repo will be somehow listed in the catalog. This could also facilitate the calculation of the vitality index.

[bvhme]

Great proposal, I would like to say that the components seperately are 'codebases' or 'products' in their own right, so them having a `publiccode.yml` in the file seems right to me. Perhaps we can tell people that the metarepo, and especially the `README` in that can serve as an important entrypoint for the project and give them a reason to work on that. This readme could include things such as high level overview, the business cases etcetera.

…

On Tue, 18 Jun 2019 at 11:03, libremente ***@***.***> wrote: Current situation We are experiencing some issues with projects containing different repos inside. Let's make an example. PROJECT A --> microservice 1 --> microservice 2 ... --> microservice n Right now we are encouraging PAs to create a new repo, let's call it metarepo, in order to store the information regarding the whole project. This metarepo will contain the publiccode.yml file which will be indexed in the catalog. However, this solution has some flaws: a) if the metarepo does not have a clear README file, the information regarding the other repositories (containing the n microservices) will be rather vague or lost. This is suboptimal. b) the vitality index is calculated exclusively based upon the information regarding the metarepo and this does not make sense. Proposal We should add a key, called for example components or microservices containing the list of repositories "related" to the project. As such, the metarepo will still contain the publiccode.yml file inside but each repo will be somehow listed in the catalog. This could also facilitate the calculation of the vitality index. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/italia/publiccode.yml/issues/62?email_source=notifications&email_token=AAOUFPNRWKND36QCXNMNXWDP3CQGHA5CNFSM4HY523N2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4G2CSOUQ>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAOUFPJGAGLK2JW4WBIYNZLP3CQGHANCNFSM4HY523NQ> .

[libremente]

the components seperately are 'codebases' or 'products' in their own righ

@bvhme I understand your point of view and I believe this is true for well built standalone microservices or components which can be deployed "independently" from the other pieces. However, we are facing some products which have several not really independent pieces so I fear that inserting a publiccode.yml in each of those repo would generate confusion when browsing the catalog. I don't have a clear solution to this yet.

This readme could include things such as high level overview, the business cases etcetera.

Exactly, this is our view of a metarepo.

[Smart-City-Muenster]

Hi there from the city administration of Münster, Germany, and thanks for publiccode.yml!

We are starting to use the publiccode.yml for two current projects (and all other future projects, too, of course) and are facing this exact problem.
One project has a frontend and a backend in two repositories. They are not to be used indepently, though.
It's similar with a different project that splits up to three repositories: source code, documentation to rebuild hardware and design files.

What to do with these project structures? One publiccode.yml per project in the "main" repo (whatever this is)? The same publiccode.yml in every repo?

(Referencing the corresponding issues: bCyberGmbH/leezenflow-doku#2 , bCyberGmbH/leezenflow-design#1 , reedu-reengineering-education/smart-city-dashboard-backend#4 )

[bzg]

Hi all, this is also an issue I expect we will be facing with many source code from the French public sector.

I am in favor of the metarepo approach.

I'm tempted to use dependsOn to list the repositories that contribute to the metarepo, but I'm not sure entries in dependsOn can have a URL - if not, that's certainly a problem. (Also, this suggestion relies on my understanding of dependsOn as something different than the list of libraries one can find in e.g. package.json. I don't think adding the list of dependencies on publiccode.yml is a good idea. But this example suggests that my interpretation is okay.)

Another (complementary?) suggestion is to add a key to declare that a publiccode.yml belongs to a metarepo. Something like publiccodeYmlType: "meta" - but that name is not good enough IMHO.

I would refrain from adding a microservices key, as systems may have various architectures, this would lead to create more keys. If dependsOn cannot be used to declare contributing repos, I'd use something neutral like "components" - or anything that cannot be confused with dependencies/libraries/packages/etc.

[ruphy]

@Smart-City-Muenster it's a pleasure to have you here! I'm happy you're finding publiccode.yml useful!

I agree with your concerns and I also see what @bzg is saying. I think that having a components could indeed be a useful key. So far we've decided to either promote a repository as "main" repository, or simply create a new lightweight one which refers to the various components (see as an example https://github.com/italia-software/gitlab).

I agree that a machine-readable solution would be preferrable, and I love the name components for it! Would you like to open a Pull Request adding the key to the spec? We can then work on refining the details of it and get it approved

[Smart-City-Muenster]

Would you like to open a Pull Request adding the key to the spec?

Yes! Due to private reasons (good ones!) my PR is a bit late, sorry for that.

[libremente]

I am getting back to this issue since I am experiencing a similar situation with several mono repos which share some complexity with the multi microservices architecture.

I have the feeling that in this thread we are discussing the simplest case where all the components of a project have the same licensing and authorship information. In fact, in such a case it is feasible to create a metarepo and store those common information in its publiccode.yml (or have a single publiccode.yml in the root of the monorepo).
However, things become a bit more complex when the components have different licensing/authorship info. In that case, what should you write in the metarepo's publiccode.yml?

I can quickly think of 5 scenarios but there are many more combinations possible:

0. Do nothing, which means adding just some context info in the longDescription field. It could be something very basic like:

longDescription: |
  This is a meta repo containing general information about $PROJECT.
  $PROJECT has a microservices architecture, composed by different repos with different licensing/authorship info.
  For more information regarding each of those components, please see:
  -  [microservice1](#) (MIT, Copyright X)
  -  [microservice2](#) (Apache-2.0, Copyright Y)
  -  [microservice3](#) (BSD-3-Clause, Copyright Z).

1. A quick&dirty solution could be to transform the single component entry into an array carrying some metadata relative to that component, such as name, link, license and authorship info. However this means creating a nested subset of keys which is a maintenance hell, I see it as a kind of antipattern so I think this should be avoided at all cost. Something like:

components:
  - name: microservice1
    url: #
    license: MIT
    mainCopyrightOwner: X
  - name: microservice2
    url: ##
    license: Apache-2.0
    mainCopyrightOwner: Y

2. A slightly more complex solution which means more work both on the "releaser" side and from the various catalogues' pov:
   a. create a metarepo with the relevant information regarding the $PROJECT. The license key here applies to the project as a whole.
   b. The components key lists the URLs for each component, acting as a manifest file (as proposed in issue add key components #169).
   c. Now I think that each of those components' URL should point to a repo that MUST contain a dedicated publiccode.yml (as @bvhme was mentioning previously). Also, that publiccode.yml should use some key, like for example applicationSuite to indicate their "parent" and have some sort of back link to $PROJECT. In this way we are indicating that each component exists as a standalone project with its own licensing and authorship information, however it is usually used as part of a larger $PROJECT (with the relevant license in that case). Probably in this case the applicationSuite key should become a URL itself but that's another issue. Of course this introduces extra efforts for the catalogues which will need to handle the relationship and show it somehow (see Consider creating a microservices key #62 (comment)).

3. Make a better use of the dependsOn/open key since it is kind of never used (at least from my previous experience in the Italian Catalogue). Of course some changes are needed in order to fit this case but it can be managed kind of easily.

4. Introduce a relationships key on top of dependsOn, it's kind of a wrapper around many needs. I am thinking at adding an extra layer that could work in this way:

relationships:
  - type: "dependsOn" # this type reflects the previous "dependsOn" key and allows the use of the complex "dependency" object
    name: PostgreSQL
    version: "3.2"
    optional: true
  - type: "component" # this is a new type that allows the use of a novel complex object, this is just an example taken from 1. above
    name: "Public Frontend"
    url: ##
    license: Apache-2.0
    mainCopyrightOwner: Y

of course this is the most disruptive one since we are introducing several breaking changes but, since we already introduced the use of custom/special data formats we could introduce yet another one and customize it as we wish (i.e., simple one: just the url, complex one: as shown in the example above).

Having said all the above, I will probably start to adopt solution 0 since it's the most straightforward and maintains the flat nature of publiccode.yml, however I am seeing this situation in more and more projects everyday so I think it could be nice to come up with an elegant solution someday soon. WDYT?