since 1.8 automating full install/configuration of harbor is harder

[paulczar]

Before Harbor 1.8 I could configure Harbor to use external Auth via config file or environment variable, all automation tooling know how to do both of those things. whether its chef or puppet, or something like confd pulling from consul. Even better languages like spring have integration to use kubeconfig maps as config sources.

These are all things we've learned to treat as code, and as devops/sre/whatever we have built up strong muscles around managing, testing, securing, etc, treating config changes as "code changes".

With the new configuration API however we cannot do any of those things, and instead of tried and true trusted ways to configure our app, the developers are forcing us to use an API without a decent client tool. now instead of the simple task of "write file -> restart service" we have to rewrite automation tooling to take our list of preferred settings and sling them into the API via curl or similar.

What's even more painful is that some settings are still set via environment variable (such as UAA_CA_ROOT) so its not even completely consistent.

Proposal:

1. We bring configuration via ENV or config file back and we mark anything set in them as read-only in the API/UI.
   or
2. We write a config cli* and people can use that, as its less error prone than just running curl -X POST a bunch of times.

• a config cli is how spinnaker's halyard works, you start halyard, run a bunch of hal commands to configure it and then tell it to deploy/apply and it then deploys all of the spinnaker microservices with your settings. It's slightly different in that its actually managing the lifecycle of the components via halyard vs via kube manifests directly, but the idea is there.

[paulczar]

One more thing ... When setting the config settings via the API, the settings can later still be overwritten by the Web Console ... so we still suffer from the same problem of config drift as "the configuration in my app does not match the configuration in the automation", it's just this time a user setting via the app wins over the user setting via the automation.

I know a lot of folks use automation ( and the frequency of rerunning it every say 30 seconds ) to re-mediate changes when somebody changes the live config to be different from the expected config.

It's a bit like Kubernetes itself ... If you set something in a deployment and then later somebody modifies a pod owned by that deployment ... when it comes to reconcile that .. the deployment being the owner of the podTemplate will win. This should be the same case for configuring an application.

[paulczar]

Essentially configuration should have a single source of truth, and in this modern age of devops/ci-cd that sort of truth should be an artifact that can be traced and audited. Something like one of the following:

1. config-management - chef/puppet/helm code + declarative state - both stored in github, both tested on change, and ideally applied on by ci/cd tool. changes auditable via github and/or ci/cd tooling.

2. centralized config service - consul, etcd, etc. Usually requires a client app that takes values and writes them to a template/config file, although some apps can request config directly from them. the centralized config service should track changes and be auditable.

3. cowboys sharing an admin password clicking around in the webui changing settings at will.

One of the above is an anti-pattern ...

[xaleeks]

@paulczar Hey paul, you raise some good points and we are evaluating some of the options you mentioned but I want to give some color to everyone on why we made those changes, there's quite an audience here it seems :) Prior to 1.8, everything was stuffed into the config file and it was growing out of control, and opening a floodgate in terms of user behavior of just modifying a single file for harbor admins. Just mod it, copy it over, and deploy, but you can see how that can become problematic. Some of those same params in the cfg which were editable in the web console only persists through the DB but won't backdate the config file obviously, leading to stale config files. The decision to create the yml as it is currently was really to create a clearer seperation between system variables and user/project level settings so it got gutted essentially, but creating two complementary subsets of a previous superset. So there shoudn't be any param that can be edited via different paths basically, we were really striving for the single version of truth you mentioned. User settings should be set via API/UI, it makes no sense to leave these in the cfg. The UAA_CA_ROOT is indeed an an oversight, but I think you might have found the only guy in the intersection, thanks for catching that :)

The current setup using API does require curling, and it can be (a bit?) more painful and the harbor instances have to be up and running already. We have explored a cli option previously like the one you mentioned as replacement but advantages over curling did not tip it in its favor compared to more urgent features. Can you elaborate on specific automation tooling you have in mind here, maybe share some specific use cases? ie. I'm guessing users like the old setup because they can auto deploy multiple harbor instances via a single config file by just changing the hostname or something to that effect, without having to tinker with automating APIs or hitting buttons on the UI.

"the developers are forcing us to use an API without a decent client tool." Yep, good client tooling is indeed what we are after. But I don't think throwing everything in the yaml is what we want.

[paulczar]

Thanks for the response! I still believe that having a configuration API for setting underlying config settings is an antipattern. If you look at similar software, you set all of this stuff up via config files or env variables:

examples:

• Grafana - https://grafana.com/docs/auth/github/
• Concourse - https://concourse-ci.org/cf-uaa-auth.html

[voor]

Just want to add some emphasis to this one:

I know a lot of folks use automation ( and the frequency of rerunning it every say 30 seconds ) to re-mediate changes when somebody changes the live config to be different from the expected config.

This is particularly important in highly regulated environments that require frequent audits and compliance to know that settings are correctly configured. We need to be able to track who makes config changes and when those config changes occurred, and know that the last set configuration is what is still live in the environment. This is extremely easy to do when the configuration lives in a single source of truth (a git repository) where we have a lot of mechanisms to handle everything mentioned -- pull requests, signed commits, and RBAC on accessing the repositories. Industry is leaning heavily towards that declarative format with prevention of drift.