edit

Release Notes

Note

Cloud Foundry Container Runtime (CFCR) was formerly known as Kubo. Some CFCR assets still use the Kubo name.

v0.16.0

Download the release artifact.

Release Date: April 06, 2018

  • A new etcd bosh release is being used (cfcr-etcd-release), this is to enable deployments with multiple etcd nodes -- story.
  • Prevent unnecessary route creation in default kube-controller-manager config -- story.
  • System specs can be applied even when a cloud provider is not configured -- story.
  • Enable inter-container communication in flannel, while retaining the original source ip. -- story.
  • CFCR can be deployed to BOSH-lite -- story.

  • Fix: Made the node drain (even) more robust -- story.

Other minor changes

Component Versions

The following table lists the component versions for CFCR v0.16.0:


Component Version
Kubernetes 1.9.6
Flannel 0.10.0
ETCD 3.3.1
Docker 1.13.1
CNI 0.5.2
Stemcell 3541.10

Conformance Tests Results

v0.15.0

Download the release artifact.

Release Date: March 20, 2018

  • Removed edge case where timeouts during upgrades could lead to Etcd data loss -- story.
  • Added the ability to configure the HTTP(s) proxy to be used by the Kubernetes control plane -- story.
  • Made Kube-DNS use its own configuration so that the Kubelet configuration is not exposed in the Kube-DNS container -- story.
  • Fix: Allowed \ and | to be used in vCenter passwords -- cloudfoundry-incubator/kubo-release#180.
  • Fix: Made the node drain more robust -- story #155549518, story #156008895 and cloudfoundry-incubator/kubo-release#181.
  • Fix: Included a kube-proxy dependency (conntrack) to fix error logs -- story.
  • Fix: Can use kubectl top node against a CFCR cluster. Caveat: the --heapster-scheme='https' flag needs to be included -- story.
  • GCP: Stopped the Kube-Controller-Manager from creating unnecessary routes -- story.
    • This was causing high Google Cloud API usage.
  • AWS: Added the ability to provide AWS credentials in the BOSH manifest -- story.
    • Previously, AWS access could only be granted by setting the IAM profile in the cloud config. In cases of BOSH Directors that are used by multiple deployments, it is necessary to provide the credentials in the BOSH manifest.

Other minor changes

Component Versions

The following table lists the component versions for CFCR v0.15.0:


Component Version
Kubernetes 1.9.5
Flannel 0.10.0
ETCD 3.3.2
Docker 1.13.1
CNI 0.5.2
Stemcell 3541.9

Conformance Tests Results

v0.14.0

Release Date: February 20, 2018

  • Kubernetes v1.9.3 -- cloudfoundry-incubator/kubo-release#176.
  • Flannel v0.10.0 -- cloudfoundry-incubator/kubo-release#169.
  • BOSH DNS v0.2.0 -- cloudfoundry-incubator/kubo-deployment#261.
  • GOVC v0.16.0.
  • Golang v1.9.4.
  • BOSH Stemcell v3541.4.
  • CFCR can now be deployed on an environment paved by BBL -- story.
  • Exposed OpenID authentication properties -- cloudfoundry-incubator/kubo-release#101.
  • logging-level BOSH property can be used to control the logging level of kube-proxy -- cloudfoundry-incubator/kubo-release#163.
  • HTTP(s) Proxy BOSH properties will be used for Kubernetes interactions with the IaaS -- cloudfoundry-incubator/kubo-release#130.
  • Nodes can now be deployed across multiple AZs on GCP -- story.
  • Nodes get tagged appropriately by Kubernetes to ensure that workloads are properly spread across AZs.
  • System workloads are now applied as part of the apply-addons BOSH errand -- story.
  • System workloads have been a cause of many deployment issues.
  • Enabled the API server audit logs -- story.
  • Audit logs can be disabled if the kube-apiserver.enable_audit_logs BOSH property is set to false.
  • Disabled the read-only port in the Kubelet -- story.
  • Disabled cAdvisor in Kubelet -- story.
  • Disabled the security context manipulation when privileged containers are off -- story.
  • The API server will not try to fix malformed requests anymore for security reasons -- story.
  • The API Server will clean up terminated pods more often to avoid running out of disk space -- story.
  • The API server will unmount volumes of terminated pods for security reasons -- story.
  • Most BOSH jobs switched to use BPM -- story.
  • From the BPM readme: "[BPM] crucially provides a security barrier such that if one of the jobs on your machine is compromised then the incident is limited to just that job rather than all jobs on the same machine".
  • OpenStack: Exposed cloud-provider.openstack.ignore-volume-az BOSH property for the OpenStack Cloud Provider -- cloudfoundry-incubator/kubo-release#166.
  • OpenStack: Exposed region BOSH variable for the OpenStack Cloud Provider -- cloudfoundry-incubator/kubo-deployment#262.
  • Fix: UAA credentials and vCenter passwords are now redacted in BOSH logs -- story.
  • Fix: to ensure that workers will pick the correct node name during rolling upgrades -- cloudfoundry-incubator/kubo-release#170.
  • Fix: to ensure that nodes get properly drained before they stop, in order to minimize workload downtime during a rolling upgrade -- story.
  • vSphere Fix: vCenter password with special characters (&, #, etc) can now be used with CFCR without breaking the deployment -- story.
  • Experimental: An ops-file can now be used in conjunction to the kubo-deployment in order to experiment with the multi-master setup -- story.

Component Versions

The following table lists the component versions for CFCR v0.14.0:


Component Version
Kubernetes 1.9.3
Flannel 0.10.0
ETCD 3.2.14
Docker 1.13.1
CNI 0.5.2
Stemcell 3541.4

Conformance Tests Results

v0.13.0

Release Date: January 25, 2018

Download the release artifact.

  • Kubernetes 1.9.2 -- story and story.
  • Flannel 0.9.1 -- story.
  • RBAC as the default authorization mode.
  • Support for VM power-offs and restarts -- story.
    • Reliance on certain functionality provided by BOSH was causing restarting VMs to fail.
  • Secure communications between system specs (Dashboard, Heapster and InfluxDB) -- story and story.
  • Ability to configure the timeout for system specs -- story.
    • The BOSH property is kubernetes-system-specs.timeout-sec and is set to 20 minutes by default.
  • Ability to update addon specs without experiencing API downtime -- story.
  • Ability to get diagnostic information if a system pod fails to be applied -- story.
  • Ability to have the default storage class be used in PVCs that do not specify a storage class -- story.
  • Ability to rotate the Kubernetes API certificate -- story.
  • Ability to use the syslog addon in a CFCR deployment -- story.
  • Fix: to not print secrets in user-facing scripts -- story.
  • Fix to not have more than one nodes go down during an upgrade -- story.
  • vSphere Fix: to avoid a synchronization issue that was causing master to fail to start -- story.
  • OpenStack Fix to have CFCR properly configure Kubernetes in order to communicate securely (TLS) to OpenStack -- cloudfoundry-incubator/kubo-release#156.

Component Versions

The following table lists the component versions for CFCR v0.13.0:


Component Version
Kubernetes 1.9.2
Flannel 0.9.1
ETCD 3.2.14
Docker 1.13.1
CNI 0.5.2
Stemcell 3468.20

Conformance Tests Results

v0.12.0

Release Date: January 10, 2018

Download the release artifact.

  • Use Kubernetes 1.8.6 -- story.
  • Enable secure access to the Dashboard via a NodePort when using RBAC -- story.
  • Privileged container support is turned off by default. There is a new property named allow_privileged_containers in director.yml which can be used to enable the feature -- story.
  • also cloudfoundry-incubator/kubo-deployment#252 and cloudfoundry-incubator/kubo-release#153
  • Don't update the master node when scaling up workers -- story.
  • Switch to use cfcr.internal as a TLD instead of .kubo -- story.
  • Disable all profiling / tracing endpoints by default -- story.
  • Always validate ServiceAccount tokens exist in etcd as part of authentication -- story.
  • Stop the Kubernetes API Server from serving unsecured and unauthenticated access in localhost -- story.
  • Remove unnecessary flag from kube-proxy -- story.
  • Make applying addon specs not fail if the specs are empty -- story / cloudfoundry-incubator/kubo-release#150.
  • Bump system specs timeout to work with slower environments -- story.
  • Implement logic to never lose more than one worker nodes during update -- story.
  • Restrict the data directory permissions for etcd -- story.
  • Use SSL for etcd peer connections -- story.
  • We are currently running single-node etcd clusters so no peer connections are established. Nevertheless, etcd would listen for peer connections over plain HTTP.
  • Openstack: openstack_tenant is not required in the director.yml as it is obsolete for OpenStack Keystone v3. The property still exists as it is needed by OpenStack Keystone v2 -- story.
  • Openstack: VMs deleted from the IaaS in OpenStack do not appear as ghost nodes -- story.
  • The fix for this issue introduces new director.yml properties as the OpenStack K8s Cloud Provider needs to be configured:
    • auth_url, openstack_username, openstack_password, openstack_project_id, openstack_domain
  • Caveat: the BOSH director needs to have human_readable_vm_names set to false in order for the Kubelet to register with the API Server successfully. See K8s issue: kubernetes/kubernetes#57765.
  • Fix: deploy_k8s to fail when the addon specs are not successfully applied -- story.
  • Fix: regression in abac authorization mode -- story.
  • vSphere PR Support CFCR deployments on vSphere environments with multiple datacenters -- cloudfoundry-incubator/kubo-release#127 / cloudfoundry-incubator/kubo-release#148.

BOSH Release

Component Versions

The following table lists the component versions for CFCR v0.12.0:


Component Version
Kubernetes 1.8.6
Flannel 0.5.5
ETCD 3.2.10
Docker 1.13.1
CNI 0.5.2
Stemcell 3468.13

Conformance Tests Results

Download the conformance test results.

v0.11.0

Release Date: December 20, 2017. Download the release artifact.

  • Rename the CF Routing properties in director.yml to follow a consistent naming style. story.
    • [ACTION REQUIRED]: If you use CF Routing in CFCR, you must update your director.yml with the new CF Routing property names.
  • Implement ability to deploy CFCR on an existing BOSH Director and make the experience for BOSH-native users comparable to the one in cf-deployment, bosh-deployment, and concourse-deployment. PR.
    • [ACTION REQUIRED]: If you deploy CFCR by means of bosh deploy, you should re-examine the names of the manifest and the ops-files as various changes have been made.
  • vSphere: Support persistent workloads on vSphere environments that do not use Resource Pools. Resource Pools are optional in vSphere and CFCF already supports vSphere environments that make use of Resource Pools. story.
  • Looked into an issue with the kubernetes-system-spec post-start script. story / Github Issue.

Theme: Security

  • AWS: Reduce the permissions of the AWS IAM policies for the master and worker nodes so that if credentials leak from an AWS CFCR node, the permissions associated with it are minimal. story.
    • [ACTION REQUIRED]: If you deploy CFCR on AWS, you must re-run Terraform to update the IAM Policies.
  • Use stemcell version 3468.13. story.
  • Disallow anonymous requests to the API server. story.
  • Alter the permissions of files with sensitive information so that they cannot be read by non-root users. story.
  • Make ETCD only listen to TLS connections so that ETCD-bound traffic in the cluster cannot be sniffed. story.
  • Disallow exec and attach commands to privileged pods. story.
  • vSphere: Escape back-slashes for vSphere users in cloud config. PR.

Component Versions

The following table lists the component versions for CFCR v0.11.0:


Component Version
Kubernetes 1.8.4
Flannel 0.5.5
ETCD 3.2.10
Docker 1.13.1
CNI 0.5.2
Stemcell 3468.13

Conformance Tests Results

Download the conformance test results.

v0.10.0

Release Date: December 8, 2017. Download the release artifact.

  • New property addons_spec_path in director.yml. Operators can use this property to provide a K8s spec file that is applied to the cluster when it comes up. story.
  • New property worker_count in director.yml. Operators can use this property to configure the number of K8s workers. story.
  • Enabled the K8s aggregation layer to support API server extensions. story.
  • CFCR was tested to run with 20 workers and "chatty" workloads. story.
  • Exposed the K8s API connection properties via a BOSH link. story.
  • [ACTION REQUIRED]: The HAProxy (proxy) routing mode is not longer supported. story.
  • [ACTION REQUIRED] GCP: The service_account property in director.yml is no longer supported. It has been replaced by service_account_master and service_account_worker which can be used to reference GCP service accounts that are provided to master and worker VMs separately. story.
  • GCP: New properties service_key_master and service_key_worker in director.yml. Operators can use these properties to enable the K8s cloud provider to use a GCP service account without having to change the BOSH cloud config. story.
  • GCP: The GCP K8s Service Catalog was tested on CFCR. story.
  • GCP: Predefined a standard storage class to be applied when CFCR is deployed on GCP. It uses the gce-pd PV provisioner. story.

Community contributions

Component Versions

The following table lists the component versions for CFCR v0.10.0:


Component Version
Kubernetes 1.8.4
Flannel 0.5.5
ETCD 3.2.10
Docker 1.13.1
CNI 0.5.2
Stemcell 3468.5

Conformance Tests Results

Download the conformance test results.

v0.9.0

Release Date: November 22, 2017. Download the release artifact.

Features

  • CFCR has been added to the Certified Kubernetes Conformance Program. See the Conformance Test Results below.
  • The Docker BOSH release has been updated to v30.1.4.
  • The ETCD and master nodes are colocated on the same VM. Deployments of v0.9.0+ have 3 worker nodes and 1 master/ETCD node.
  • BOSH has been updated to v264.1.
  • The Kubernetes Dashboard is accessible with RBAC mode as cluster admin. The Dashboard needs to be exposed via kubectl proxy.

Bug Fixes

Component Versions

The following table lists the component versions for CFCR v0.9.0:


Component Version
Kubernetes 1.8.2
Flannel 0.5.5
ETCD 3.1.8
Docker 1.13.1
CNI 0.5.2
Stemcell 3445.11

Conformance Tests Results

Download the conformance test results.

v0.8.1

Release Date: November 10, 2017. Download the release artifact.

Features

  • Upgraded Kubernetes version to v1.8.2

Bug Fixes

  • Bug in authorization switch mechanism: all clusters were deployed in RBAC by default. New property in director.yml to set desired mode (ABAC|RBAC).

Component Versions

The following table lists the component versions for CFCR v0.8.1:


Component Version
Kubernetes 1.8.2
Flannel 0.5.5
ETCD 3.1.8
Docker 1.11.0
CNI 0.5.2
Stemcell 3445.11

v0.8.0

Release Date: November 3, 2017. Download the release artifact.

Features

  • Upgraded Kubernetes version to v1.8.1
  • Bosh DNS replaces Power DNS
  • Memory limit is configurable
  • Kubelet resource reservation flags exposed: kube-reserved, system-reserved, eviction-hard. See Kubernetes docs for more information.
  • Internal routing from workers to masters through BOSH DNS -- no need for HAProxy or LB to route cluster internal traffic
  • User can load balance traffic from external load balancers

Improvements

  • Removed worker_node_tag property to set worker tags automatically for GCP load balancers

Bug Fixes

Component Versions

The following table lists the component versions for CFCR v0.8.0:


Component Version
Kubernetes 1.8.1
Flannel 0.5.5
ETCD 3.1.8
Docker 1.11.0
CNI 0.5.2
Stemcell 3445.11

Upgrading from v0.7.0

Perform the following steps to upgrade an existing CFCR v0.7.0 cluster to v0.8.0:

  1. Clone the new version of kubo-deployment.
  2. Log in to the CredHub server on your BOSH Director with the CredHub CLI.
  3. Delete the current Kubernetes certificate from CredHub:

    $ credhub delete -n "${director_name}/${deployment_name}/tls-kubernetes"

  4. Verify that the appropriate stemcell is installed in BOSH. To view the uploaded stemcells, run the following command.

    $ bosh stemcells

    To upload a new stemcell, run bosh upload stemcell STEMCELL_URL.
  5. See Deploying Bosh Director for information on how to update the BOSH Director.
  6. See Deploying CFCR for information on how to upgrade your CFCR cluster.

v0.7.0

Release Date: September 7, 2017. Download the release artifact.

Features

  • kubo-release tarball bundled with kubo-deployment
  • The deploy-k8s script deploys local release by default
  • Cluster self-healing capabilities enables the recoveru of worker VMs
  • Support for persistent volumes in GCP, AWS, and vSphere
  • Improved documentation to install on GCP and AWS

Bug Fixes

  • Removed AWS-related tags for other platforms
  • Password issues in vSphere: GitHub issue #102