doc(upgrade): add known issue for v1.4.2 to v1.5.0 upgrade path

The container images needed for the upgrade log infra are not included
in the shipped ISO image so it's impossible for air-gapped clusters to
conduct the upgrade. Providing several workarounds to make the upgrade
proceed.

Signed-off-by: Zespre Chang <[email protected]>
Co-authored-by: Adam Toy <[email protected]>
Co-authored-by: Jillian Maroket <[email protected]>

Author: 3 people

docs/upgrade/v1-4-2-to-v1-5-0.md

@@ -45,3 +45,87 @@ The correct status is displayed after the upgrade to v1.5.0 is completed.
 
 Related issues:
   - [[BUG] Harvester cluster status keep in NotReady on the second joined node from iso installation](https://github.com/harvester/harvester/issues/7963)
+
+### 2. Air-gapped upgrade stuck with `ImagePullBackOff` error in Fluentd and Fluent Bit pods
+
+The upgrade may become stuck at the very beginning of the process, as indicated by 0% progress and items marked **Pending** in the **Upgrade** dialog of the Harvester UI.
+
+![](/img/v1.5/upgrade/upgrade-dialog-with-empty-status.png)
+
+Specifically, Fluentd and Fluent Bit pods may become stuck in the `ImagePullBackOff` status. To check the status of the pods, run the following commands:
+
+```bash
+$ kubectl -n harvester-system get upgrades -l harvesterhci.io/latestUpgrade=true
+NAME                 AGE
+hvst-upgrade-x2hz8   7m14s
+
+$ kubectl -n harvester-system get upgradelogs -l harvesterhci.io/upgrade=hvst-upgrade-x2hz8
+NAME                            UPGRADE
+hvst-upgrade-x2hz8-upgradelog   hvst-upgrade-x2hz8
+
+$ kubectl -n harvester-system get pods -l harvesterhci.io/upgradeLog=hvst-upgrade-x2hz8-upgradelog
+NAME                                                        READY   STATUS             RESTARTS   AGE
+hvst-upgrade-x2hz8-upgradelog-downloader-6cdb864dd9-6bw98   1/1     Running            0          7m7s
+hvst-upgrade-x2hz8-upgradelog-infra-fluentbit-2nq7q         0/1     ImagePullBackOff   0          7m42s
+hvst-upgrade-x2hz8-upgradelog-infra-fluentbit-697wf         0/1     ImagePullBackOff   0          7m42s
+hvst-upgrade-x2hz8-upgradelog-infra-fluentbit-kd8kl         0/1     ImagePullBackOff   0          7m42s
+hvst-upgrade-x2hz8-upgradelog-infra-fluentd-0               0/2     ImagePullBackOff   0          7m42s
+```
+
+This occurs because the following container images are neither preloaded in the cluster nodes nor pulled from the internet:
+
+- `ghcr.io/kube-logging/fluentd:v1.15-ruby3`
+- `ghcr.io/kube-logging/config-reloader:v0.0.5`
+- `fluent/fluent-bit:2.1.8`
+
+To fix the issue, perform any of the following actions:
+
+- Update the Logging CR to use the images that are already preloaded in the cluster nodes. To do this, run the following commands against the cluster:
+
+  ```bash
+  # Get the Logging CR names
+  OPERATOR_LOGGING_NAME=$(kubectl get logging -l app.kubernetes.io/name=rancher-logging -o jsonpath="{.items[0].metadata.name}")
+  INFRA_LOGGING_NAME=$(kubectl get logging -l harvesterhci.io/upgradeLogComponent=infra -o jsonpath="{.items[0].metadata.name}")
+
+  # Gather image info from operator's Logging CR
+  FLUENTD_IMAGE_REPO=$(kubectl get logging $OPERATOR_LOGGING_NAME -o jsonpath="{.spec.fluentd.image.repository}")
+  FLUENTD_IMAGE_TAG=$(kubectl get logging $OPERATOR_LOGGING_NAME -o jsonpath="{.spec.fluentd.image.tag}")
+
+  FLUENTBIT_IMAGE_REPO=$(kubectl get logging $OPERATOR_LOGGING_NAME -o jsonpath="{.spec.fluentbit.image.repository}")
+  FLUENTBIT_IMAGE_TAG=$(kubectl get logging $OPERATOR_LOGGING_NAME -o jsonpath="{.spec.fluentbit.image.tag}")
+
+  CONFIG_RELOADER_IMAGE_REPO=$(kubectl get logging $OPERATOR_LOGGING_NAME -o jsonpath="{.spec.fluentd.configReloaderImage.repository}")
+  CONFIG_RELOADER_IMAGE_TAG=$(kubectl get logging $OPERATOR_LOGGING_NAME -o jsonpath="{.spec.fluentd.configReloaderImage.tag}")
+
+  # Patch the Logging CR
+  kubectl patch logging $INFRA_LOGGING_NAME --type=json -p="[{\"op\":\"replace\",\"path\":\"/spec/fluentbit/image\",\"value\":{\"repository\":\"$FLUENTBIT_IMAGE_REPO\",\"tag\":\"$FLUENTBIT_IMAGE_TAG\"}}]"
+  kubectl patch logging $INFRA_LOGGING_NAME --type=json -p="[{\"op\":\"replace\",\"path\":\"/spec/fluentd/image\",\"value\":{\"repository\":\"$FLUENTD_IMAGE_REPO\",\"tag\":\"$FLUENTD_IMAGE_TAG\"}}]"
+  kubectl patch logging $INFRA_LOGGING_NAME --type=json -p="[{\"op\":\"replace\",\"path\":\"/spec/fluentd/configReloaderImage\",\"value\":{\"repository\":\"$CONFIG_RELOADER_IMAGE_REPO\",\"tag\":\"$CONFIG_RELOADER_IMAGE_TAG\"}}]"
+  ```
+
+  The status of the Fluentd and Fluent Bit pods should change to `Running` and the upgrade process should continue after the Logging CR is updated.
+
+- On a computer with internet access, pull the required container images and then export them to a TAR file. Next, transfer the TAR file to the cluster nodes and then import the images by running the following comands on each node:
+
+  ```bash
+  # Pull down the three container images
+  docker pull ghcr.io/kube-logging/fluentd:v1.15-ruby3
+  docker pull ghcr.io/kube-logging/config-reloader:v0.0.5
+  docker pull fluent/fluent-bit:2.1.8
+
+  # Export the images to a tar file
+  docker save \
+    ghcr.io/kube-logging/fluentd:v1.15-ruby3 \
+    ghcr.io/kube-logging/config-reloader:v0.0.5 \
+    fluent/fluent-bit:2.1.8 > upgradelog-images.tar
+
+  # After transferring the tar file to the cluster nodes, import the images (need to be run on each node)
+  ctr -n k8s.io images import upgradelog-images.tar
+  ```
+
+  The upgrade process should continue after the images are preloaded.
+
+- (Not recommended) Restart the upgrade process with logging disabled. Ensure that the **Enable Logging** checkbox in the **Upgrade** dialog is not selected.
+
+Related issues:
+- [[BUG] AirGap Upgrades Seem Blocked with Fluentbit/FluentD](https://github.com/harvester/harvester/issues/7955)