# perfdata-fix - check and repair of the VM metrics collector on a vStack installation

One script, three modes. It first checks the state, then ( only on an explicit
command ) applies exactly the fixes whose conditions are met, and on revert
restores everything as it was.

## Affects

- vStackOS 3.x installations: VM metrics collector
  `/usr/local/sbin/vstack-vm-perfdata` ( package vStack-vm-perfdata )
- nodes running the Perl collector `/usr/local/bin/vStack-vm-perfdata-cpu-feeder`:
  the 2.x line and nodes upgraded with `pkg upgrade` without replacing the
  collector
- not applicable when there is no collector or it already carries the fix -
  the script detects this and does nothing

## Problem

Every 10 seconds the metrics collector reads the full vCPU state of every VM
( `bhyvectl --get-all` ), although it uses only the vCPU count and memory
figures. Each such read stops the guest vCPU; one cycle makes about a hundred
of them, and the guest stalls for up to hundreds of milliseconds. What the
partner sees: Linux guests log `clocksource: Marking clocksource 'tsc' as
unstable` and switch to the HPET clock ( slower timekeeping in guests, higher
load ), and any guest briefly freezes every 10 seconds.

## What it fixes

- cxx-collector - in the collector binary the `bhyvectl --get-all` call is
  replaced with a call to the wrapper `/usr/local/sbin/vpd-bhyvectl`, which
  reads only statistics and topology ( `bhyvectl --get-stats --get-cpu-topology` ).
  These reads do not stop the vCPU. Only the command string changes, 36 bytes
  of the same length; before writing, the script verifies the bytes in place
  and checks the patched copy
- perl-feeder - the same replacement in the Perl collector source
- restart-service - restarts the collector service if it was running

Metrics do not change: the same series with the same values. Neither VMs nor
the node are rebooted.

## How to verify the signature and run

As root on every node of the cluster:

```
cd /tmp
fetch https://repo.vstack.com/hotfixes/perfdata-fix/perfdata-fix.sh
fetch https://repo.vstack.com/hotfixes/perfdata-fix/perfdata-fix.sh.sig
openssl dgst -sha256 -verify /usr/share/keys/vstack/vstackos.pub \
  -signature perfdata-fix.sh.sig perfdata-fix.sh
```

Wait for `Verified OK` ( otherwise the file is damaged or tampered with - do
not run it, contact support ), then:

```
sh perfdata-fix.sh          # check mode, report only, changes nothing
sh perfdata-fix.sh apply    # apply the fixes found
sh perfdata-fix.sh revert   # restore the original state
```

check and apply spend about 25 seconds measuring vCPU stops.

## Reading the report

```
== perfdata-fix on node01, mode: apply ==
os release:         14.2-RELEASE
C++ collector:      needs-fix, running: yes
Perl CPU feeder:    absent, running: no
verdict: fixes needed: cxx-collector
applied: cxx-collector, backup: /usr/local/sbin/vstack-vm-perfdata.orig
service vstack_vm_perfdata:  restarted, running
measuring vCPU0 freezes on 5 VM(s) for 25 s ...
vCPU freezes:       0 in 25 s
result: OK
```

`vCPU freezes` - how many times vCPU0 of the first five VMs on the node was
stopped during the measurement ( counter `number of times idle requested at
exit` ). Before the fix - hundreds or more, after - 0.

Collector state: `needs-fix` - the fix is needed, `applied` - the fix is
already in place, `not-affected` - the collector does not have the bug,
`absent` - not installed.

Verdicts:
- `everything is in order, nothing to do` - no fixes needed, nothing changed
- `fixes needed: ...` - list of required fixes; check mode changes nothing,
  apply mode reports each fix with an `applied: ...` line
- `fixes are not applicable` - the collector is not installed, nothing changed

Exit codes: 0 - fine ( nothing to fix / not applicable / fixes applied ),
3 - check found something to fix, 2 - the collector is fixed but vCPUs are
still being stopped, 1 - error or failed apply ( rolled back ).

## Rollback

Before patching, the original file is saved next to it with the `.orig`
suffix. If the service does not come up after the patch - automatic rollback:
the files are restored from `.orig`, the wrapper is removed, the service is
restarted, exit code 1.

`sh perfdata-fix.sh revert` restores the files from `.orig` and removes the
wrapper.

On failure send support: the full script output and the output of
`pkg query '%n-%v' | grep -i perfdata`.

## Known limitations

- A Linux guest that has already switched from TSC to HPET does not switch
  back by itself: TSC returns after the guest reboots. No new switches occur
  after the fix
- `pkg check -s` reports a checksum mismatch for the collector binary - that is
  the fix itself. Reinstalling the same package version restores the original
  binary: run the script again. After upgrading to a version that carries the
  fix, check reports `not-affected`
- exit code 2: something other than the collector stops vCPUs - send the
  output to support