This TIL ties in with my post from yesterday on TIL: Multi-user Homebrew.
If you have multiple user accounts on your Mac there it is likely that they all have access to be able to unlock FileVault. As a general rule I would recommend only having a single non-admin account have the ability to unlock FileVault.
The following command will show you how to remove a named user from FileVault using their
sudo fdesetup remove -user <Username>
This unfortunately does not give any output, so you will need to check the users associated with the the volumes by using:
sudo fdesetup list
The above will return you an output like below: