The OnlyTickMontagesAndRefreshBonesWhenPlayingMontages setting added in 5.4 is broken. It only works because it unintentionally forces a second call to UAnimInstance::UpdateAnimation (via RefreshBoneTransforms) which generates generates the montage evaluation data by performing a full update of the animation graph. If this second update doesn't happen, we don't have any montage evaluation data so when we call evaluate on the slot node in the anim graph, we don't generate the correct pose. This second update is also performed on the game thread so is particularly detrimental to any licensee that uses this setting.
There maybe other bugs hiding here since the desired codepath would be to only perform the montages in UAnimInstance::UpdateAnimation and not the anim graph. I'm unsure if the anim graph is then in a valid state to evaluate.