Rebinding an action mapping doesn't seem to take effect immediately if the Pawn class used has Pawn.h as its parent class. However this'll take effect immediately with DefaultPawn.h. In addition, the new key binding will take effect once the user presses the key that the action used to be mapped to, and then goes back to pressing the new key.
With Test Project:
The project has an action mapping called "Test Action", that prints "Test Action" to the screen when triggered. Pressing C and B will change the key that maps to this action between J and K respectively.
Note: You'll see 2 different behaviors, depending on whether the default Pawn class is "ExtendsFromDefaultPawn" vs "ExtendsFromPawn".
Result: Input will not be detected and test action will not print to screen, unless the J key is pressed first before the K key.
Expected: "Test Action to screen" will print to screen without a pause in input.
There's no existing public thread on this issue, so head over to Questions & Answers just mention UE-82242 in the post.