When in browser mode, double click/middle click/enter key on an image would take you to the viewer mode.
In viewer mode, double click/middle click/enter key would enter fullscreen.
In fullscreen, double click/middle click/enter key would return back to viewer mode.
Another user brought up the exact same suggestion but for XnView Classic all the way back in 2005 (not a typo), and I personally have actually wanted this as well for over a decade:
Switching Mode: Browser ▶ Viewer ◀ ▶ Fullscreen
Moderators: XnTriq, helmut, xnview