Since Domino 8.5 there have been many files in the data/domino directory on the server. So every time you refresh the files tab you have to wait while the server scans the whole data/domino directory. Although some contents like domino/js is not scanned this can take some time. Fortunatly there is a notes.ini setting which disable file scans in the domino directory.
To enable this feature you cann add ADMIN_CLIENT_SKIP_DOMINO=1 to the notes.ini of the server or you can set this value with SET CONFIGURATION ADMIN_CLIENT_SKIP_DOMINO=1 on the console.
I have tried this on my development server. Without ADMIN_CLIENT_SKIP_DOMINO=1 the file enumeration runs 4 seconds and with this setting in the notes.ini of the server the enumeration has finished immediatly after pressing the F9 key.