ut_filename_hash(): Add better casts to please the compiler:

warning C4307: '*': integral constant overflow

This regression was introduced in
commit dd77f072 (MDEV-22841).