The Smart Albums feature is implemented as it is because it provides instant results. This makes it very useful and convenient.
Searching against the filesystem would be slow, and the resulting delay when clicking on a Smart Album could confuse some users.
Moreover, if the search against the filesystem was also constructed to use exiftool fields, then the search would be even slower, as all the files in the filesystem target area would additionally have to be scanned by an exiftool process. So that would be even worse.
I understand the need to be able to search the file system from time to time. But in the Search Dialog, saved search presets are provided for exactly this purpose. Admittedly it is an extra couple of clicks compared to saving it as a smart album. But it does not take long compared to the time a metadata search of the filesystem will take.
If repetitive searches for the same data (which is what a smart album does, by definition) are to be frequently performed, leveraging the catalog is to be encouraged.