Hi Steve,
I use my jointer fairly often so this wouldn't work for me but if you don't plan on using it much I don't see a problem with your plan as long as you check for square each time you reattach the fence.
I am in the process of building a miter saw station along a 16' wall in my garage shop and intentionally made the work surface high, about 42" to the top of the saw table. I'm about 6' 1" and as I age my back bothers me if I bend over too much so the height should work well for me and I can always shorten it if necessary since the cabinets are sitting on 2x bases. I've included cabinets with drawers and shelves for cutoffs under the work surface but I've seen other arrangements where folks store lumber on racks underneath or have rolling work stations that are parked there when not in use. In my case, I just needed the storage and work surface area more than the floor space.
HTH
Chris