You can't directly access the GPU hardware on modern systems (at least without a massive re-engeneering effort like the Asahi Linux GPU driver for Apple hardware), so any 'minimal graphics API' still goes through a massive amount of wrapper code in the operating system and GPU driver.
Also, on older home computer systems the hardware was essentially the rendering- and audio-engine. Those systems were designed from the ground up to be convenient to program by directly writing hardware registers from assembly code (e.g. you didn't have to draw a sprite yourself, you just wrote a hardware register with a pointer to the sprite's image data in memory, and then wrote another hardware register to update the current sprite position and the video hardware did the rest). On modern hardware, providing such an interface is the job of drivers.
Also, on older home computer systems the hardware was essentially the rendering- and audio-engine. Those systems were designed from the ground up to be convenient to program by directly writing hardware registers from assembly code (e.g. you didn't have to draw a sprite yourself, you just wrote a hardware register with a pointer to the sprite's image data in memory, and then wrote another hardware register to update the current sprite position and the video hardware did the rest). On modern hardware, providing such an interface is the job of drivers.