Comment by solarkraft

Are there benchmarks that effectively measure this? This is essential information when speccing out an inference system/model size/quantization type.