While front-end is often where most of the problems in website perceived performance lies, back-end performance is a major issue in Drupal 7, and currently even more so in Drupal 8.
Most backend optimizations focus on improving SQL queries, switching to document-based storage, and globally reducing PHP inefficiencies.
However, experience gathered around large media sites during last year brought evidence of a usually ignored dimension to performance: cache tuning.