The LaLiga giants, who won the first leg 2-1 in Amsterdam, lost 4-1 on their home patch.
Ajax raced into a three-goal lead, thanks to strikes from Hakim Ziyech, David Neres and Dusan Tadic.
Marco Asensio pulled one back for Madrid, before Lasse Schoene whipped in a stunning free-kick.
Borussia Dortmund are also out of the competition.
The Bundesliga side lost 1-0 to Tottenham Hotspur at home, after suffering a 3-0 whitewash in the first leg.