Tucker: Let’s get together for breakfast this Sunday.
Paul: Let’s make it lunch. I rarely get out of bed before noon on the weekends.

Something that happens rarely does not happen very often. It happens only occasionally. It is not impossible. Paul probably does get out of bed before noon on the weekend at least a few times a year. But he needs a very good reason.