State media said the tragedy struck in the mountainous Hoa Binh province of northern Vietnam at around midnight local time
A 51-year-old man was able to escape the house but his wife, daughter and two grandchildren were buried when the hillside collapsed.
Yagi is the most powerful storm in Asia this year has killed at least 14 and injured 176 others, according to Vietnamese state media.