The following is from page 45 of Lashon HaKodesh, History, Holiness, & Hebrew by Rabbi Reuven Chaim Klein.
Why did Balaam and the builders at Babel speak Lashon HaKodesh?
The Zahar (Noach 75b), the principal textbook of Kabbalah, also adopts
the approach favored by Rashi and Midrash Tanchuma that the builders of the Tower of Babel spoke Lashon HaKodesh. The Zahar writes that because the builders of the Tower of Babel had arranged their sin by using Lashon HaKodesh for communication, they were granted supernatural powers to complete their ambitious project. Lashon HaKodesh is an especially holy language and when used for good, it can help significantly raise one's spiritual potential. However, the converse is also true: When Lashon HaKodesh is used to sin, it enables the sinner to sink deeper into the depths of evil. Thus, explains the Zahar, God stopped the builders from finishing their project of rebellion against Him by causing them to forget Lashon HaKodesh, thereby cutting off their supernatural support.
Based on this concept, Rabbi Aharon Roth (1893-1946) and Rabbi Yoel
Teitelbaum note that a verbal sin (e.g., immoral speech, slander, blasphemy) committed in Lashon HaKodesh is a much more serious offense than such a sin committed in any another language.
Source: Rabbi Roth's Ma'arnar Tzahali V'Rini (chps. 2-4, printed at the end of Shornrei Ernunirn, Jerusalem, 2002), and Rabbi Teitelbaum's VaYoel Moshe (Ma'arnar Lashon HaKodesh §18-19).