Minimization with respect to a matrix X subject to orthogonality constraints X'X = I has wide applications in polynomial optimization, combinatorial optimization, eigenvalue problems, the total energy minimization in electronic structure calculation, sparse principal component analysis, community detection and matrix rank minimization, etc. These problems are generally difficult because the constraints are not only non-convex but also numerically expensive to preserve during iterations. This talk will present a few recent advance for solving these problems.