Gordon was born in Pictou County, Nova Scotia, and educated at Pictou Academy, University of Glasgow (MA 1863, BD 1866, DD 1895), and Berlin University.
Gordon was ordained as a Presbyterian Minister in 1866 and served successively in charges at Truro, Ottawa, Winnipeg, and Halifax.
In 1880, he wrote the book Mountain and Prairie, an account of a journey from Victoria, B.C. to Winnipeg, Manitoba via the Peace River pass.
He was an army chaplain during Louis Riel's North West Rebellion of 1885.
From 1894 to 1902, before coming to Queen's, Gordon taught Theology at the Halifax Presbyterian College.
He was appointed Principal of Queen's in 1902 after the death of fellow Pictou native George Monro Grant. Gordon initially declined the post, saying that a man younger than himself should be sought for the position. It was his friend Sir Sandford Fleming who convinced him to finally accept, and he was officially installed at the end of that same year. No one could entirely replace the forceful and charismatic Grant, but Gordon was a popular and well-regarded leader who safely led the University through a period of far-reaching change.
The separation of Queen's from the Presbyterian Church, which took years of effort and was finally accomplished in 1912, ensured that the University would get its share of the private and public funding that was increasingly limited to non-denominational colleges. Gordon also led Queen's through the trying early years of the First World War, when enrollment and funding both plummeted, and the unification of the Faculty of Practical Science with the School of Mining and Agriculture in 1916 (see Engineering and Applied Science, Faculty of).
He retired in failing health in 1916, aged 73.
Gordon's family has several ties with Queen's besides his position. In 1869, Gordon married Eliza Simona MacLennan, a clergyman's daughter, and the main dining room in Ban Righ Hall was named in her honour. Gordon's daughter Wilhelmina graduated from Queen's in 1905 with an MA. She was the first female faculty member at Queen's, beginning as a teaching assistant in English in 1909.
He is buried in the Cataraqui Cemetery.
Gordon's papers are held in the Queen's Archives.