Born at Adelaide, South Australia, on 10 February 1855, James Allen was the son of James Allen and his wife, Esther Bax. After his mother died James was taken to Dunedin, New Zealand, by his father, sometime between 1856 and 1859. In 1861 or 1862 he and his brother were entrusted to the care of an uncle at Cockhill, Somersetshire, England. They were orphaned with the death, in 1865, of James Allen senior, who had returned to Dunedin to attend to his business interests.
Allen boarded with his brother at a school in Somersetshire before proceeding in 1869 to Clifton College, Bristol, where he gained a scholarship in natural science. After graduating BA from St John's College, Cambridge, in 1877, he obtained a temporary teaching position at Harrow, but soon relinquished it to return to Dunedin. He took with him his cousin Mary Jane Hill Richards, whom he had married at Evercreech on 23 August 1877. They were to have six children.
Allen, who had inherited substantial property in Dunedin, soon became well-known. He was a keen sportsman, having excelled at rugby and rowing at Cambridge, and represented Otago at rugby in 1882. Two years earlier he had won a seat on the Dunedin City Council. He resigned in March 1883, and in the following year returned to England to attend a three-year course at the Normal School of Science and Royal School of Mines in London, in which he excelled. Taking up his business activities on his return to Dunedin, Allen established substantial interests in coal- and goldmining.
Allen also entered national politics. Standing as a conservative in the Dunedin East electorate in 1887, he sensationally defeated the premier, Sir Robert Stout. Although losing his seat in 1890, he re-entered parliament two years later when he won the seat of Bruce at a by-election. He proved an effective parliamentarian. Never an outstanding orator, he impressed by his solid dependability, intelligence, and practical good sense; qualities offset, however, by a certain dourness and lack of humour. Although seen by some as a potential leader of the conservative grouping that became the Reform Party – Allen himself may have had hopes in this direction – he eventually became deputy to the more populist William Massey.
Allen travelled extensively overseas. He spent nine months in Europe in 1898 and another six months in 1904, and was again overseas for three months in 1908. Participation in a six-week-long parliamentary tour of the Cook Islands and other islands, including Tahiti and Samoa, in 1903 stimulated his interest in the Pacific islands; he subsequently published New Zealand's possessions in the South Seas.
Allen had a long-standing concern with defence matters, having received his first military training at Clifton College. He was an active member of the Volunteer Force from April 1891, when he assumed command of the Dunedin Naval Artillery Volunteers. He later served on the Otago Divisional Staff and, in 1911, was appointed coast defence commander of the Otago Military District with the rank of colonel, a position he held until his resignation in July 1912. He was president of the Otago branch of the Navy League from 1912. In Parliament he was the opposition's leading spokesman on defence issues.
Allen also had a deep interest in education. He had been appointed the last life member of the Council of the University of Otago in 1887, and later served as vice chancellor (1903–9) and chancellor (1909–12); as a member of the Senate of the University of New Zealand he supported such progressive measures as the appointment of New Zealand examiners. Allen championed the cause of the university's School of Mines, but was also heavily involved in the development of other disciplines. Outside the university he helped establish, in 1889, technical classes and the Dunedin Free Kindergarten Association, and sat on the Otago Boys' and Girls' High School Board.
Allen was well prepared, therefore, to take the portfolios of both defence and education in Massey's Reform ministry in July 1912. But his major responsibility was finance, a field in which he had been an outspoken critic of Sir Joseph Ward's policy. He sought to bring departmental expenditure under more rigorous control and to alleviate the adverse effects of previous short-term borrowing overseas. However, both political necessity and development imperatives demanded the continuation of the Liberal policy. Allen was soon resigned to seeking a further loan for public works, and departed for London in December 1912. On his return journey from this successful mission the following April, he visited Canada, where he was well received as an imperial champion.
As minister of defence Allen established an effective working relationship with Major General A. J. Godley, the British officer who commanded the New Zealand forces. He took steps to improve the efficiency of the compulsory military training scheme, the introduction of which he had strongly supported while in opposition; encouraged preparations for the possible dispatch of an expeditionary force overseas; sought to develop an aviation capability; and (in contrast to his predecessors) strongly supported close defence ties with Australia.
Allen's most notable peacetime achievement was, however, the creation of the New Zealand naval forces in 1913. Long convinced that New Zealand must move away from the existing system of subsidies to the Royal Navy, he came to office determined to establish a locally based naval training programme. After difficult discussions with Winston Churchill during his London visit, Allen secured British agreement to the loan of a training ship, Philomel, and personnel to begin his scheme. Allen foresaw New Zealand developing a substantial naval unit as part of an imperial force in the Pacific.
In the sphere of education, Allen proved receptive to new ideas. En route to London in January 1913 he visited schools in Italy and Switzerland, and was favourably impressed by the Montessori system of education. He was responsible for legislation in 1914 which provided for statutory funding of the university colleges and a board of studies. However, his support, as a private member, for Bible reading in state schools aroused substantial opposition.
Following the outbreak of the First World War in August 1914, Allen's considerable administrative skills were clearly demonstrated as he oversaw the development and maintenance of New Zealand's war effort, which centred on the expeditionary force dispatched to the Middle East in October 1914. The rapidity with which this force (as well as a smaller force which captured German Samoa on 29 August 1914) was raised reflected well on Allen's pre-war approach to defence.
Although the formation of the National coalition government in August 1915 led to the removal of his finance and education responsibilities, Allen was soon confronted with an even heavier burden. For three extended periods from August 1916, lasting 23 months in all, he was acting prime minister while Massey and Ward attended imperial discussions and the peace conference. He was effectively New Zealand's war leader, and had to deal with a succession of industrial and war-related crises. He found his task increasingly difficult as the coalition faltered following the Allied victory in November 1918.
In order to achieve his overriding objective of maintaining New Zealand's military contribution to the imperial cause, Allen had to overcome many administrative problems; he also, increasingly, faced political problems as war-weariness affected public attitudes. With his low-key approach Allen surmounted a succession of obstacles. To meet the need for a steady stream of trained reinforcements to the New Zealand Expeditionary Force, the government introduced conscription in 1916. That the new system operated effectively owed much to Allen's firm but sensitive handling of this divisive issue. He was responsible for war pensions and the rehabilitation of repatriated soldiers, and was angered by the cabinet's refusal to appoint his son-in-law as director of repatriation.
Allen's wartime performance was, nevertheless, surrounded by controversy. Administrative shortcomings, especially in relation to Trentham Military Camp, led to a crescendo of criticism in 1915, which he bore stoically. Compounding the strain was the news that his son, John, had been killed in June 1915 while serving with British forces at Gallipoli, a campaign that Allen described privately as 'ill-conceived and mad'. The establishment of the coalition lessened the stress on him, but his generally uninspiring and parsimonious approach set limits to his popularity among both his cabinet colleagues and the public, and he had to endure continuing criticism. He drew satisfaction, however, from increasing public recognition. Appointed a KCB in 1917, he was vindicated by the favourable outcome of an independent investigation in 1918 of the administration of the Defence Department. Moreover, his efforts on behalf of the expeditionary force were formally acknowledged by its commander, Godley, in 1919.
Following the coalition's demise in August 1919, Allen reluctantly became minister of finance once again. Wearied by his wartime efforts and tired of politics, he wanted to stand down at the 1919 general election in order to make a lengthy visit to the United Kingdom. He was persuaded otherwise by Massey, and eventually retained his seat by a narrow margin. In November 1919 his responsibilities were expanded to include the administration of New Zealand's interests in the Pacific islands. As the first minister of external affairs, Allen instituted constitutional arrangements for administering New Zealand's League of Nations mandate in Western Samoa. In February–March 1920 he led a parliamentary party on a visit to the Cook Islands, Western Samoa and Fiji.
Allen's opportunity to leave politics came with his appointment as high commissioner for New Zealand in the United Kingdom. He sailed for Britain on 30 April 1920. During his six-year term he served as an ex officio member of a number of imperial organisations and helped direct the British Empire Exhibition of 1924–25. He also represented New Zealand at the League of Nations, journeying to Geneva each year to attend the assembly and meetings of the Permanent Mandates Commission. Appointed to a commission to oversee the League's finances, he apparently exerted considerable influence while exhibiting his characteristic cheese-paring approach. In March 1926 he was elected vice chairman of the special assembly to consider Germany's admission to the League. Although sceptical of the League's chances of outlawing war, Allen saw value in the better understanding it promoted among nations. For security he preferred to rely on imperial defence plans based on the power of the Royal Navy.
Following his return to New Zealand Allen was prominent in the Institute of Pacific Relations, presiding over the New Zealand branch from its inception in 1927 until 1939 and serving as a member of the institute's Pacific Council. He also resumed his association with Dunedin's All Saints' Anglican Church, which was built on land donated by his father. Before becoming a minister of the Crown he had been a vicar's churchwarden, and he was now elected to the vestry of that church. Licensed as a lay reader, he conducted services in some of the outlying churches. He was also vice president of the New Zealand Bible-in-Schools League and chaired the Otago branch.
Allen, who was made a GCMG in 1926, resumed his parliamentary career when he was appointed to the Legislative Council on 1 June 1927. He was an active member until he retired from public life in 1938, and was not reappointed when his second term ended on 31 May 1941. Allen resided at Arana, his house in Dunedin, until his death on 28 July 1942, three years after that of his wife. He was buried in Dunedin's northern cemetery.