Samuel Aaron Goldstein, son of Woolf Goldstein, a jeweller, and his wife, Sarah, was born in London, England, by his own account on 12 June 1852. He received his rabbinical training at Jews' College, London, and was ordained at the age of 22. He taught Hebrew at a school in Yorkshire, then at West Hartlepool. On 14 June 1876 at Aldershot, Southampton, he married Eva Phillips, who was noted for her proficiency in languages and skills as a musician. The couple were to have two sons.
Samuel Goldstein became rabbi to the Hebrew congregation at Toowoomba, Queensland, Australia. An appointment at West Maitland, New South Wales, followed and in 1880 he was appointed rabbi to the Hebrew congregation at Auckland, New Zealand. He was to hold this post for the next 54 years.
When Goldstein arrived in Auckland at the age of 26, he faced the task of ministering to a small but growing Jewish congregation. It expanded from 373 in 1881 to about 950 by 1935. During the depression of the 1890s, as a contribution 'toward removing the difficulty under which our community is labouring', he reduced his own annual stipend by £75. He swept and cleaned the synagogue to earn the extra money he needed to support his family. From about 1895 Eva Goldstein's health was poor, preventing her from taking an active part in the work of the congregation. For many years the rabbi personally nursed his wife, who became bed-ridden.
Contemporaries praised Goldstein for his leadership qualities, learning, and contributions to Judaism. He worked hard to maintain orthodox Jewish life and to preserve unity in a congregation with diverse needs and aspirations. Some saw this as a striving for harmony and judged him a liberal-minded man who 'did not unduly stress the customs of the Jewish faith'. When, for example, his congregants did not attend synagogue on the second days of festivals, he decided not to observe them officially until a strictly orthodox member of the congregation insisted that he do so. Nevertheless, he sometimes alienated congregants by his extreme adherence to the tenets of Judaism, especially those relating to Sabbath observance, intermarriage and conversion. One example was Goldstein's exclusion from the congregation of anyone who let a daughter marry out or who married out themselves. Another was his refusal to facilitate intermarriage, even when his synagogue committee was prepared to and had received the concurrence of the chief rabbi. Although his efforts to build a strong, united Orthodox Jewish community were not entirely successful, they should be seen in the context of the assimilative pressures faced by members of the small and remote diaspora.
Goldstein was interested in civic affairs. He held office in the New Zealand Society for the Protection of Women and Children, the Society for the Prevention of Cruelty to Animals, and the Auckland Ladies' Benevolent Society. During and after the South African War (1899–1902) he was secretary of the Patriotic Society. He was a member of the Auckland City Council's library committee, a Freemason and a founder of the Société littéraire française.
Goldstein participated enthusiastically in the wider community and developed friendships with non-Jews of note, such as Sir George Grey. His activities stemmed from strongly held feelings about the obligations that Jews owed to the 'British realm'; they also reflected his diverse interests. He was a keen gardener, he played the cello, often taking it with him on the tram to entertain his congregants in their homes, and he was a learned man who 'read widely and thought deeply'. Christian theology students benefited from his knowledge of Hebrew language and literature. He was also knowledgeable about English, French and German literature.
Goldstein was president of the Auckland Zionist Society for 22 years. He is said to have served the Society 'with singular devotion'. In 1928 the title Morenu (teacher), was conferred on him by the chief rabbi of the British Empire, Dr J. H. Hertz, in recognition of his 50 years of service to Judaism. In December 1934 Goldstein retired. Eva Goldstein died in April 1935 and five weeks later, on 29 May, in Auckland, Samuel Goldstein died, survived by a son. A 'tall, stately and refined man', he was a dedicated rabbi who had made a significant contribution to Auckland Jewry and to the city of Auckland.