Jacob Wexler was born in Liepāja (German: Libau), Latvia. His parents were members of the Jewish Labour Bund. His father, Tuvia, was an accountant, and his mother, Sonia (Sarah-Sheina née Krupnik), was a seamstress. In addition to Jacob, they had three more children, Avraham, Esther, and Meir. In 1923, the family relocated to Germany, and Tuvia Wexler enrolled in the Faculty of Mathematics at the University of Hamburg. The children attended Jewish schools in the city. The family had to live frugally, and in 1926, Tuvia was forced to stop his studies. He became religiously observant, and the Wexler household became religious. Jacob, who was a member of the Jewish-Socialist youth movement HeHalutz, refused to cooperate with the change.
In 1930, despite his father's objections, Wexler began studying at the University of Fine Arts of Hamburg, where he was introduced to twentieth-century art, especially German Expressionism. In March 1933, following the Nazi Party's rise to power, he was forced to abandon his studies. In April 1935, he married Annie Roch. In the summer of 1935, the couple arrived in Palestine and joined Kibbutz Ein HaHoresh.