Pattinson played for Dandenong Cricket Club in Victorian Premier Cricket and also played for Australia in the 2008 Under-19 World Cup in Malaysia.

On 23 December 2009, James broke the record for Victorian One Day cricket by taking 6 for 48 off 10 overs with 1 maiden, despite his efforts Victoria still lost to NSW at the SCG.

On 9 January 2011, James was picked up by IPL team Kolkata Knight Riders for US$100,000 to play for them in IPL 2012.

Pattinson was named in the Australian ODI squad for its tour of Bangladesh in April 2011. He made his ODI debut in the third and final match of the series and took the wicket of Imrul Kayes. Pattinson joined the Australian Test squad for the 2011 tour to Sri Lanka, but did not play a Test.

Pattinson made his Test debut in the 2011/12 home series against New Zealand, in the first Test in Brisbane on 1 December 2011. He took 5 wickets for 27 runs in the second innings of his debut Test and helped Australia to a nine-wicket win, then took five wickets in the first innings of the second test, and was named Man of the Series, with fourteen wickets at an average of 14.00 across the two tests. He won another Man of the Match award in his next Test, the 2011 Boxing Day Test against India, with match figures of 6/108 and 55 runs. James Pattinson was not included in Australia's squad for the 2012 T20 World Cup.

Before the 3rd Test against India in March 2013, Australia dropped Pattinson, along with Shane Watson, Mitchell Johnson and Usman Khawaja following a breach of discipline.^[11] Michael Clarke, the captain, revealed that the extreme step had been taken as a result of repeated infractions which led to Watson flying back home and contemplating Test retirement.^[12] Former players reacted with astonishment at the harsh decision taken by the team management.^[13]

Pattinson later returned to the Australian Test team during the West Indies tour of Australia in December 2015. In his first Test match back, Pattinson achieved his fourth five wicket haul, taking 5/27, which equaled his previous best bowling figures. He also reverted to his old bowling action, which gives him more control and pace that touches close to 150 km/h, but at a risk of re-injuring his back.^[14][15]

On the 20th April 2017, Pattinson was named in Australia's squad for the 2017 ICC Champions Trophy.