Rosberg recovered from a slight loss of power early in the session to go just 0.14 seconds quicker than Hamilton.
It was an incident-filled session that included a spectacular double-spin by Red Bull driver and current world champion Sebastian Vettel.
Ferrari's Fernando Alonso beat Williams' Felipe Massa to third while Jenson Button claimed fifth place for McLaren.
The session was marred by sporadic rain showers but after a 10-year absence, the redesigned Red Bull Ring provides a demanding challenge in a spectacular setting under the Styrian Mountains.
The Red Bulls of Canadian Grand Prix winner Daniel Ricciardo and Vettel only managed 13th and 15th on their home track while Kimi Raikkonen went 11th-quickest, 0.7secs down on team-mate Alonso.
A number of drivers found themselves off the track as they get to grips with unfamiliar surrounds, and the field looks delicately poised ahead of the eighth grand prix of the year.