Ukraine can still win the war if given the right support, he added.
He also announced the US would allocate an extra $713m (£559m) of military aid to Ukraine and other European nations.
Russian President Vladimir Putin has accused the West of attempting to "split Russian society and destroy Russia from within".
Mr Austin, a retired four-star general, was speaking after meeting Ukraine's President Volodymyr Zelensky in Kyiv. Also involving Secretary of State Antony Blinken, the visit marked the highest-level trip to Ukraine by US officials since the invasion began over two months ago.
The meeting between the US and Ukrainian parties, which ran for over three hours, comes as Russia escalates its military campaign in the south and east of the country.
At a news conference in Poland after the visit, Mr Austin told reporters the US wants to see "Russia weakened to the degree that it can't do the kinds of things that it has done in invading Ukraine".
The Pentagon chief added that US officials still believed Ukraine could win the conflict with "the right equipment" and the "right support".
BBC diplomatic correspondent James Landale observed that Mr Austin's comments calling for a weakened Russia were unusually strong for a US defence secretary. It is one thing to help Ukraine resist Russian aggression, it is quite another to speak of weakening Russia's capabilities, he said.
Mr Putin's accusations of the West attempting to "destroy Russia", made during a speech on Monday, appeared to be in response to Mr Austin's comments.
US defence officials said of the millions announced in fresh military funding, nearly $332m (£260m) would be allocated to Ukraine.
It brings the total US security assistance provided to Ukraine since the invasion began to more than $3.7bn (£2.9bn).
Mr Zelensky has been pleading with Western leaders to increase the flow of military equipment for weeks, vowing that his forces can overcome Russia's military if provided with fighter jets and other vehicles.
Last week the US confirmed it had supplied Ukrainian troops with howitzer artillery cannons and anti-artillery radars for the first time.
Russia's ambassador in Washington said Moscow had sent a diplomatic note demanding an end to the US supply of weapons to Ukraine.
Mr Blinken announced some US diplomatic staff would start to return to Ukraine from next week. They are expected to be based in Lviv initially, with a longer-term plan to re-open the US Embassy in the capital, Kyiv.
President Joe Biden intends to nominate Bridget Brink, a career diplomat, as US ambassador to Ukraine - a post that has remained vacant for more than two years.
Mr Blinken also defended America's diplomatic approach, telling reporters that the Western alliance the Biden administration had assembled had put pressure on President Putin's government.
"The strategy that we've put in place, massive support for Ukraine, massive pressure against Russia, solidarity with more than 30 countries engaged in these efforts, is having real results," Mr Blinken said. "And we're seeing that when it comes to Russia's war aims, Russia is failing, Ukraine is succeeding."
"A sovereign, independent Ukraine will be around a lot longer than Vladimir Putin's on the scene," Mr Blinken added.
Speaking following the meeting, Mr Zelensky said his government appreciated "the unprecedented assistance" from Washington and added that he "would like to thank President Biden personally and on behalf of the entire Ukrainian people for his leadership in supporting Ukraine".
President Biden's always made clear that the US won't directly intervene in the war in Ukraine. He won't be sending American troops to join the battle.
But the truth is that the US is getting more deeply involved - and that's been highlighted by the words of the US defence secretary.
America has dramatically increased the supply of weapons to Ukraine in recent weeks. Despite Russian warnings, it's made clear it'll be sending more. Mr Austin's words underline that America is not a bystander in this war. America wants to see Russia defeated.
More than that, it wants to see Russia's military machine reduced to such an extent that it will not be able to threaten any another European ally in the future. The fear, shared by Nato allies, is that any kind of defeat for Ukraine will only embolden President Putin's ambitions.
Mr Austin has now set out clearly that the US has its own strategic goals for this war - even if it's theoretically not participating. The goal is to stop President Putin and to weaken Russia's military to such an extent that it will no longer be able to threaten other nations.
To some extent that's already happened. Military experts already believe it will take years for Russia to recover from its military losses. That might also send a signal to another nation which the US is increasingly concerned about - namely China.
For President Putin it's more evidence that he's not just in a war with Ukraine. No doubt it will feed his own narrative for the purposes of domestic support. He's long portrayed Nato as a threat to Russia.
In recent days Russia has refocussed its advance on Ukraine's eastern Donbas region and US sources believe that Moscow has committed over 76 battalion tactical groups inside Ukraine.
On Monday, the UK's ministry of defence said in an update that Moscow had made minor advances in the region but had "yet to achieve a breakthrough" because of logistical issues.
The update said: "Ukraine's defence of Mariupol has also exhausted many Russian units and reduced their combat effectiveness."