The Cats held their main session for the week under grey skies at Simonds Stadium on Wednesday afternoon, and Kelly, who missed last weekend's win over Carlton after being unable to shake off an achilles problem, was the notable absentee.
He is yet to be officially ruled out, with Geelong coach Chris Scott telling reporters earlier in the day that "we'd like to think that he'll be available".
But it now seems highly likely that Kelly will not make the trip to Sydney.
In contrast, Rivers, who strained the posterior cruciate ligament in his right knee during the Cats' win over Hawthorn on round one, moved freely during the 10 minutes of the session that was open to the media.
Having missed the round two victory over North Melbourne and the win over Carlton last weekend, the former Melbourne defender is set to add some strength to Geelong's backline as it tries to contain the reigning premiers.
However, the Cats are considering giving athletic big man Mark Blicavs a rest.
"There will be weeks where we don't play him," Scott said.
"Maybe even this week, after three games of footy, might be a week where we consider giving him a little bit of a spell. He's not a hardened AFL player.
"We're still yet to make that decision. But irrespective of whether Mark plays in any given week we see him as being a valuable part of our team later on in the year.
"If we were going to compare where he's at now to our expectations, it would be off the charts.
"He's got a long way to go to reach his peak, but we're really happy with the role he's doing for the team and very impressed with the way he's developing as an individual."
The Cats have lost the dash and creativity of Travis Varcoe, who underwent shoulder surgery on Wednesday.
Varcoe dislocated his shoulder during the opening quarter of the victory over the Blues.
"It's really unfortunate that he's in this position, but the prognosis is as expected," Scott said.
"He's going to miss three months. The fortunate part of that is he's back running after five or six weeks, so he's going to be in pretty good condition when he's available to be involved in contact training and matches.
"But we'd love to have him in the meantime. Unfortunately we won't, and that gives another opportunity to one of our young players this week.
"It's bad news for Trav, but it's not the end of the world."
With Kelly unlikely to return to face the Swans, a couple of youngsters find themselves at head of the queue to replace Varcoe.
"George Horlin-Smith played exceptionally well in the VFL last week, so he's been pushing hard all year – he played round one, obviously," Scott said.
"Mitch Brown continues to press his case … and we have some other options outside those guys, but they're the main ones."
Another of the club's talented young players, Daniel Menzel, has recovered well from his comeback match in the VFL on Sunday.
"We're confident that he could have played more footy, but we took the conservative option to ice him at three-quarter time and he's pulled up really well," Scott said.
"I think he's probably pulled up worse from some of the training sessions he's done leading into that game.
"That's a good sign. He'll play a little bit more this week.
"But … he'll need to play quite a few more VFL minutes before we'll consider him for the next step."
