There are daily buses from Athens to Olympia via Pyrgos. Passengers have to take a bus from Athens to Pyrgos and then another bus from Pyrgos to Olympia. The bus trip from Athens to Pyrgos is 4 hours and buses depart from Kifissos KTEL Bus Station. To go to Kifissos KTEL Bus Station, passengers have to take Bus 051 from Omonoia or Bus X93 from the airport of Athens. The bus trip from Pyrgos to Olympia is about 30 min.
There are also daily buses to Pyrgos from many towns of Greece, including Thessaloniki, Patra, Tripoli, Kalamata, Kyparissia and Ioannina. The bus trip from Pyrgos to Patra is 2 hours.
You can find more details on the time schedules of the buses at the website of KTEL Ilias.
Here you can find details about Pyrgos - Olympia and here about Athens - Pyrgos.