Footage broadcast by the Japanese media shows firefighters trying to dose the flames as well as thick plum of black smoke rising against the skyline.
According to NHK, 1,281 households were urged to evacuate, affecting at least 2,986 people. Around 500 of them fled and were taken to shelter at a local welfare center and a junior high school.
No reports of casualties were available following the disaster, but the affected area is said to be over 100 square meters (1,076 square feet).
The refinery is owned by the TonenGeneral Group, and has the capacity to process 132,000 barrels of crude per day. It produces gasoline, kerosene and benzene.
A refinery employee told The Japan Times she was worried that winds might spread the fire further.
More than six hours into the firefighting effort, the reason behind the blaze remains unclear. A unit producing lubricant oil is believed to have caught fire; no blast was reported.
Oil refining is one of the major industries in the coastal city located in the Wakayama Prefecture on Honshu island.