U.S. stocks initially slipped after the release of Yellen's prepared speech to graduates of the University of Baltimore, but then added to earlier gains.
Wall Street hit record highs last week as investors piled on bets that the anticipated fiscal boost from the incoming administration of U.S. President-elect Donald Trump would support riskier assets, Reuters reported.
"Traditionally, as you head into the week before Christmas, you see volumes slow down and a somewhat trendless market as people begin to position their portfolios for next year," said Matt Jones, U.S. head of equity strategy at J.P. Morgan Private Bank in New York.
The Dow Jones industrial average rose 44.89 points, or 0.23 percent, to 19,888.3, the S&P 500 gained 4.5 points, or nearly 0.2 percent, to 2,262.57 and the Nasdaq Composite added 22.78 points, or 0.42 percent, to 5,459.94.
The dollar edged back into positive territory after the release of Yellen's remarks, eyeing a 14-year high against a basket of currencies touched last week.
The Turkish lira and Russian rouble fell to session lows against the greenback on news the Russian ambassador to Turkey was killed in a gun attack at an art gallery in the Turkish capital of Ankara.
The lira was last down about 0.6 percent at 3.525 per dollar while the rouble hit a session low of 62.045 per dollar before retracing to 61.854, according to Reuters data.
The safe-haven Japanese yen added to gains after the report of the ambassador being shot, rising more than 1 percent against the dollar . However, it retraced much of those gains after Yellen's remarks.
"It seems like she is acknowledging the continued improvement in the jobs market. That's pretty consistent with what she and other policymakers have been saying," said Eric Viloria, currency strategist at Wells Fargo Securities in New York.
U.S. Treasury prices pared gains after surging higher following the news from Turkey.
The yield on 10-year U.S. Treasuries, which moves inversely to the note's price, was last 2.55 percent, pushing up from the day's lows.
Europe's index of 300 leading shares retreated from Friday's 11-month high and fell 0.07 percent. Germany's DAX index rose 0.2 percent while France's CAC slipped 0.22 percent. Britain's FTSE 100 edged up 0.08 percent.
Japan's Nikkei stock index, which has benefited from the yen's sharp fall against the dollar, snapped its nine-day winning streak, edging down from Friday's one-year high.
MSCI's all-country world index that tracks stock markets around the globe rose 0.04 percent.
Oil prices were mixed but held around $55 per barrel, with little news to influence the market.
Brent futures fell 0.45 percent to $54.96 a barrel, while U.S. West Texas Intermediate crude rose 0.5 percent to $52.17.