From the World Heritage inscription:
Near the gardens of the Taj Mahal stands the important 16th-century Mughal monument known as the Red Fort of Agra. This powerful fortress of red sandstone encompasses, within its 2.5-km-long enclosure walls, the imperial city of the Mughal rulers. It comprises many fairy-tale palaces, such as the Jahangir Palace and the Khas Mahal, built by Shah Jahan; audience halls, such as the Diwan-i-Khas; and two very beautiful mosques.
The Agra Fort really gives you an impression of what it was like to live like a Mogul Emperor. Unlike the Red Fort in Delhi, the Agra Fort gives a much stronger impression of being a palace. The buildings, which were used as residences for the royal family, are almost entirely covered with marble and semi-precious stones.
The fort is also very close to the Taj Mahal and one can easily be seen from the other. In fact, if you visit the fort in the afternoon, you can get excellent photos of the Taj Mahal.
The Red Fort is a popular destination in conjunction with visits with the Taj Mahal as the two are so close together. Both attractions can easily be visited in a morning or afternoon.
The late afternoons are probably a better time to visit Fort Agra as most of the day trips from Delhi will be gone by then, and the crowds will be smaller.