For classic colonial fare, it’s hard to top genteel Essex, Connecticut.

Essex, Connecticut, is the kind of town associated with acres of madras and belts embroidered with little ducks. It is painted New England white, except for colonial-era bricks; it has impeccable Revolutionary credentials; and every other building sports a plaque with Puritan-sounding names, such as "Ezra Clark House, 1827."

Places like Essex are thick on the ground in New England. But Essex—which lies approximately 35 miles east of New Haven and encompasses three villages: Essex Village, Ivoryton, and Centerbrook—has made lists of best small towns in America, an achievement perhaps influenced by a lack of "Things to Do." Here, it seems, is a place whose attraction is simply itself.

Take the Griswold Inn, a landmark built in 1776 and one of the oldest continuously operating hostelries in the country. The "Gris" is a welcome anomaly among New England inns: It hasn't been prettified to death. Rooms are tidy, but you won't feel you're settling into luxury. The Windsor chairs may creak when you lean back, and the brass bed may not have been polished recently. The Gris's dining rooms—from the Covered Bridge Room, made with wood from a covered bridge, to the Gun Room, with a firearm collection dating to the 1400s—form a rambling warren centered upon a bar praised by writer Lucius Beebeas "the most handsome barroom in America." With a barrel vault of a ceiling and enough nautical memorabilia to make one expect a slap of spray in the face, it looks like a cross between an Irish saloon and a yacht club. Pints are served with fresh popcorn, and live bands get feet tapping. Says a local enjoying some refreshment at the bar, "The Nature Conservancy once called this stretch of the Connecticut River 'one of the last great places.' I think they meant to include this bar."

Once you've checked in, get out to explore the area—and feel free to pocket the tour map with descriptions of noteworthy buildings; all the buildings are noteworthy. Essex's sea captains put their town on the map with prosperous shipbuilding ventures and a brisk trade in West Indies rum, molasses, and sugar—and, in Ivory-ton, a world-class reputation for crafting ivory, toothpicks to piano keys. The trade captains did their successes proud in the vernaculars of Georgian, federal, and Greek Revival. One mansion was inspired in part by the grand Flagler Mansion in Palm Beach, Florida. Owner E. E. Dickinson made his money distilling a local shrub into that universal barbershop astringent, witch hazel. His home's splendor, and that of Essex's other mansions, made it clear that here was a small town doing very well for itself, "best" or not.