A massive and ever growing amount of digital image and video contentis available today, on sites such asFlickr and YouTube, in audiovisual archives such as those of BBC andINA, and in personal collections. In most cases, it comes withadditional information, such as text, audio or other metadata, that forms arather sparse and noisy, yet rich and diverse source of annotation,ideally suited to emerging weakly supervised and active machinelearning technology. The ALLEGRO project will take visual recognitionto the next level by using this largely untapped source of data toautomatically learn visual models. The main research objective ofour project is the development of new algorithms and computer softwarecapable of autonomously exploring evolving data collections, selectingthe relevant information, and determining the visual models mostappropriate for different object, scene, and activity categories. Anemphasis will be put on learning visual models from video, aparticularly rich source of information, and on the representation ofhuman activities, one of today's most challenging problems in computervision. Although this project addresses fundamental researchissues, it is expected to result in significant advances inhigh-impact applications that range from visual mining of the Web andautomated annotation and organization of family photo and video albumsto large-scale information retrieval in television archives.