The ability to understand and manage social signals of a person we are communicating with is the core of social intelligence. Social intelligence is a facet of human intelligence that has been argued to be indispensable and perhaps the most important for success in life.Although each one of us understands the importance of social signals in everyday life situations, and in spite of recent advances in machine analysis and synthesis of relevant behavioural cues like blinks, smiles, crossed arms, laughter, etc., the research efforts in machine analysis and synthesis of human social signals like empathy, politeness, and (dis)agreement, are few and tentative. The main reasons for this are the absence of a research agenda and the lack of suitable resources for experimentation.The mission of the SSPNet is to create a sufficient momentum by integrating an existing large amount of knowledge and available resources in Social Signal Processing (SSP) research domains including cognitive modelling, machine understanding, and synthesizing social behaviour, and so: (i) enable creation of the European and world research agenda in SSP, (ii) provide efficient and effective access to SSP-relevant tools and data repositories to the research community within and beyond the SSPNet, and(iii) further develop complementary and multidisciplinary expertise necessary for pushing forward the cutting edge of the research in SSP. The collective SSPNet research effort will be directed towards integration of existing SSP theories and technologies, and towards identification and exploration of potentials and limitations in SSP. A particular scientific challenge that binds the partners is the synergetic combination of human-human interaction models and tools for human behaviour sensing and synthesis within socially-adept multimodal interfaces.