For the past few months I have been working with academic and student researcher colleagues on a project to produce an online resource about social theory. It has raised design questions but the most interesting aspect of the exercise has been talking to academic colleagues (again academics and research students) about the role of theory in their work. Theory as a concept has always troubled me but through the project I could see how widely this ‘trouble’ was shared, particularly among new researchers. Having someone ask about theory puts many of us on the defensive. It can easily lead to a guessing game (what is the questioner thinking when he / she uses the word theory?) or game playing (they are going to catch me out as I have not read X, or read but misinterpreted X, or not read Y who refutes X). Further, theory is never going to be straightforward as it gathers together many different assumptions about epistemology, methodology and indeed the purpose of doing social research. The same word, theory, is used to describe anything from cause and effect associations; models of economic behaviour; ethnographic description of social phenomena; justice in hypothetical situations and so on .
A second problem with theory is that theorisation, especially inductive theorisation, seems so vague. Nearly everyone we spoke to described experiencing insights (for some best described as ‘aha’ moments) when ‘things fell into place’, for example when several cases of similar behaviour could be grouped under a larger concept or when the relationship between two kinds of actions became clear. However these leaps of the imagination are rarely articulated by academics themselves or indeed by those writing about research methods. This leaves the new researcher knowing that they have something very important to carry out but little in the way of advice on how to do it. So is there anything that might help in addressing the difficulties that theory seems to raise?
I think there are some core associations about theory that can be articulated . First, theory points to an attempt to abstract something from the data (a model, a rich description or conceptual category) which allows for understanding phenomena in more generalizable or at least more relatable ways. Without theory you are reporting on what is happening in a local context, theorising allows you to see the potential general significance. People we interviewed in our project further spoke about their attention to vocabulary when theorising, finding ways to articulate ideas using a very precise and often abstract language. They were also aware how this language worked within a theoretical tradition, perhaps a discipline tradition but equally a cross disciplinary one.
Second, theory for our interviewees was further concerned with explanation. This does not necessarily mean cause and effect explanation, explanation could as easily be focused on the actions and consequences of actions. Even if we have competing definitions of theory, the task, as our interviewees saw it, was to locate the tradition of theory in which one’s work sits. This is not an ‘anything goes’ approach but rather a requirement to argue for the value and the shortcoming of a tradition.
Third, it is possible to celebrate rather than run away from the idea of subjectivity in the process of theorisation. Social research might be as much art as science but it is a special kind of artistry based upon an intense, the only word I can use I am afraid is, engagement with both literature and data  . In my experience it is striking that natural scientists and mathematicians, or a least those who have been interested in reflecting on the process of research in these fields, are not only willing but enthusiastically embrace intuition and indeed the aesthetics of theorisation. As famously expressed by Polanyi 
The affirmation of a great scientific theory is in part an expression of delight. The theory has an inarticulate component acclaiming its beauty, and this is essential to the belief that the theory is true. No animal can appreciate the intellectual beauties of science.
I am not sure we would talk about theorisation in the same way in social research, but we could. Perhaps scientists feel more willing to romanticise theory as the popular conception (whether right or wrong) is that their work concerns hard facts and the data they are working with have an objective quality. In contrast in social research the data will always be questioned on ground of reliability, validity or whatever; to admit to subjectivity in theorisation just seems to be one more step in a shaky process.
Through the project we have a better idea about supporting theory but there is still much more to do.
 see from the student point of view
Kiley, M. (2015) ‘I didn’t have a clue what they were talking about’: PhD candidates and theory, Innovations in Education and Teaching International, 52:1, 52-63.
 See several different associations made with theory in Krauze, M. (2016) The meanings of theorizing, The British Journal of Sociology, 67 : 1, 23-29.
 Not surprisingly C Wright Mills work on the sociological imagination is offered repeatedly as an example of doing theory, as more recently does Umberto Eco on ‘how to write a thesis’. Good as these contributions are they, perhaps the recurring references to both illustrate the very restricted range of other reported experiences.
 Polanyi, M. (1983) Personal Knowledge. Toward A Post-Critical Philosophy, London: Routledge & Kegan Paul, pp. 133-135.