Chapter 3: Research Methods

Posted on August 23, 2012

0


For the sources of the quotes, please use the Bibliography (which includes links to those available on the  web).

Don’t forget that it’s fine for you to use this work for your own purposes, so long as you credit the source. For private or commercial purposes please credit Charley Newnham and include a link to her Linked In profile. For academic purposes please click here for the complete reference.

———— >

 

3. RESEARCH METHODS

 

 “It is better to have ideas and for some of them to be wrong, than to always be right by having no ideas at all.”  Edward de Bono (2009)

 

This chapter describes the methodology used to obtain indicative answers to the questions identified at the end of the last chapter.  It comprises of sections titled:

  • Research Strategy
  • Study Ethics
  • Questionnaire Creation
  • Data Collection and Framework for Analysis
  • Summary

Study limitations are recognised and noted within the text of this chapter.

 

3.1 Research Strategy

Research for this study can be explained is explained under two subheadings: 3.11 Literature Review, and 3.12 Primary Data from Organisational Leaders and “Resilience” Business Units.

 

3.11 Literature Review

Initial research involved reviewing academic books and papers relevant to the subject of organisational resilience.  The most relevant sources were chosen and discussed, as outlined below.

In the context of studying what appeared to be an emerging field, an early decision was made to use data from studies and materials produced as recently as possible. A brief scan of the bibliography shows many sources from this 2011 and 2010, and the vast majority from this millennium.  However, quality material that pre-dated this was included where appropriate and where later studies were not available.

The core sources used for this study are the four books and one paper referenced in the previous chapter.  Further academic papers were sourced primarily via Google’s Scholar Search and access was gained using the researcher’s Cranfield University credentials.  Initial searches were run on wide topic themes such as “organisational resilience” and “leadership for resilience”.  As the work progressed searches were tightened to the issue in hand, for example, “average tenure of chief executive officers in the UK”.   Some papers were identified by individuals following the blog for this dissertation (see below), and were sourced using the same method as for other papers.

As the intended outcome was to produce work to assist leaders of organisations employing at least 100 people and possibly many thousands, popular academic business titles were used to source case studies and editorial opinion. The Harvard Business Review (HBR) was the journal of choice in these cases because of its academic robustness, significant reputation and global, professional readership.  The publication of HBR’s “The Failure Issue” (Harvard Business Review, 2011) was particularly useful and references to number of articles from this issue are found in this work.

An online blog was also used to share the progress of this work online.  This was a deliberate strategy to harness literature recommendations and new ideas from others researching the field.  The blog was launched before the thesis proposal was submitted and, via the blog, contact was made with many individuals, including resilience academics Liisa Valikangas[1] and Ned Powley[2], and particularly recent resilience PhD graduate Amy Stephenson[3].  Gaining access to Stephenson’s work before publication by ResOrgs at the end of August 2011 was invaluable.  The blog that accompanied this work can be found online at http://www.goldordust.wordpress.com.

 

3.12 Primary Data

The lines of enquiry identified at the end of the previous chapter revealed a need to gather data from two sources.   Information pertaining to organisational leaders and leadership had to be sought from organisational leaders.  Information regarding “Resilience” business units could be usefully obtained by approaching such units directly.

In both cases it was decided that the “tried and tested research strategies and data collection techniques” (Biggam, 2010) associated with a survey/questionnaire (defined as “a method for collecting primary data in which a sample of respondents are asked a list of carefully structured questions chosen after considerable testing, with a view to eliciting reliable response” (Balahur & Steinberger, 2009)) would be used for each strand.

It is useful to consider the primary data strands separately:

 

3.12(a) Organisational Leaders Strand

Most of the lines of enquiry identified relate to behavioural preferences of organisational leaders in relation to resilience.  Respondents able to provide useful data in this strand were organisational leaders.

In keeping with the studies interest in making this work useful to leaders of large organisations as well as small ones, the decision was made to target leaders of reasonably established companies that were likely to have cause to have considered at least some of the resilience function issues (e.g. risk management, business continuity, security) as well as their own management principles.

Required samples size was carefully considered.  Statistics were sought to define the potential population of ‘organisational leaders’ but none credible sources for this data were not identified.  It was not therefore possible to consider a population, confidence level and confidence interval calculation (Biggam, 2010) to determine a truly representative sample for this study.  As a degree of analysis of potential trends within the data was intended, the sample set needed to be big enough to allow this to a reasonable degree.  However, as it was acknowledged that the resulting data was intended to provide “rich and detailed insights” (Collis & Hussey, 2009) and a potential basis for further research rather than outright conclusions, a particularly large sample was not a significant requirement. Cranfield academic guidance[4] indicated that “about twenty” respondents would be needed in order to be able to discuss the results of this strand with a level of credibility: as shown later, the number of responses significantly exceeded this.  Furthermore, statistical relevance calculations were applied to all apparent data correlations to assure significance had been secured.

A number of strategies for data collection were considered, including case studies of individual leaders or leaders within one organisation, structured interviews and a survey.  A survey that included questions that could be used for descriptive and elementary analytics provided an efficient way of asking uniform questions to individuals suitable for providing data (Collis & Hussey, 2009).

The decision to use a convenience sample (Biggam, 2010) was made at the outset of this study.  There are issues with the usefulness of a convenience sample because no claim can be made that it represents an entire population of any kind (Biggam, 2010).  However, as this study is exploratory, adding to the resilience discussion and providing a basis for further research, the convenience sample method is “perfectly acceptable” (Biggam, 2010) and as the survey will be conducted online, it is in fact the only acceptable method (Schonlau, Fricker, & Elliott, 2002).  Practically speaking, it also innovative techniques to increase the number and variety of respondents, as shown in the Data Collection section.

In order to provide useful data, it was decided that responses would be solicited from organisational leaders working in reasonably established organisations. (These terms are defined in the section titled Data Collection and Framework for Analysis.)

 

3.12(b) Resilience Business Unit Strand

The other area that required primary investigation was around the scope, purpose and output of “Resilience” business units emerging in organisations.   While the Organisational Leaders strand asks about behaviour preferences and opinion, which are subjective (Balahur & Steinberger, 2009) the information required on “Resilience” business units was primarily factual.

The respondents best able to provide useful data in this strand were individuals working in business units with the word “Resilience” in their titles.  In order to maintain a degree of similarity between the data sets of the two strands, it was decided that the same qualifying rules would apply to ensure that responding organisations were reasonably established.

Research was undertaken to understand how many organisations might have departments with “Resilience” as a keyword in their title in order to understand the potential population.  No obvious sources of this information were available.  Searches were performed on Google and Linked In[5].  This showed there was very few, compared with, say, Human Resources, which seems to appear in most organisations with more than 100 employees.  It was therefore decided that a response from a minimum of ten UK organisations would provide sufficient data to be able to discuss the scope, roles and outputs of such departments.  In the event, this number was significantly exceeded.

The data required from the “Resilience” business units were relatively straightforward and a survey provided an efficient way of asking uniform questions to individuals suitable for providing data (Collis & Hussey, 2009).   The decision was again made to use a convenience sample for exactly the same reason as given above for the Organisational Leadership strand.

  

3.2 Study Ethics

As with any study, the ideas included in this thesis come from a variety of sources.  These are duly credited throughout the study, however, it has not always been noted where other individuals recommended the examination of a particular text.

Questionnaire respondents were assured anonymity to encourage responses.  Though respondents were asked provide an email address if they wanted a copy of the study findings, not every respondent did and, as per the researcher’s stated promise, email addresses were detached from the study data before it was analysed.

Primary data from the surveys has been retained in electronic files. They are backed up in a Dropbox storage file, though the Dropbox file will be removed when this thesis has been marked.   The analysis process has been retained so several versions of the data exist, from the raw data downloaded from the online survey through all the ‘working out stages’.  The raw data (excluding any email address) can be made available to Cranfield marking staff if required, if the data is not copied nor retained.

 

3.3 Questionnaire Creation

As already noted, a questionnaire was created for each strand: Organisational Leaders and Resilience Departments.

The lines of enquiry were dictated by the questions identified at the end of the Issues and Literature Review.   Copies of the questionnaires are included as Appendix 2 (Organisational Leader Questionnaire) and Appendix 3 (Resilience Business Unit Questionnaire).

Considerations and activities during the creation of the questionnaires included the following:

ü Ensuring survey questions were formulated so that answers would inform the study questions

ü Using an online platform to maximise response opportunities from suitable personal contacts, online contacts and via online forums including this study’s blog

ü Ensuring the security of data (Schonlau, Fricker, & Elliott, 2002)

ü Keeping language simple and universal (de Vous, 1991) with only a few questions per screen

ü Using multiple choice or Likert scale answer options (May, 1993)where appropriate, but using matrix questions sparingly, to assist with ease of completion and reducing response errors (Schonlau, Fricker, & Elliott, 2002)

ü Forcing answers only on rare occasions where no data can be analysed without that answer (Schonlau, Fricker, & Elliott, 2002)

ü Placing classification questions at the beginning of the survey (May, 1993), asking factual questions first and moving to more difficult questions at the end of the survey (de Vous, 1991)

ü Avoiding double-barrelled, duplicate and leading questions (de Vous, 1991)

ü Avoiding questions encouraging ‘prestige bias’ (de Vous, 1991)

ü Changing the style of question frequently to keep the respondent interested (Schonlau, Fricker, & Elliott, 2002)

ü Setting criteria and practice for excluding responses that did not meet study requirements/standards to ensure data quality (Schonlau, Fricker, & Elliott, 2002)

ü Embedding coding for responses in online survey to make analysis easier when data was returned (de Vous, 1991)

ü Ensuring the questionnaire looked professional to gain confidence of respondents (May, 1993)

ü Offering anonymity and ‘something’ in return for completing the survey (May, 1993)

ü Piloting the questionnaire in person so issues could be identified and rectified (May, 1993) and to ensure people didn’t give similar answers to every question (de Vous, 1991)

ü Staggering invitations to complete the survey to ensure respondents could report any problems, meant any unforeseen issues could be addressed (Schonlau, Fricker, & Elliott, 2002)

3.4 Data Collection and Framework for Analysis

3.41 Organisational Leader Questionnaire

The terms ‘organisational leader’ and ‘reasonably established organisations’ had to be defined in order to seek out those who qualified, disqualify those that did not, and use the data appropriately after collection.  It was important that the definitions provided an objective measures so no-qualifying responses could be discarded:

For ‘organisational leader’, terms such as leader, senior manager and director are used in different ways in different organisations so were not appropriate.  Instead it was decided that seniority would be established by counting the number of line managers between an individual and the organisation’s Chief Executive Officer (CEO)/Managing Director (MD) or equivalent.  So, if a respondent had no line management between themselves and the CEO/MD/equivalent, or indeed was that person, then they would select zero; if a respondent reported to someone who reported to the CEO/MD/equivalent they would select 1, and so on.   Anyone choosing 1 or 0 is considered by this study to be an organisational leader because they are at the very top of a large organisation or, in a smaller organisation where fewer layers may exist the actions of the respondent at this level in a smaller population have significant ability to influence the resilience state.

The term ‘reasonably established’ was defined for the purpose of this study as employing at least 100 employees and having existed for at least 5 years.  These figures are somewhat arbitrary: the figure of more than 100 employees was chosen as there was legislation that applied to UK organisations that had at least this number of staff in 2007 (BIS, 2008).  5 years was chosen as many businesses fail within their first 3-4 years (Bright Ideas, 2011).   It is acknowledged this is likely to skew results as those responding may work in organisations with higher levels of resilience as those that have not thrived or which may fail before they reach the five-year mark are excluding from the study.  As this study is intended to be useful to leaders of reasonably sized organisation this decision is acceptable, however it is noted that findings may be somewhat affected as a result.

As the aim of this questionnaire was to look at whether the presence of particular circumstances or factors might correspond with an inclination (or disinclination) for behaviours that contribute to organisational resilience, the questionnaire had to include a way of indicating each respondent’s propensity for resilience behaviour (PRB).  This was achieved by amending the questions in Stephenson’s tool to ask leaders to indicate their own propensity towards particular behaviours, rather than respond for their organisation.  More specifically:

Each leader had to be asked to indicate the strength of their inclination to undertake particular actions and responsibilities that contribute positively to organisational resilience in a manner that allowed a numerical PRB value to be determined for each respondent. As seen in the literature review, Stephenson’s resilience tool identified 13 key attributes of resilience, and her work (Stephenson, 2011) also includes the wording of each question posed to her respondents (included as Appendix 1).   A question for each identified attribute was created – using Stephenson’s wording where possible – to ask the respondent to determine how much responsibility they personally took for each attribute. An additional seven questions were added: five to cover attributes indicated by the Issue and Literature Review but not prominently covered by Stephenson’s tool, and two where Stephenson’s tool asked about two factors within the same grouping.  The full list of PRB attributes for which one question was taken or created is noted in Table 2.3.

On the questionnaires, the PRB questions were broken into four groups with other questions inserted between.   This was done to ensure the style of question changed frequently so as not to bore the responder, and to minimise the possibility of respondents manipulating their responses to PRB questions if they realised they were those that would be measured.  Notes on how the questions were group appear later in this section.

Table 3.1: List of PRB attributes noting where wording was available from the Stephenson tool

PRB Attribute Wording for questions guided by Stephenson (2011)?
Silo mentality Yes
Staff engagement & involvement Yes
Information & knowledge Yes
Innovation & creativity Yes
Devolved and Responsive Decision Making Items Yes
Internal & external situation monitoring & reporting items Yes
(Continuity) planning strategies Yes
Participation in Crisis Exercises Yes
Capability & Capacity of Internal Resources Yes
Proactive posture Yes
Capability & capacity of external resource items Yes
Recovery priority items Yes
Leadership, management and governance Yes
Impact of strategy on business No
Corporate values No
Knowledge management No
Innovation/acquisition discipline No
Understanding of key dependencies No

Each question required an answer from a five point rating scale where the response was weighted according to the scale (e.g. never = 1, rarely =2, sometimes=3, often=5, all the time=5).  The values of the PRB questions for each respondent were added together to provide a total PRB value for each respondent.

The PRB value provided an indicative basis to look at whether the presence of other circumstances or factors identified in the literature review might correspond with a higher inclination (or disinclination) for resilient behaviours.

It is important to note that this survey does not purport to provide statistically sound and robust evidence due to a number of limitations, some of which are noted below.  It simply seeks to offer indicative information that could be used as a basis for further research.

Limitations that suggest resulting data may only be used for indicative purposes and a basis for further research include:

  • Concerns around a leader’s willingness to complete a questionnaire for this work made it necessary to keep it as short as possible: where Stephenson’s tool uses several questions to ascertain an indicating value for each PRB factor, this survey did not.  This limits the value of the resulting data, but ensured a good response rate in a limited timeframe
  • The questions were not robustly tested in the same way they were for Stephenson’s tool.  However, the wording of the questions were guided by that tool, as well as the information noted in the section ‘Questionnaire Creation’ (e.g. avoiding questions encouraging ‘prestige bias’ (de Vous, 1991))
  • Convenience samples generally do not support statistical inference” (Schonlau, Fricker, & Elliott, 2002).  Due to short timeframes and limited resources, this survey was conducted with a convenience sample.
  • It is also noted that some questions intended for inclusion were removed during the testing phase when useful versions could not be determined.  For example:
    • the survey doesn’t ask if respondents preferred to grow their organisation organically or via mergers and acquisitions (important to Carmeli & Markham, 2011) because trial respondents suggested mergers and acquisitions could be organic
    • To determine whether leaders understood ‘organisational resilience’ as a disruption management system or a holistic strategic capability, two definitions were offered for the term, neither of which contained the words holistic or strategic to avoid prestige bias (see note C below)
    • No scale was available to measure an individual’s level of realistic optimism.  A simple question was therefore devised to provide an indication of the individual’s perception of their position (see note G below)

Bearing in mind the need to keep the questionnaire interesting and not group all question types together (Schonlau, Fricker, & Elliott, 2002), the survey was organised as follows:

  1. Questions 1,2,3, and 4 provide basic information on the organisation (non-qualifiers – i.e. those working in organisations less than 5 years old or with less than 100 employees – were disqualified based on their response to these questions)
  2. Question 14 seeks to establish the seniority of the responder
  3. Question 6 asks if they view ‘resilience’ as a business continuity-based or strategic capability
  4. Questions 7, 8, 15 and 17 indicate propensity to take action that increases resilience (PRB)
  5. Question 9 establishes whether the respondent makes use of the ‘resilience functions’
  6. Questions 10 and 11 ask about tenure in the current role
  7. Question 12 aims to find which responders believe they are ‘realistically optimistic’
  8. Question 13 seeks to establish motivation for organisational longevity
  9. Question 16 seeks to discover if responders require certain experiences for career progress
  10. Question 18 determines whether responders would accept constraints in return for resilience
  11. Question 19 asks an open questions about barriers encountered to resilience efforts

The intention was to use the data as follows:

  • Questions in A & B included criteria for establishing seniority within a qualifying organisation
  • For each respondent, the sliding scale answers from all D (see above) questions were totalled, giving each respondent an indicative value for their propensity to act for resilience.  These values would be compared with the other factors in A, C, E, F & G to see if there are any emerging trends.  Any such trends would be considered possible casual links worthy of further research that is outside the scope of this study
  • The remaining question sets, H, I, J and K, provide additional potential insight leaders’ thoughts on other identified gaps in the literature, and against which to examine PRB totals.

 

3.42 Resilience Business Unit Questionnaire

The qualifying criteria for this survey were that the respondent worked in an organisational department that considered its primary responsibility to be ‘resilience’ and that the word ‘resilience’ was the keyword in their department’s title.

Unlike the primary dataset above, this survey aimed to answer a simple question: what is the remit of an organisational department that appears to take responsibility for ‘resilience’ and what, if any, functions has it replaced or been merged with.

Once again it is noted that the resulting data was not intended to be statistically sound, but provide an insight as a basis for possible further research.

The survey was organised as follows:

  1. Questions 1, 2 and 3 provide basic information on the department and organisation (non-qualifiers – i.e. those working in organisations less than 5 years old or with less than 100 employees – were disqualified based on their response to these questions)
  2. Questions 4 asks about the scope of the department
  3. Question 5 asks if the organisations views ‘resilience’ as a business continuity or strategic capability
  4. Questions 6 and 7 seek to establish how the resilience department fits with other ‘resilience functions’
  5. Question 8 seeks to establish the department manager’s seniority
  6. Question 9 requests qualitative information on barriers to the resilience department
  7. Questions 10, 11 and 12 seek basic information on the organisation
  8. Questions 13 seeks to understand the extent to which the resilience department contributes to organisational resilience

The intention was to use the data as follows:

  • Comment on the scope and responsibilities of  “Resilience” business units
  • For each respondent, the sliding scale answers from all S (see above) questions will be totalled, giving each respondent an indicative value for the resilience department’s contribution to organisational resilience.
  • All other responses will be used qualitatively to seek trends on each individual area

 

3.43 Data Significance Thresholds

Where PRB values were compared with other data, correlation and significance thresholds were ascertained and interpreted as follows:

 

3.43(a) Correlation Co-efficient 

In the absence of an SPSS package, correlation coefficients were calculated using online calculator at: http://www.alcula.com/calculators/statistics/correlation-coefficient/.

The correlation measure was considered using Field’s (2006) guidance that “±0.1 is a small effect, ±0.3 is a medium effect and ±.0.5 is a large effect”.  Thus, correlations greater than 0.3 were considered interesting and those larger than 0.5 were considered valuable.

3.43(b) Statistical Significance

The statistical significance of each correlation was calculated using an online tool provided by an academic at California State University.  It uses the correlation value and the sample size to provide p-values at http://www.danielsoper.com/statcalc3/calc.aspx?id=44 .

The significance of the correlation was also considered using Field’s (2006) guidance that “when the probably falls below 0.5 (Fisher’s criterion) we accept this as giving us enough confidence to assume the test statistic is as large as it is because our model explains a sufficient amount of variation to reflect what’s genuinely happening”.  Thus p-values below 0.5 were considered to render a correlation co-efficient as significant as, at this level, the statistics suggest the result did not happen purely by chance.

 

3.44 Reaching Qualified Respondents 

Work began on reaching respondents qualified to participate in primary research long before the proposal for this study was written.  Consideration was given to the significant challenge of connecting with a sufficient pool of suitably senior organisational leaders.  A strategy was created to overcome this and executed during the study:

Personal Contacts.   The researcher was fortunate to work with interested, helpful (and kind!) senior leaders in a very well known organisation, and to know others.   Personal conversations elicited offers of a questionnaire response and, in some cases, offers to forward requests for responses to their own contacts in other organisations.  Similar offers resulted from conversations with individuals working Resilience departments of other organisations.  A visiting lecturer of Cranfield University also contacted several senior organisational leaders on my behalf, vouching for the credibility of the study and asking for responses.

The Blog.  The blog at www.goldordust.wordpress.com was created to achieve several aims.  As well as ordering the researcher’s thoughts in the pre-writing stage, it created a small community of interested persons and subject specialists.  These individuals assisted during the literature review by offering text suggestions and opinions that were shared via the blog response system (these can be seen online) and via personal emails to the researcher.

Linked In.  The professional networking site at http://www.LinkedIn.com was utilised in several ways.  The researcher added clear objectives to her profile headline, ensuring all contacts actively using Linked In were aware of her work.  She joined a number of relevant Linked In groups several months before the primary research began in order to use their community board to request responses to the relevant questionnaire. For example, a request for qualified persons to complete the Organisational Leader questionnaire was posted on Linked In’s “CXO” group community boards, which is aimed at Chief (Something) Officers of large organisations.  A similar request was made for the Resilience Department questionnaire on the ‘Resilience Professionals’ group.  About 20 Linked In groups were joined and utilised in total.   For the ‘resilience department’ strand, targets were also identified by searching for individuals with the word ‘Resilience’ in their job titles.

 

3.6 Chapter Summary

This study is exploratory.  The data findings are intended to be indicative, to add to academic and practitioner discussions, and provide a possible basis for further research.

Primary research was based on questions arising from the Literature and Issue review of the previous chapter.  It was undertaken via a blog at www.goldordust.wordpress.com and, two online questionnaires.

Questionnaires were created to survey (a) individuals working in ‘Resilience Departments’ on the scope of their unit, and (b) organisational leaders to examine whether any indicative, casual links could be made between an individual’s PRB and specific factors or circumstances. Respondents were contacted in many ways including personal email, email from a trusted colleague, via the study’s blog or a reader of that blog, and from posts on online forums that included relevant Linked In groups.

Data analysis frameworks and statistical significance thresholds were created before the questionnaires were launched online.


[1] Author of many Harvard Business Review articles on resilience and the book Resilient Organisations (Valikangas, 2010)

[2] Professor of Resilience at the US Naval Postgraduate College

[3] Former researcher at ResOrgs, PhD author (Stephenson, 2011) and founder of Stephenson Resilience

[4] Obtained during conversation between researcher and dissertation supervisor

[5] www.LinkedIn.com has an ‘advanced search’ where keywords can search job/department titles

Advertisements
Posted in: Uncategorized