Qualitative Software Engineering Research Methods

Official course outline (PDF)
Instructor: Jonathan Sillito

Week 1 - January 10-14

Readings
No readings for week 1.
A1. Sketch out a sample project
Create a short description of a question/theory along with a description of a study you could conduct to answer the question or verify the theory. Due at the start of class. Come ready to discuss your project.
 

Week 2 - January 17-21

Readings
Timothy C. Lethbridge, Susan Elliot Sim and Janice Singer. Studying Software Engineers: Data Collection Techniques for Software Field Studies. Empirical Software Engineering, 10, 311-341, 2005.

Magne Jorgensen. Forecasting of software development work effort: Evidence on expert judgement and formal models, International Journal of Forecasting, Volume 23, Issue 3, July-September 2007, Pages 449-462.

 
 

Week 3 - January 24-28

Readings
Victor R. Basili, Gianluigi Caldiera and H. Dieter Rombach. The Goal Question Metric Approach. Encyclopedia of Software Engineering, 1994.

Howard Becker. How I Learned What A Crock Was. Journal of Contemporary Ethnography, 22(1):28-35, 1993.

 
Project
Project proposal due in class on January 25 (10%). The proposal should be no more than one page and should have the following sections.
  • Team members
  • Research context
  • Venue (conference, workshop or journal)
  • Research question(s)
  • Plan for data collection

    • Week 4 - January 31 - February 4

    Readings (for Thursday's class)
    Bertram, D., Voida, A., Greenberg, S. and Walker, R. Communication, Collaboration, and Bugs: The Social Nature of Issue Tracking in Small, Collocated Teams. In Proceedings of the ACM Conference on Computer Supported Cooperative Work - ACM CSCW'2010. ACM Press, 10 pages, February.
    Follow up questions
    Follow up on gloss phrases (also push for active voice)
    A: "The estimates are combined and …"
    Q: "Can you tell me what that involves?"

    Follow up on points of confusion (generally: clarify)
    A: "That task was very difficult to estimate"
    Q: "What made it difficult to estimate?"

    A: "They think the estimates are fine, but …"
    Q: "Who thinks the estimates are fine? What makes them feel that they are fine?"

    Q: "Can you tell me what happened, step by step?"

    Follow up on labels
    A: "My team uses the planning-game."
    Q: "Can you step through a recent example?"

    Elaboration questions
    A: "Those estimates were were very accurate."
    Q: "That sounds interesting, can you tell me more about that?"

    A: "Sometimes the estimates are ignored."
    Q: "Can you give me an example of a time when the estimates were ignored?"

    Continuation questions
    Q: "Then what happened?"

    Steering questions
    Q: "You were talking about the information you used to make that estimate. Can you tell me more about …?"

    Week 5 - February 7 - 11

    Reading (for Tuesday's class)
    Andrew J. Ko, Robert Deline and Gina Venolia. Information Needs in Collocated Software Development Teams. International Conference on Software Engineering, 2007.
    Reading (for Thursday's class)
    Andrew Begel and Beth Simon. Novice Software Developers, All Over Again. ICER, 2008.
     
     

    Week 6 - February 14 - 18

    Readings (for Tuesday)

    Barthelemy Dagenais, Harold Ossher, Rachel K. E. Bellamy, Martin P. Robillard, Jacqueline P. de Vries. A qualitative study on project landscapes. In Proceedings of the ICSE CHASE workshop, 2009.

    Readings (for Thursday)

    Jonathan Sillito, Gail C. Murphy and Kris De Volder. Questions Programmers Ask During Software Evolution Tasks. In Proceedings of SIGSOFT/FSE, 2006.

    Assignments
    Assignment 1 is due by February 18th, COB

    Reading week - February 21-25

    No lectures or tutorials.

    Week 7 - February 28 - March 4

    Readings (for Tuesday)
    Martin P. Robillard, Wesley Coelho, and Gail C. Murphy. How Effective Developers Investigate Source Code: An Exploratory Study. IEEE Transactions on Software Engineering, 30(12):889-903, 2004.
    Readings (for Thursday)
    Margarete Sandelowski. Whatever Happened to Qualitative Description. Research in Nursing & Health, 23:334-340, 2000.
     
     

    Week 8 - March 7 - 11

    Readings (for Thursday)
    Treude and Storey. How tagging helps bridge the gap between social and technical aspects in software development. ICSE, 2009.
     
       

    Week 9 - March 14 - 18

    Readings (for Thursday)
    Assignment
    Assignment 2 is due in Tuesday's class
     

    Week 10 - March 21 - 25

    Slides on "further analysis" as discussed in class

    Readings (for Tuesday)
    Eisenhardt, Kathleen M. Building theories from case study research. Academy of Management Review, 14:4, pp. 532-550, 1989.
    Readings (for Thursday)
    Amy Voida, Rebecca E. Grinter, Nicolas Ducheneaut, W. Keith Edwards and Mark W. Newman. Listening in: practices surrounding iTunes music sharing. In Proceedings of CHI, 2005.
    Andrew J. Ko, Htet Htet Aung, and Brad A. Myers. Eliciting Design Requirements for Maintenance-Oriented IDEs: A detailed study of corrective and perfective maintenance tasks. In Proceedings of ICSE, 126-135, 2005.
     
     

    Week 11 - March 28 - April 1

    Readings
    Paul Dourish. Implications for Design. In Proceedings of CHI, 2006.
    Greenberg, S. and Buxton, B.Usability Evaluation Considered Harmful (Some of the Time). In Proceedings of the ACM Conference on Human Factors in Computing Systems - ACM CHI'08.

    There will be no class on March 31st
    Assignment
    Assignment 3 is due in Tuesday's class

    Week 12 - April 4 - 8

    The USRI survey will be conducted on April 5th
     
    Project
    Project presentations (20%)
    Parallel Dev group (April 7th)
    API Usability (April 7th)

    Week 13 - April 11 - 15

    Readings
     
    Project
    Knowledge sharing (April 12th)
    Search and recommenders (April 12th)
    GUI testing (April 14th)
    Project report due (April 15th - COB, 40%)