ASSESSMENT 2 GUIDE
COMM1190
Data, Insights and Decisions
Term 2, 2024
Assessment 2: Customer churn project
Description of assessment tasks
This is a group assessment with reporting being done in two stages. Students will be assigned to groups in Week 5 when this documentation is released. While reporting in done in two stages students are encouraged to commence their collaboration within their group early in the process before the submission of the Stage 1 individual reports.
Stage 1: Complete the 2-page individual task using a data set specific to your Assessment 2 project group.
This first-stage submission contains key inputs into the group work that will result in the single group report produced in Stage 2. Students who do not submit a complete, legitimate attempt of this
assessment will not be awarded marks for Stage 2.
This individual task will be separately assessed together with the Stage 2 group report, and associated marks will be available together with Stage 2 marks. Because of the nature of the relationship between the Stage 1 and 2 tasks, you will not receive your Stage 1 marks before submitting Stage 2.
Stage 2: As a group, use the results from Stage 1 to produce a report for the Head of Management Services.
You will use R to explore a dataset that includes the pilot data together with extra observations and variables (see attached Appendix A Data Dictionary). The pilot data are common to all students, but the extra observations will vary across students according to their SID as they did in Assessment 1. A group-specific data set will be determined by nominating the SID of one of the group members to
generate the single data set used by all group members in both stages of the project. Details for obtaining the personalized group-specific data set will be provided on Moodle
Your Stage 2 group mark will be common to all students in your group who have submitted a complete, legitimate attempt of Stage 1.
Note: The course content from Weeks 4, 5, and 7 will be of particular relevance to completing this Assessment.
Context of assessment tasks
The Head of Management Services of Freshland, a large grocery store chain in Australia, has made use of your updated report (from Assessment 1) to deliver a presentation to the Senior Executive Group.
→ Access this presentation via your Moodle course site.
Based on this initial analysis and recommendations, approval has been given for further analysis of customer loyalty and churn using an expanded data set. The core task will involve a comparison of predictive models and subsequent recommendations on how to use and improve these to inform. future retention policies.
The analysis in the presentation to the Senior Executive Group was based on the initial pilot data set which was used by the intern to produce the initial report and was part of the data provided to you with Assessment 1. These data
have now been extended, with extra variables being added. These extra variables are:
ltmem =1 if member ≥ 3
mamt1 Average monthly expenditure ($) in first 6 months of previous year (2023)
mamt2 Average monthly expenditure ($) in second 6 months of previous year (2023)
fr1 Frequency of monthly transactions in first 6 months of previous year; 1 (low) 2 (medium), 3 (high)
fr2 Frequency of monthly transactions in second 6 months of previous year; 1 (low) 2 (medium), 3 (high) rind XYZ risk index in the form of a predicted probability of customer churn
You and your team have been tasked with investigating alternative algorithms for predicting customer churn. Given the structure of data that has been made available, you have been advised to define churning to be when a customer has previously had non-zero transactions for at least 6 months but then has zero transactions in the next six-month period. The outcome of interest is the binary variable churn. Given the available data, an observation for a customer will have Cℎurn = 1 if mamt1 > 0 & mamt2 = 0 and Cℎurn = 0 if mamt1 > 0 & mamt2 > 0.
The Head of Management Services has given you authority to use your expert judgment to make the necessary modelling choices but has outlined an overarching research plan for you and your group to follow:
. Currently, Management Services has a basic regression model (details below) that can be used to predict future customer expenditure for members of the rewards program. It has been suggested that this could be used to generate a risk index where those with predicted expenditures that are low relative to actual
expenditures being deemed as high risk of no longer shopping at the store.
. However, there were suggestions that the existing model could be improved as a predictor of expenditures and your group has been asked to evaluate a range of model extensions.
. The current focus is on predicting churn. Based on the performance of the alternative models in predicting expenditures, choose one and analyse whether it also performs well in predicting churn.
. An analytics firm, XYZ, that uses proprietary predictive methodology has offered a trial of their products by
providing a predictor of churn. Your evaluation of predictive performance should include a comparison of this predictor with that generated by your chosen regression-based predictor.
. Based on this analysis, make recommendations on using such algorithms in initiatives targeting customers at risk of churning with the aim of retaining them as loyal customers.
o Notice that any recommendation to employ the predictors of the analytics firm would involve additional cost compared to a method produced in-house by Management Services.
o In addition, any decision to employ the predictors of XYZ will not include documentation of the methodology used to generate the predictions.
o It might also be that you conclude that neither predictor is adequate and that it would be appropriate to explore alternative predictors or approaches. You are not expected to explore such alternatives.
The base regression model used for prediction by Management Services for the ith customer takes the following form.
mamt1i = β0 + β1 agei + β2femalei + β3metroi + β4 ltmemi + ui.
Each individual group member will use the group data common to all group members to compare the predictive performance of this base model with one of the following extended models:
A: add age squared to base model
B: add regional dummy variable to base model (regional = 1 if location = 2; = 0 otherwise) C: to the base model add age squared and replace variable ltmem with variable member
D: to the base model add age squared and regional and replace ltmem with member.
In the case of groups with less than four members prioritize A and C, with B and D being optional. So, a group of 3 would choose A, C and one of B or D.
Approach to the assessment tasks
Stage 1 instructions
Compare the predictive performance of the base model with the indicated extension, using the pilot data for
estimation (training) and the non-pilot data as the testing sample. Provide a justification of this split and document any modifications to the samples used due to missing data and/or outliers.
Complete the questions and table in the attached Appendix B Individual Report Template. All questions must be attempted, and the report submitted by the due date for a student to qualify for the Stage 2 mark received by their group.
Stage 2 instructions
The group task is to use the Stage 1 results to inform. the choice of data and preferred predictive model to use in predicting churn. Recall that the idea is to associate predicted expenditures that are low relative to actual
expenditures in one 6-month period with a high probability of churning in the subsequent 6-month period. The Head of Management Services has suggested classifying a customer as someone predicted to churn if their regression
residual (actual minus predicted expenditure) was below a cutoff of, say, the 25th percentile of expenditure residuals in the pilot data. Ultimately, it is up to you how you proceed, but this does seem like a sensible approach to consider.
The risk index provided by the analytics firm XYZ provides an alternative predictor of churning. The Head of
Management Services is especially interested in the relative predictive performance of this index as the use of this in the future would involve extra costs to the firm.
Recall that the focus of the analysis and subsequent recommendations arising from the group work is to inform.
management about identifying customers at risk of churning and, hence, potential retention strategies. The Head of Management Services has outlined an overall strategy on how to proceed but ultimately there are other details that
have been left unspecified and it is the responsibility of you and your group to make the associated decisions. These are decisions that require judgement and will not necessarily be right or wrong. What is important is that the decisions are supported by sensible arguments.
Your report should explain your strategy, subsequent analysis and the recommendations that follow. Include a critical evaluation of the strengths and weaknesses of the alternative predictors and any potential improvements, as this will be an ongoing focus of management. Use the Assessment 2 marking rubric for the Stage 2 group submission as a
tool to check your work before submission and to ensure that you have addressed the assessment task in full.
Structure:
Your Stage 2 submission should take the form. of a report to the Head of Management Services. Use the initial report provided in Assignment 1 to guide you. Your Stage 2 Report should be approximately the same length and structure as that report, although the section headings will likely change, and the emphasis on graphical presentations is likely to be less. Ultimately decisions about presentation are to be made by your group.
Your group must also submit a separate file containing R code used to conduct your analysis and generate your
visualisations. No marks will be associated with this code file, but your submission will be deemed incomplete and given a mark of zero if such a file is not included.
Overall, the assessment is designed to help you develop your skills in using R for data analysis and in communicating insights from such exercises.
Writing support
The following links will take you to resources for writing support and study skills:
. Writing Skills Support
. Academic Skills One-to-One Consultations
Submission instructions
. Submit your Stage 1 report using the Turnitin assessment submission link on Moodle.
. Submit your Stage 2 report and code file as separate documents using the Turnitin assessment submission link on Moodle. You are free to choose the structure of the code file and it is not subject to a word limit.
. Late submission will incur a penalty of 5% per day or part thereof (including weekends) from the due date and time. An assessment will not be accepted after 5 days (120 hours) of the original deadline unless special
consideration has been approved. For further information please refer to Policies and Support.
. Special consideration will only be granted in the case of serious illness, misadventure, or bereavement, which must be supported with documentary evidence. In these circumstances, students must apply forSpecial Consideration. Because of the sequential nature of the assessment tasks, it is very difficult to allow
extensions without impacting the academic integrity of the assessment. As such this course does not use the short extension process that you may have seen in other courses. Moreover, in the event you are granted
special consideration due to exceptional circumstances precluding you from completing the assessment task on time you are likely to have your final exam reweighted rather than being granted an extension.