Career Counselling to Career Engineering: Making the Job Hunt Less Random

It’s recruiting season again. As we approach the thick of it in the early spring, there’s only one question on everyone’s mind: Will I find the right job? As students of Duke’s Fuqua’s MQM program, we pride ourselves on answering difficult questions with data, and the above question is no exception. We scanned to thousands of job postings, requirements, and qualifications available on the internet, and attempted to replace the random job search process with something a little more data-driven.

This project was a joined effort between me and two other classmates Atreyo Sinha and Mayank Tulsiania to figure out a smarter way to land that dream job. It was selected among over thirty projects to be featured in MQM Winter Competition Final Rounds and won 2nd runner-up prize.

Where’d we find all this data?

Glassdoor, mostly. We used the Selenium Webdriver package in Python to automatically scrape job postings from the website, and ended up with 3000+ data points from over a 100 different companies. These companies were identified on the basis of employment reports from one-year masters’ programs at Duke, MIT, and UCLA.

We focussed on positions of interest to early-career candidates with a good business-technical background split: ‘consultant’, ‘analyst’, ‘business intelligence’, and ‘data scientist’ were our keywords of choice. Once we had our entries, we were ready to begin analysing.

How’d we make sense of it?

We started by cleaning the data. This included weeding out entries in foreign languages and search queries that returned empty job descriptions, before breaking down the job postings into sub-categories. We separated each posting into smaller components including years of experience, responsibility, requirements, and technical skills.

Using Python’s NaturalLanguageToolKit package, we were able to deconstruct long, complex sentences in the job descriptions into smaller, bite-sized parts. To do this, we applied text analysis techniques like tokenization and n-grams to split sentences into smaller elements. We also used term frequencies to count relevant tokens and word clouds to visualize the popular ones. The end goal for us was to obtain a better understanding of what skills employers are looking for across different industries and experience levels.

Here’s what we found.

Our Word Cloud, shown below, shows us the most common words associated with job searches across the board. Before we propose our novel job search process, we wanted to share what employers out there are generally looking for in the Spring of 2020.

As is evident, there’s a major push across industries for new skills like data science and machine learning, with solid knowledge of processes, reporting, and solutions. Most roles have a strong team component as well, which highlights the collaborative nature of today’s workplaces, and the importance of being a good team player. Other words that feature heavily are EEOC-designated protected class terms such as veteran status, race, color etc.

Creating a smarter job search process.

To apply our more specific findings, we decided to propose a new, data-driven process which guides applicants from beginning to end.

Select an industry.Identify your experience level.Check out the salary.Check out the skills.

The order of this is important, since we wanted to place immutable factors such as experience level (unlikely to change in a short time) earlier in the process to filter out unsuitable postings.

Here’s what we found after sorting by industry and experience level:

Overwhelmingly, companies seem to have a broad definition of the word ‘analyst’, with finance classifying it as an entry-level position, and tech, consulting, and consumer retail firms seeking 2–5 years of experience for the same role. In terms of overall highest experience level, ‘manager’ roles in consulting firms seem to start off at the 4-year mark, and consulting in general seems to have the highest base requirement for work experience at around 1.5 to 2 years minimum. Jobs in the consumer retail industry seem to require the least minimum work experience, although the overall experience requirements are pretty standard across the board. Going through these results give us an idea of what roles applicants would be good fits for.

Once ideal roles are identified, we thought it’d be appropriate to talk money. Here’s an analysis of the salaries for different roles by industry:

The reason we placed salary ahead of skills is simple. Certain jobs might pay more for a similar roles and similar experience levels, which allows applicants to select the highest paying options before looking to identify the skills they’ll need. The word ‘engineer’ seems to produce a magical boost in salary, with the clear highest maximum average at around $170k. In technology and consulting, the ‘business analyst’ role has a low minimum average salary, while research analysts and managers seem to rake in quite a bit more. Finance is the highest paying industry by average maximums, but a little birdy told us that their employees aren’t the happiest…

Which moves us along to the most important section of them all, the big finale, SKILLS! Now that you’ve identified what you want to do, you’ve gotta know if you have what it takes. An examination of the required skills lets applicants develop the skill, and in case they can’t learn a bunch so quick, allows them to restart the process and find more suitable roles.

We’ll start with some technical skills.

As you can see, the most common hard skill for all analytics jobs is Python, the trendy, new-age coder’s language. Finance and consumer love a bit of SQL, while consultants still swear by Microsoft Excel and PowerPoint. SAS is another language used only in the consumer retail space, which is consistent with the reporting-based nature of the job. Overall, Tableau is another important hard skill that employers across all industries look for.

Moving on to analytical skills, here’s what we found:

Modeling and visualization are key across finance and consulting, while statistics and mathematics find a mention across all industries. Most early level jobs also consider reporting to be an essential job skill, with a ~16% token rate in consulting jobs.

But hard skills and analytical skills, while useful, only get you so far. Here’s a few of the softer skills that employers look for in candidates.

Communication is a top skill in every industry, which makes sense because collaboration and team work is essential to any organisation with a heavy focus on people. As industries look to disrupt and create out-of-the-box solutions, innovation also become a massive component of work culture. And then there’s the timeless classic, leadership, particularly in consulting, where it fits in the top three soft skills.

So, what are we trying to tell you?

Not much, really. The job search will still be challenging and tedious, but we think that using data to smartly go about the process will make it that much easier and more rewarding. We’re not claiming to reinvent the wheel, but we are providing a data-driven solution for our peers who are faced with similar conundrums. Our insights prioritize finding the perfect job, as opposed to simply finding a job.

Putting together the results of our various text analyses, we find that employers tend to look for similar things across the board when it comes to specific positions and industries. There’s no reason why you shouldn’t be able to methodically reverse engineer the path to your dream job.

As thought leader and business expert Frank Sonnenberg so eloquently says, “If you don’t know why you’d hire you, neither will they”.