Skilljar Coffee Chats showcase different ways customers are using our platform, including demos with special guests. This month’s Customer Education Coffee Chat, hosted by Cutler Bleecker, Skilljar’s Customer Training Manager, featured a conversation with Kpayah Tamba, Sr. Director of Certification for Okta.
Kpayah’s background is in electrical engineering and he came to credentialing somewhat by accident, although he always had a passion for assessments and measurement.
There are a lot of ways to gain opportunity in life, but what testing does in society is it really helps us to find and hire the right people, including those from underserved, underrepresented communities who can earn opportunities by getting certified. Testing can really open doors for people.— Kpayah Tamba, Sr. Director for Certification, Okta
Kpayah shared how Okta, a leading cloud identity provider, integrates Skilljar with other platforms such as Examity, Caveon, and Credly to develop, secure, and deliver their certification exams and issue digital badges. Okta’s vision is to accelerate a world where everyone can safely use any technology. So naturally, they are very passionate about connecting people to resources in a simple, seamless way.
Okta offers five global certifications targeted at customers, partners, and employees: Professional, Administrator, Consultant, Developer, and Architect, with many more pending on their product roadmap.
So grab your coffee and read the recap!
Building high-stakes assessments
Okta’s methodology for building certification exams includes the following stages:
Analysis
Like most certification programs, Okta starts every project with a job task analysis where they work with experts across Okta. They have recently shifted to a more balanced approach incorporating insights from external experts such as customers and partners.
It is important for us to know exactly how our customers are using Okta products, not just how we think they should be using our product.— Kpayah Tamba
Design
They go through this information and begin the blueprint phase, figuring out what topics to cover, the weights for each section, and which content (and how much) goes into what parts of the exam.
Development
Okta’s exam development team includes certification program managers, but these people are not Okta product experts. So they train them on how to write exam questions using authoring tools and item bank/item ordering solutions from Caveon Scorpion. As facilitators, Kpayah’s team spends a lot of time working through the exam development process with these experts. They hold remote synchronous meetings over different time zones that are about three hours in length. They have a data structure for writing questions and set quotas for the amount of items they want to write in these development workshops. Then they have a group peer review to gain feedback. They repeat the process until they get the amount of content they need.
Standard Setting
- How can you gauge how people are performing on exams?
- What score should be achievable in what time limit?
- How do you know if a particular task is too easy or too difficult?
You need standards.
Sometimes even judges may “misjudge” how certification candidates would perform on tests. It’s hard to put yourself in the shoes of a qualified candidate, understand the things they know and when they’re trying their best, and predict how a minimally competent person will perform. Okta brings judges together in workshops to make necessary adjustments to the exams based on ongoing assessment with these factors in mind.
Delivering exams securely
Security-centric design and development
- How can you protect the integrity of your exams?
- How can you ensure that the people who are getting certified actually know what they’re supposed to know?
For Okta, this starts with a security-centric design development approach. This means recruiting judges who sign NDAs (non-disclosure agreements) and have some clear guidance on what they can and cannot do regarding discussing the exams they’ve worked on.
For example, not disclosing or sharing example information any way in a public forum, or even privately. They use Caveon Scorpion for item writing and assign different judges to different objectives based on need.
Log settings within Caveon Scorpion can tell if a person tried to take a screenshot during the exam or if the exam was not in full-screen mode at the time of taking. These things are tracked as security measures within this platform.
Additionally, because they randomize questions, every item is delivered uniquely. Using discrete option multiple choice, they can determine any leaked questions through a complex process using watermark patterns.
Discrete option multiple choice (DOMC):
Instead of showing all of the answer options at one time, as is usually done, options are randomly presented one at a time. For each presented option, the test taker chooses YES or NO indicating whether he or she thinks it is the correct one. Regardless of whether a question is answered correctly, based on a configurable probability setting, an additional option may be presented before the next question is displayed.
Source: Caveon
Detect Theft (Watermarking):
If an individual attempts to share part of an item they see on an exam, and post it online, a testing program can trace the stolen content back to the administration instance. Okta is using watermarks with the DOMC item type. The white spaces on the screen tend to have the delivery ID repeated, and the delivery ID is unique per exam delivery.
Strong candidate agreement
“Security was an easy thing for everyone to buy into, being that Okta is all about protecting identity. We wouldn’t be able to gain the trust of our customers if they felt like people were getting certified without doing the work.” – Kpayah Tamba
Okta wants to make sure that all candidates understand the “do’s” and “don’ts” of exam taking. They have crystal clear rules and use several outlets to drive awareness of their Certification Program Agreement (CPA), which users click to accept when taking the exam. Okta publicly displays their CPA as a preparation resource on their certification webpage for each course. This is built into their test registration process through Skilljar and users receive an email with the agreement link as well. The test proctor must ensure the user has accepted the CPA during the admissions process.
Having such a strict process serves as a deterrent to the user and should they try to commit any form of exam fraud, strict action is taken.
Strict proctoring rules
Proctoring enables us to double down above and beyond our candidate agreement and secure question build and asset rules; it creates more trust in our experts, tools, and program assets.— Kpayah Tamba
Okta partnered with Examity to ensure users could take exams 24/7 and remotely. Before the widespread acceptance of remote proctoring as a secure way to proctor high stakes exams, examinees had to travel long distances to get to testing centers. So they wanted to create ease in exam access and reduce their carbon footprint so users could take the test whenever and wherever they want.
To enable this, they needed strict online program proctoring to ensure security. From the moment the exam session starts, the session is recorded, the proctor goes through a set of scanning processes for live online monitoring. Examity also enables ID verification of the users.
Putting it all together with Skilljar
Skilljar gives you as a customer a blank canvas to paint your program. You decide what it will look like. Skilljar is the heart and soul of our certification program.— Kpayah Tamba
Here is what the certification ecosystem for Okta looks like on the front end:
Okta – https://certification.okta.com/
Skilljar – powers education and certification platform, the “heart and soul of the program”
Caveon – test development and delivery; results sent back to Skilljar
Examity – security/online proctoring
Credly – badging
It looks complicated but the key is to focus on an experience where users only have to sign in once with SSO to get through the entire process.
They use Okta single sign-on for test takers to access Okta’s platform through Skilljar. They can read about an exam, which is essentially a course they can take, and schedule the exam through the Examity single sign-on. At this point, it’s seamless for them – the UI changes, but they don’t have to sign in again.
Users take an exam, which is delivered from Caveon. Once they take the exam, they’re done and the exam results are sent back to Skilljar. Okta has rules in Skilljar so that if a person passes the exam, that triggers an event in Credly, which issues a digital badge.
In closing, Kpayah offered this advice to anyone looking to launch a certification program:
Set the right expectations with your leadership team; make sure you’re all on the same page. It’s not a case of if you build it they will come. You have to market it to them. You have to drive demand for your program.
Read more blog posts
Ready to take Skilljar for a spin?
Take an interactive tour of Skilljar, or book your demo with our team.