MCQs

Disclaimer:

Educational Use Only:

The website links provided are to be used strictly for educational and knowledge-gaining purposes only.

No Misuse:

Do not misuse the links or the information contained therein for any commercial, illegal, or unethical activities.

Respect for Websites and Businesses:

Respect the intellectual property and business interests of the websites and their owners. Do not attempt to alter, copy, or redistribute content without proper authorization.

Acknowledge Value of Knowledge:

Recognize and value the knowledge and resources shared by these websites. Use the information responsibly and ethically.

No Endorsement:

The provision of these links does not imply endorsement of the linked sites, their content, or their products and services. Users are responsible for evaluating the suitability and accuracy of the information.

User Responsibility:

By accessing these links, you agree to use the information at your own risk and to comply with all applicable laws and website terms of use.

Overall MCQs for Reference

Which of the following best describes A/B testing?
a. Comparing two or more models on training data
b. Comparing two versions of a system to see which performs better
c. Selecting the model with the shortest training time
d. Running cross-validation on a dataset
Answer: b. Comparing two versions of a system to see which performs betterseldon+1
What does the “A” in “A/B testing” most commonly refer to?
a. Algorithm
b. Alternate version
c. Control or original version
d. Automated process
Answer: c. Control or original versiondatalemur
What is a primary goal of A/B testing in machine learning?
a. To perform data preprocessing
b. To train a new model
c. To determine which model or system yields better performance
d. To visualize data distributions
Answer: c. To determine which model or system yields better performanceseldon+1
In A/B testing, what does the “B” group usually represent?
a. Baseline
b. Backup model
c. New version or treatment group
d. Bad data
Answer: c. New version or treatment groupgithub+1
Which statistical method is typically used in traditional A/B testing to compare groups?
a. Principal Component Analysis
b. t-test or z-test
c. K-means clustering
d. Decision Tree
Answer: b. t-test or z-testgithub+1
Which metric should you select before starting an A/B test?
a. Cost of ML model
b. Metric that measures success (e.g., conversion rate, accuracy)
c. Memory usage
d. Data shape
Answer: b. Metric that measures success (e.g., conversion rate, accuracy)statsig+1
How are test subjects usually assigned in an A/B test?
a. By age
b. Randomly
c. By performance
d. Alphabetically
Answer: b. Randomlydatalemur
What is the main risk of running multiple A/B tests simultaneously?
a. Losing data
b. Increased chance of Type I error (false positives)
c. Training overload
d. Duplicated models
Answer: b. Increased chance of Type I error (false positives)datalemur
What does “statistical power” refer to in A/B testing?
a. The ability to reject the null hypothesis when it is true
b. The ability to correctly detect a real effect or difference
c. Number of users in an experiment
d. Number of servers used
Answer: b. The ability to correctly detect a real effect or differencedatalemur
What is a “null hypothesis” in the context of A/B testing?
a. There is a significant difference between A and B
b. There is no difference between control and treatment
c. The model does not use data
d. The sample is not random
Answer: b. There is no difference between control and treatmentdatalemur
Which of these is a best practice for determining sample size in an A/B test?
a. Decide sample size after the experiment
b. Use only the largest dataset available
c. Calculate sample size based on desired power, effect size, and significance level
d. Randomly select any sample size
Answer: c. Calculate sample size based on desired power, effect size, and significance leveldatalemur
Why are confidence intervals important in A/B testing?
a. They determine training time
b. They measure uncertainty in estimated effects
c. They show how scalable the test is
d. They compare features
Answer: b. They measure uncertainty in estimated effectsdatalemur
What is “bucketing” in A/B testing?
a. Binning all test data for cross-validation
b. Assigning participants to groups based on predefined criteria
c. Running models in parallel
d. Splitting datasets at random points
Answer: b. Assigning participants to groups based on predefined criteriadatalemur
If an A/B test runs for too short a duration, which is a likely issue?
a. Low statistical power
b. More significant results
c. Overfitting
d. Too many variants
Answer: a. Low statistical powerdatalemur
What is “family-wise error rate” in the context of A/B testing?
a. Error margin within family of algorithms
b. Probability of making at least one Type I error in multiple comparisons
c. Grouping test results by gender
d. Summing errors for all users
Answer: b. Probability of making at least one Type I error in multiple comparisonsdatalemur
Which method is commonly used to control family-wise error rate?
a. Random Forest
b. Bonferroni correction
c. Clustering
d. KNN
Answer: b. Bonferroni correctiondatalemur
What is a control group?
a. The group receiving the new variant
b. The group with the highest metrics
c. The group exposed to the original system
d. The group with corrupted data
Answer: c. The group exposed to the original systemgithub+1
In which scenario would you consider Multi-Armed Bandit (MAB) over traditional A/B testing?
a. When you need clear statistical guarantees
b. When you aim to minimize regret and maximize reward during the test
c. When you have no metrics
d. When you want only visualizations
Answer: b. When you aim to minimize regret and maximize reward during the teststatsig+1
How does a Bayesian A/B test differ from a frequentist A/B test?
a. Ignores prior beliefs
b. Updates belief continuously and can stop early
c. Uses only fixed sample size
d. Never provides confidence intervals
Answer: b. Updates belief continuously and can stop earlystatsig
What does “segmenting users” in A/B testing involve?
a. Creating unique datasets
b. Dividing users by characteristics such as age or device
c. Changing the sample size
d. Merging control and treatment
Answer: b. Dividing users by characteristics such as age or devicedatalemur
What is the main limitation of traditional A/B testing in ML deployments?
a. Equal traffic split subjects half the users to the possibly worse variant
b. Only two models can be compared
c. Requires deep learning
d. Needs massive datasets
Answer: a. Equal traffic split subjects half the users to the possibly worse variantseldon+1
Which of the following is an adaptive alternative to A/B testing for ML model experiments?
a. Cross-validation
b. Multi-Armed Bandit
c. Bagging
d. Boosting
Answer: b. Multi-Armed Banditseldon+1
Which key factor helps to determine the minimum detectable effect size in an A/B test?
a. Model accuracy
b. Desired statistical power
c. Learning rate
d. Number of clusters
Answer: b. Desired statistical powerdatalemur
What does it mean for an A/B test result to be “statistically significant”?
a. The observed difference is likely due to chance
b. The observed difference cannot be explained by random variation alone
c. The models are not interpretable
d. The difference can be ignored
Answer: b. The observed difference cannot be explained by random variation alonedatalemur
In machine learning A/B testing, which outcome is usually reported to stakeholders?
a. p-value and confidence interval
b. Data types
c. Model parameters
d. Loss values only
Answer: a. p-value and confidence intervalstatsig+1
Which experimental design ensures each user only experiences a single variation?
a. Full factorial
b. Between-subjects design
c. Within-subjects design
d. Nested cross-validation
Answer: b. Between-subjects designdatalemur
What is a “Type I error” in the context of an A/B test?
a. Failing to detect a real difference
b. Detecting a difference when none exists
c. Calculating a confidence interval
d. Mislabeling groups
Answer: b. Detecting a difference when none existsdatalemur
What is the role of a holdout set in A/B testing for machine learning?
a. Used only for training
b. Used only for feature engineering
c. Kept aside for final model validation after experiments
d. For hyperparameter tuning
Answer: c. Kept aside for final model validation after experimentsstatsig
When is it appropriate to end an A/B test?
a. When p-value is just below 0.5
b. When the predefined sample size and duration have been reached
c. When the model accuracy increases
d. After just 10 samples
Answer: b. When the predefined sample size and duration have been reachedstatsig
Which of these increases the risk of “peeking” bias in A/B testing?
a. Ignoring features
b. Frequently checking results and stopping early
c. Running only one test
d. Using large datasets
Answer: b. Frequently checking results and stopping earlydatalemur
When should you use stratified sampling in A/B testing?
a. When all users are similar
b. To ensure groups remain balanced by important characteristics
c. When splitting data by random hash
d. Never
Answer: b. To ensure groups remain balanced by important characteristicsdatalemur
What is meant by “statistical significance threshold”?
a. Cut-off for data preprocessing
b. Alpha value below which results are considered statistically significant
c. The number of groups in the experiment
d. Dataset size
Answer: b. Alpha value below which results are considered statistically significantdatalemur
What is the risk of running too many A/B tests with overlapping users?
a. More accurate results
b. Dependence among tests, inflating error rates
c. Improved sample size
d. Reduced model performance
Answer: b. Dependence among tests, inflating error ratesdatalemur
Why is random assignment key in A/B testing?
a. To reduce bias in test assignment
b. To maximize profit
c. To cut computation costs
d. To find outliers
Answer: a. To reduce bias in test assignmentdatalemur
Which step comes first in the A/B testing workflow?
a. Collect data and analyze results
b. Define hypothesis and select key metrics
c. Train the final model
d. Deploy the model
Answer: b. Define hypothesis and select key metricsdatalemur
What is “regret” in the context of adaptive experimentation?
a. Not conducting enough tests
b. Cumulative loss from not always serving the better model
c. Forgetting to record metrics
d. Overfitting the data
Answer: b. Cumulative loss from not always serving the better modelstatsig
Why should an A/B test cover a sufficient time period?
a. To let servers cool down
b. To capture variation in user behavior over time
c. Because models need time to train
d. For simplicity
Answer: b. To capture variation in user behavior over timedatalemur
When deploying models in A/B tests, what traffic split is most common?
a. 90-10
b. 50-50
c. 30-70
d. 60-40
Answer: b. 50-50github+1
What makes Multi-Armed Bandit (MAB) different from A/B testing?
a. It never compares models
b. It dynamically reallocates user traffic based on performance
c. It ignores user feedback
d. It requires cross-validation
Answer: b. It dynamically reallocates user traffic based on performanceseldon+1
How do you interpret a p-value of 0.03 in A/B testing?
a. There is a 3% chance of observing results at least as extreme if null hypothesis is true
b. The results are not significant
c. The treatment is always better
d. The effect size is small
Answer: a. There is a 3% chance of observing results at least as extreme if null hypothesis is truedatalemur
Which is the primary drawback of Bayesian A/B testing?
a. Not adaptive
b. Requires informative priors and more computation
c. Cannot update beliefs
d. Does not provide credible intervals
Answer: b. Requires informative priors and more computationstatsig
In A/B testing, what is ‘lift’?
a. Increase in model complexity
b. The incremental improvement in a key metric between B and A
c. Lowering the confidence interval
d. Change in the dataset size
Answer: b. The incremental improvement in a key metric between B and Avwo+1
Which one is a sequential testing procedure in A/B tests?
a. t-test
b. K-means clustering
c. Sequential Probability Ratio Test (SPRT)
d. Clustering
Answer: c. Sequential Probability Ratio Test (SPRT)datalemur
Why are online controlled experiments called the “gold standard” in ML evaluation?
a. Because they always succeed
b. Because they provide unbiased causal estimation
c. Because they are faster
d. Because they use big data
Answer: b. Because they provide unbiased causal estimationseldon+1
What should be reported alongside results in A/B tests for transparency?
a. Confidence intervals and sample sizes
b. Server response times
c. Training parameters
d. Model visualizations
Answer: a. Confidence intervals and sample sizesstatsig+1
What happens if you end the A/B test before achieving statistical power?
a. Decreases bias
b. Increases risk of inconclusive or misleading results
c. Improves performance
d. Changes significance threshold
Answer: b. Increases risk of inconclusive or misleading resultsstatsig
What is commonly used to avoid assignment bias in web-based A/B tests?
a. User ID hash randomization
b. Model-based assignment
c. Alphabetic sorting
d. Gender-based groups
Answer: a. User ID hash randomizationdatalemur
Which term describes adjusting significance thresholds due to multiple sequential looks at the data?
a. Data drift
b. Alpha spending
c. Model pruning
d. Cross-validation
Answer: b. Alpha spendingdatalemur
Which process ensures no “carryover” effect in A/B tests?
a. Parallel design
b. Between-subjects design
c. Sequential group testing
d. Data shuffling
Answer: b. Between-subjects designdatalemur
What is a potential pitfall of not segmenting users properly in A/B tests?
a. Larger control groups
b. Skewed or biased results due to confounding variables
c. Improved lift
d. Simpler analysis
Answer: b. Skewed or biased results due to confounding variablesstatsig+1

Blog

Systemethic Technologies

Leave a Reply Cancel reply

Useful links

Other Terms & Policies

Contact Info