Into The Gray Zone: The Hazards Of AI Without Oversight
Contents
From facial recognition to autonomous vehicles, artificial intelligence (AI) and machine learning (ML) technologies are rapidly transforming industries in profound ways. But these powerful innovations come with unforeseen risks.
Behind the scenes, unregulated “shadow” AI/ML systems are being deployed without oversight, accountability, or transparency.
As these opaque models take on real-world decisions affecting human lives, a chorus of concerns has emerged around their potential dangers if developed recklessly.
Without illumination, these shadow systems embed unseen biases and inequities.
As AI proliferates, our society faces a choice: continue down our current path of breakneck deployment, or confront head-on the hazards within these technologies and bring accountability.
Detect, Block, And Log Risky AI Prompts
PromptShield™ is the first AI-powered firewall and defense platform that protects enterprises against the most critical AI prompt risks.
What Is Shadow AI/ML?
The term “shadow AI/ML” describes AI and ML systems that operate without sufficient transparency or accountability.
These models are deployed for sensitive tasks like:
- Facial recognition
- Predictive policing
- Credit decisions
- Content moderation
However, they frequently lack documentation, auditing, or governance over their internal logic.
The proprietary nature of many shadow AI systems prevents ethical scrutiny of their inner workings, even as they take on critical real-world functions.
This opacity around how shadow AI/ML models operate has raised concerns, especially as they become entrenched in high-impact domains.
For one, the lack of transparency and oversight for shadow AI systems raises significant risks around biased or flawed decision-making.
If the training data contains societal biases or discrimination, those can be perpetuated and amplified by the AI algorithms.
For example, facial recognition has exhibited racial and gender bias if models are trained on datasets that lack diversity.
Similarly, predictive policing systems trained on historically biased crime data can disproportionately target specific communities.
Even with unbiased data, algorithms can entrench societal prejudices if developer teams lack diversity and awareness of inclusiveness.
Furthermore, the autonomous nature of shadow AI taking actions without human involvement can lead to harmful outcomes.
If the model makes incorrect predictions or recommendations, there is no failsafe to prevent real-world harm.
For instance, AI screening job applicants could develop biased notions of ideal candidates and discount qualified people unjustly.
The Breach Report
PurpleSec’s security researchers provide expert analysis on the latest cyber attacks.
Examples Of Shadow AI
- Recruitment Bias: In 2018, Amazon scrapped an AI resume screening tool that exhibited bias against women. The algorithm penalized resumes containing words like “women’s chess club”, downgrading graduates of all-women’s colleges. This resulted in biased recommendations favoring male applicants.
- Biased Content Moderation: In 2020, Facebook’s AI mistook Black men’s posts discussing racial justice as violations of policies against hate speech and nudity. The automated moderation suppressed their voices during an important movement.
- Autonomous Vehicle Accidents: In 2018, an Uber self-driving car struck and killed a woman crossing the street in Arizona. The AI failed to identify the pedestrian at night. In 2016, a Tesla car in autopilot mode fatally crashed into a tractor-trailer that its sensors did not recognize. The accident killed the Tesla driver.
- Harmful YouTube Recommendations: In 2019, YouTube’s AI recommendation algorithm was found to steer viewers down extremist “rabbit holes”, recommending increasingly radical and divisive content. This amplified harmful misinformation.
- Racial Profiling In Healthcare: A 2020 study found a widely used healthcare algorithm exhibited racial bias, underestimating black patients’ needs compared to white patients. This could exacerbate health disparities.
- Toxic Chatbots: In 2016, Microsoft launched Tay, an AI chatbot that began spouting racist, sexist, and offensive views after being targeted by trolls online. This demonstrated risks of uncontrolled machine learning.
- Discriminatory Hiring Practices: HireVue, an AI recruiting tool, was found to favor certain intonations and speech patterns, potentially disadvantaging minorities during video interviews.
The rapid pace of AI development using complex techniques like deep learning exacerbates these issues with shadow systems.
The rush to deploy before thoroughly evaluating for fairness and safety means potential harms are outpacing governance.
And the black-box nature of deep learning algorithms makes it difficult to audit internal processes, even as they take on sensitive tasks.
Such instances underscore how today’s largely unregulated AI can lead to real ethical perils around bias, censorship, security, and safety.
Steps For Addressing The Risks
To address the risks of shadow AI systems, the AI/ML community needs to prioritize practices and principles that increase accountability, auditability, and transparency.
- Thorough documentation and procedures are essential – data provenance should be tracked to evaluate for bias, and every stage of model development needs to be recorded.
- Ongoing performance monitoring, especially across different demographic groups, can identify if the model exhibits unfair bias.
- Independent 3rd-party auditing of algorithms for discrimination and ethical failures is also critical.
- For high-risk AI applications like self-driving vehicles and social moderation, maintaining meaningful human oversight and decision validation is key to preventing harm. Humans must remain “in the loop” for reviewing and approving AI-generated outputs that impact human lives and society.
- In some cases, certain sensitive use cases may require restrictions on AI deployment until robust governance is established, rather than rapidly deploying shadow models with unchecked risks.
- Adopting standards like the EU’s Ethics Guidelines for Trustworthy AI will also guide the community toward fair, accountable AI development and integration.
- Organizations must ensure their AI teams represent diverse perspectives to identify potential harms. Democratically governing these rapidly evolving technologies is crucial to uphold ethics and human rights.
$35/MO PER DEVICE
Enterprise Security Built For Small Business
Defy your attackers with Defiance XDR™, a fully managed security solution delivered in one affordable subscription plan.
Conclusion
Realizing the promise of AI/ML responsibly will require deliberate efforts from all stakeholders.
Policy makers, researchers, developers, and civil society must collaborate to illuminate the processes within shadow systems through increased transparency and accountability measures.
Establishing ethical governance for AI will be crucial for earning public trust amidst rapid technological change.
The path forward demands sustained diligence – continually evaluating AI systems for bias, auditing algorithms, and maintaining human oversight for high-risk applications.
With sound ethical foundations guiding AI innovation and integration into society, these transformative technologies can be developed for the betterment of all.
But an ethical AI future relies on coming together to shed light on shadow systems today.
Share This Article
