Connect with us

Technology

Many AI model safety assessments have significant limitations

Published

on

Many safety evaluations for AI models have significant limitations

Despite the growing demand for AI security and accountability, today’s tests and benchmarks will not be enough, a brand new report finds.

Generative AI models—models that may analyze and generate text, images, music, video, and more—are coming under increasing scrutiny for his or her tendency to make mistakes and usually behave unpredictably. Now, organizations from public sector agencies to big tech firms are proposing recent benchmarks to check the safety of those models.

At the tip of last yr, the startup Scale AI created lab dedicated to assessing how well models adhere to security guidelines. This month, NIST and the U.K. AI Safety Institute released tools designed to evaluate model risk.

However, these tests and model testing methods could also be insufficient.

The Ada Lovelace Institute (ALI), a British non-profit organization dedicated to artificial intelligence research, conducted test who interviewed experts from academic, civil society, and vendor modeling labs and examined recent research on AI security assessments. The co-authors found that while current assessments will be useful, they should not comprehensive, will be easily fooled, and don’t necessarily provide guidance on how models will perform in real-world scenarios.

“Whether it’s a smartphone, a prescription drug, or a car, we expect the products we use to be safe and reliable; in these sectors, products are rigorously tested to ensure they’re safe before being deployed,” Elliot Jones, a senior researcher at ALI and co-author of the report, told TechCrunch. “Our research aimed to examine the limitations of current approaches to assessing AI safety, assess how assessments are currently being used, and explore their use as a tool for policymakers and regulators.”

Benchmarks and red teaming

The study’s co-authors first surveyed the tutorial literature to determine an summary of the harms and risks that current models pose and the state of existing assessments of AI models. They then interviewed 16 experts, including 4 employees of unnamed technology firms developing generative AI systems.

The study revealed that there’s wide disagreement across the AI ​​industry on the perfect set of methods and taxonomies for evaluating models.

Some evaluations only tested how well the models matched benchmarks within the lab, not how the models might impact real-world users. Others were based on tests designed for research purposes, not on evaluating production models—yet vendors insisted on using them in production.

We’ve written before concerning the problems with AI benchmarking. This study highlights all of those issues and more.

Experts cited within the study noted that it’s hard to extrapolate a model’s performance from benchmark results, and it’s unclear whether benchmarks may even show that a model has a certain capability. For example, while a model may perform well on a state exam, that doesn’t mean it can have the ability to resolve more open legal challenges.

Experts also pointed to the issue of knowledge contamination, where benchmark results can overstate a model’s performance if it was trained on the identical data it’s being tested on. Benchmarks, in lots of cases, are chosen by organizations not because they’re the perfect assessment tools, but due to their convenience and ease of use, experts said.

“Benchmarks run the risk of being manipulated by developers who may train models on the same dataset that will be used to evaluate the model, which is equivalent to looking at an exam paper before an exam or strategically choosing which assessments to use,” Mahi Hardalupas, a researcher at ALI and co-author of the study, told TechCrunch. “Which version of the model is being evaluated also matters. Small changes can cause unpredictable changes in behavior and can override built-in safety features.”

The ALI study also found problems with “red-teaming,” the practice of getting individuals or groups “attack” a model to discover gaps and flaws. Many firms use red-teaming to judge models, including AI startups OpenAI and Anthropic, but there are few agreed-upon standards for red-teaming, making it difficult to evaluate the effectiveness of a given effort.

Experts told the study’s co-authors that finding individuals with the correct skills and experience to steer red teaming efforts will be difficult, and the manual nature of the method makes it expensive and labor-intensive, a barrier for smaller organizations that don’t have the mandatory resources.

Possible solutions

The foremost the reason why AI rankings have not improved are the pressure to release models faster and the reluctance to run tests that might cause issues before launch.

“The person we spoke to who works for a foundation modeling company felt that there is more pressure within companies to release models quickly, which makes it harder to push back and take assessments seriously,” Jones said. “The major AI labs are releasing models at a speed that outpaces their ability or society’s ability to ensure they are safe and reliable.”

One ALI survey respondent called evaluating models for safety an “intractable” problem. So what hopes does the industry—and those that regulate it—have for solutions?

Mahi Hardalupas, a researcher at ALI, believes there’s a way forward, but it can require greater commitment from public sector entities.

“Regulators and policymakers need to be clear about what they expect from ratings,” he said. “At the same time, the ratings community needs to be transparent about the current limitations and potential of ratings.”

Hardalupas suggests that governments mandate greater public participation in the event of assessments and implement measures to support an “ecosystem” of third-party testing, including programs to offer regular access to any required models and datasets.

Jones believes it could be mandatory to develop “context-aware” assessments that transcend simply testing a model’s response to a command, and as an alternative consider the sorts of users a model might affect (akin to people of a certain background, gender, or ethnicity), in addition to the ways wherein attacks on models could bypass security measures.

“This will require investment in fundamental evaluation science to develop more robust and repeatable evaluations based on an understanding of how the AI ​​model works,” she added.

However, there’s never a guarantee that a model is protected.

“As others have noted, ‘safety’ is not a property of models,” Hardalupas said. “Determining whether a model is ‘safe’ requires understanding the contexts in which it is used, to whom it is sold or shared, and whether the safeguards that are implemented are appropriate and robust to mitigate those risks. Baseline model assessments can serve exploratory purposes to identify potential risks, but they cannot guarantee that the model is safe, much less ‘completely safe.’ Many of our interviewees agreed that assessments cannot prove that a model is safe and can only indicate that the model is unsafe.”

This article was originally published on : techcrunch.com
Continue Reading
Advertisement
Click to comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Technology

The company is currently developing washing machines for humans

Published

on

By

Forget about cold baths. Washing machines for people may soon be a brand new solution.

According to at least one Japanese the oldest newspapersOsaka-based shower head maker Science has developed a cockpit-shaped device that fills with water when a bather sits on a seat in the center and measures an individual’s heart rate and other biological data using sensors to make sure the temperature is good. “It also projects images onto the inside of the transparent cover to make the person feel refreshed,” the power says.

The device, dubbed “Mirai Ningen Sentakuki” (the human washing machine of the longer term), may never go on sale. Indeed, for now the company’s plans are limited to the Osaka trade fair in April, where as much as eight people will have the option to experience a 15-minute “wash and dry” every day after first booking.

Apparently a version for home use is within the works.

This article was originally published on : techcrunch.com
Continue Reading

Technology

Zepto raises another $350 million amid retail upheaval in India

Published

on

By

Zepto, snagging $1 billion in 90 days, projects 150% annual growth

Zepto has secured $350 million in latest financing, its third round of financing in six months, because the Indian high-speed trading startup strengthens its position against competitors ahead of a planned public offering next yr.

Indian family offices, high-net-worth individuals and asset manager Motilal Oswal invested in the round, maintaining Zepto’s $5 billion valuation. Motilal co-founder Raamdeo Agrawal, family offices Mankind Pharma, RP-Sanjiv Goenka, Cello, Haldiram’s, Sekhsaria and Kalyan, in addition to stars Amitabh Bachchan and Sachin Tendulkar are amongst those backing the brand new enterprise, which is India’s largest fully national primary round.

The funding push comes as Zepto rushes so as to add Indian investors to its capitalization table, with foreign ownership now exceeding two-thirds. TechCrunch first reported on the brand new round’s deliberations last month. The Mumbai-based startup has raised over $1.35 billion since June.

Fast commerce sales – delivering groceries and other items to customers’ doors in 10 minutes – will exceed $6 billion this yr in India. Morgan Stanley predicts that this market shall be value $42 billion by 2030, accounting for 18.4% of total e-commerce and a pair of.5% of retail sales. These strong growth prospects have forced established players including Flipkart, Myntra and Nykaa to cut back delivery times as they lose touch with specialized delivery apps.

While high-speed commerce has not taken off in many of the world, the model seems to work particularly well in India, where unorganized retail stores are ever-present.

High-speed trading platforms are creating “parallel trading for consumers seeking convenience” in India, Morgan Stanley wrote in a note this month.

Zepto and its rivals – Zomato-owned Blinkit, Swiggy-owned Instamart and Tata-owned BigBasket – currently operate on lower margins than traditional retail, and Morgan Stanley expects market leaders to realize contribution margins of 7-8% and adjusted EBITDA margins to greater than 5% by 2030. (Zepto currently spends about 35 million dollars monthly).

An investor presentation reviewed by TechCrunch shows that Zepto, which handles greater than 7 million total orders every day in greater than 17 cities, is heading in the right direction to realize annual sales of $2 billion. It anticipates 150% growth over the following 12 months, CEO Aadit Palicha told investors in August. The startup plans to go public in India next yr.

However, the rapid growth of high-speed trading has had a devastating impact on the mom-and-pop stores that dot hundreds of Indian cities, towns and villages.

According to the All India Federation of Consumer Products Distributors, about 200,000 local stores closed last yr, with 90,000 in major cities where high-speed trading is more prevalent.

The federation has warned that without regulatory intervention, more local shops shall be vulnerable to closure as fast trading platforms prioritize growth over sustainable practices.

Zepto said it has created job opportunities for tons of of hundreds of gig employees. “From day one, our vision has been to play a small role in nation building, create millions of jobs and offer better services to Indian consumers,” Palicha said in an announcement.

Regulatory challenges arise. Unless an e-commerce company is a majority shareholder of an Indian company or person, current regulations prevent it from operating on a listing model. Fast trading corporations don’t currently follow these rules.

This article was originally published on : techcrunch.com
Continue Reading

Technology

Wiz acquires Dazz for $450 million to expand cybersecurity platform

Published

on

By

Wizardone of the talked about names within the cybersecurity world, is making a major acquisition to expand its reach of cloud security products, especially amongst developers. This is buying Dazzlespecialist in solving security problems and risk management. Sources say the deal is valued at $450 million, which incorporates money and stock.

This is a leap within the startup’s latest round of funding. In July, we reported that Dazz had raised $50 million at a post-money valuation of just below $400 million.

Remediation and posture management – two areas of focus for Dazz – are key services within the cybersecurity market that Wiz hasn’t sorted in addition to it wanted.

“Dazz is a leader in this market, with the best talent and the best customers, which fits perfectly into the company culture,” Assaf Rappaport, CEO of Wiz, said in an interview.

Remediation, which refers to helping you understand and resolve vulnerabilities, shapes how an enterprise actually handles the various vulnerability alerts it could receive from the network. Posture management is a more preventive product: it allows a company to higher understand the scale, shape and performance of its network from a perspective, allowing it to construct higher security services around it.

Dazz will proceed to operate as a separate entity while it’s integrated into the larger Wiz stack. Wiz has made a reputation for itself as a “one-stop shop,” and Rappaport said the integrated offering will proceed to be a core a part of it.

He believes this contrasts with what number of other SaaS corporations are built. In the safety industry, there are, Rappaport said, “a lot of Frankenstein mashups where companies prioritize revenue over building a single technology stack that actually works as a platform.” It could be assumed that integration is much more necessary in cybersecurity than in other areas of enterprise IT.

Wiz and Dazz already had an in depth relationship before this deal. Merat Bahat — the CEO who co-founded Dazz with Tomer Schwartz and Yuval Ofir (CTO and VP of R&D, respectively) — worked closely with Assaf Rappaport at Microsoft, which acquired his previous startup Adallom.

After Rappaport left to found Wiz together with his former Adallom co-founders, CTO Ami Luttwak, VP of Product Yinon Costica and VP of R&D Roy Reznik, Bahat was one in all the primary investors. Similarly, when Bahat founded Dazz, Assaf was a small investor in it.

The connection goes deeper than work colleagues. Bahat and Rappaport are also close friends, and she or he was the second family of Mickey, Rappaport’s beloved dog, referred to as Chief Dog Officer Wiz (together with LinkedIn profile). Once the deal was done, the 2 faced two very sad events: each Bahat and Mika’s mother died.

“We hope for a new chapter of positivity,” Bahat said. The cycle of life does indeed proceed.

Rumors of this takeover began to appear earlier this month; Rappaport confirmed that they then began talking seriously.

But that is not the one M&A conversation Wiz has gotten involved in. Earlier this 12 months, Google tried to buy Wiz itself for $23 billion to construct a major cybersecurity business. Wiz walked away from the deal, which might have been the biggest in Google’s history, partly because Rappaport believed Wiz could turn into a fair larger company by itself terms. And that is what this agreement goals to do.

This acquisition is a test for Wiz, which earlier this 12 months filled its coffers with $1 billion solely for M&A purposes (it has raised almost $2 billion in total, and we hear the subsequent round will close in just a few weeks). . Other offers included purchasing Gem security for $350 million, but Dazz is its largest acquisition ever.

More mergers and acquisitions could also be coming. “We believe next year will be an acquisition year for us,” Rappaport said.

In an interview with TC, Luttwak said that one in all Wiz’s priorities now’s to create more tools for developers that have in mind what they need to do their jobs.

Enterprises have made significant investments in cloud services to speed up operations and make their IT more agile, but this shift has include a significantly modified security profile for these organizations: network and data architectures are more complex and attack surfaces are larger, creating opportunities for malicious hackers to find ways to to hack into these systems. Artificial intelligence makes all of this far more difficult when it comes to malicious attackers. (It’s also a chance: the brand new generation of tools for our defense relies on artificial intelligence.)

Wiz’s unique selling point is its all-in-one approach. Drawing data from AWS, Azure, Google Cloud and other cloud environments, Wiz scans applications, data and network processes for security risk aspects and provides its users with a series of detailed views to understand where these threats occur, offering over a dozen products covering the areas, corresponding to code security, container environment security, and provide chain security, in addition to quite a few partner integrations for those working with other vendors (or to enable features that Wiz doesn’t offer directly).

Indeed, Wiz offered some extent of repair to help prioritize and fix problems, but as Luttwak said, the Dazz product is solely higher.

“We now have a platform that actually provides a 360-degree view of risk across infrastructure and applications,” he said. “Dazz is a leader in attack surface management, the ability to collect vulnerability signals from the application layer across the entire stack and build the most incredible context that allows you to trace the situation back to engineers to help with remediation.”

For Dazz’s part, once I interviewed Bahat in July 2024, when Dazz raised $50 million at a $350 million valuation, she extolled the virtues of constructing strong solutions and this week said the third quarter was “amazing.”

“But market dynamics are what trigger these types of transactions,” she said. She confirmed that Dazz had also received takeover offers from other corporations. “If you think about the customers and joint customers that we have with Wiz, it makes sense for them to have it on one platform.”

And a few of Dazz’s competitors are still going it alone: ​​Cyera, like Dazz, an authority in attack surface management, just yesterday announced a rise of $300 million at a valuation of $5 billion (which confirms our information). But what’s going to he do with this money? Make acquisitions, after all.

Wiz says it currently has annual recurring revenue of $500 million (it has a goal of $1 billion ARR next 12 months) and has greater than 45% of its Fortune 100 customers. Dazz said ARR is within the tens of hundreds of thousands of dollars and currently growing 500% on a customer base of roughly 100 organizations.

This article was originally published on : techcrunch.com
Continue Reading
Advertisement

OUR NEWSLETTER

Subscribe Us To Receive Our Latest News Directly In Your Inbox!

We don’t spam! Read our privacy policy for more info.

Trending