Artificial intelligence (AI), personal data and consent

Artificial intelligence (AI) seems to be everywhere, and has been getting almost as much investment funding as media attention. Is it the latest tech buzzword or is it changing—and will continue to change—nearly everything about how we create and work? Who owns the input data and the results?

AI development has been said to rest on the pillars of algorithms, hardware, and data. Data is the pillar that is the least “solved”, and user consent is an important part of the question.

The rapid advancement of AI training and uses of the technology has raised concerns about user consent and ethical implications with uses of personal data. If users’ data is used to train AI, do they have rights to the outputs? Should organizations needing AI training data have to obtain consent for data already published online? For how many granular purposes should AI tools or services providers have to obtain explicit consent from users?

AI refers to the development of machines that can perform tasks that typically require human intelligence. This includes areas such as text or speech recognition, problem solving, and decision-making. Developing AI often requires input of large amounts of data to help the systems “learn”.

Machine learning is a subset of AI that focuses on developing algorithms and models that enable computers to learn from data and make predictions or decisions without being explicitly programmed. It’s a way for computers to “learn” from examples and improve their performance over time.

Large Language Models are a recent breakthrough in AI research, designed to understand and generate human-like language. ChatGPT from OpenAI and Bard from Google are examples of publicly accessible LLMs. Some tools developed using them can be used for SEO, marketing content, and other business purposes.

The purpose of training an LLM is to enable it to understand the structure, meaning, and context of human language, which, for one use, enables more accurate responses when queried by people.

LLMs are trained on vast amounts of text from books, articles, websites, and other sources. To date there have been data privacy issues with content being scraped and analyzed without the creators’ or owners’ consent. There is the possibility that data accessed could be sensitive, in addition to having been used without consent.

AI training, also known as machine learning training, refers to the process of teaching an AI system to learn patterns and make predictions or decisions based on data provided to it. Training is crucial to developing AI systems that can perform specific tasks, recognize patterns, provide accurate information, or make informed judgments.

The training process goes through a number of steps. In short, they begin with procuring relevant data and making it ready for use, to selecting what the model will be meant to do with AI training data sets, to data input and analysis. Then there’s working to make output or predictions match actual outcomes or improve in accuracy, and ensuring the AI model works well on any data set, including real world data, and not just on AI training data. AI models have to pass all of the steps before they’re ready for broader use.

Organizations could raise questions about what defines “use” of personal data. How much change renders it no longer personal data? For example, to get the data into a format the training model could use, it may have to be transformed from the format in which when collected. Additionally, should an organization need to get consent to use data to train AI models even if it’s for research only and not commercial purposes? Maybe no one but the researchers will ever have access to it.

AI can be trained on many kinds of data. What the trainers need may depend on what the system is meant to be able to do, e.g. answer questions, make decisions, generate graphics or text, etc.

Some common types of training data for AI include:

• text – e.g. from books, articles, websites, or social media; used for translation, sentiment analysis, chatbot development, etc.
• images – from large numbers of labeled images; used for image recognition, object detection, and image generation
• audio – e.g. from spoken words, sounds, or acoustic patterns; used for speech recognition, voice assistants, and audio analysis models
• video data – from video sequences; used in video analysis, surveillance, video generation, and to learn temporal patterns
• gaming data – from gameplay data and interactions; used to develop game play and strategy
• structured data – e.g. from databases or spreadsheets; used for predictive analytics, recommendation systems, or fraud detection
• sensor data – from cameras, lidar, radar, etc.; used for autonomous vehicle systems, industrial automation, etc.
• healthcare data – from medical imaging like x-rays and MRIs, patient records, and clinic data; used for assistance in diagnoses, treatment, and research
• financial data – from existing financial data from markets and transaction records; used for stock price prediction, credit scoring, and fraud detection
• genomic data – from DNA sequences, genetic markers, and other related biological data; used for personalized medicine and improving understanding of genetics
• simulation data – from data generated by simulations; used for learning how systems behave under different conditions

Many of these kinds of AI training data are explicitly referenced in data privacy laws. Many are types of personal data, and some are PII data, also called personally identifiable information. Some of these types of data are also categorized under privacy laws as sensitive, meaning they could do greater harm if accessed or used without authorization.

Healthcare, genomic, and financial information are particularly significant examples of sensitive personal data. Sensitive data usually requires user consent to collect or use under data privacy law, while data that is personal, but not sensitive, sometimes only requires consent before being sold or used for targeted advertising, profiling, etc.

It’s also important to note that not all batches of training data are equal. Quality, quantity, diversity, and permission for use can vary widely. That can have a significant impact on the “learning” and performance of the systems. It could also mean consent is required to use some types of data in the training batch, but not for others. Poorly balanced or non-diverse data can also produce skewed results, sometimes with offensive or legally precarious output, like systems that produce discriminatory recommendations or inaccurate identification.

Under many privacy laws, data subjects have the right to have their data corrected by the entity that has collected it, if it’s incomplete or inaccurate. What about if their data is correct, but used to produce inaccurate results? What are their rights then? Uses of these technologies pose many complex questions for regulators that include the ethics of automation.

Research firm Gartner has predicted that by the end of 2023, 65% of the world’s population will have its personal data protected by data privacy regulations. By 2024 they predict the number will be 75%. The only things changing faster than privacy regulation coverage are technology itself and demands for data. It helps drive everything from scientific breakthroughs to marketing campaigns.

But data isn’t like air. It doesn’t just freely exist for anyone to use. A lot of the data that exists, and that organizations want access to, is generated by people, and thus they have rights regarding protection of it and access to it. Consumers are increasingly savvy these days about data privacy and their rights where their personal data is concerned. Even if they may not understand how AI systems and other functions work in detail.

With the passing of more privacy legislation around the world, organizations need to be increasingly careful about meeting their data privacy responsibilities. Potentially huge fines, like some of those levied under the European Union’s General Data Protection Regulation (GDPR), also highlight the importance of taking privacy regulations and consumers’ rights seriously.

There are ever more potential sources of user data, especially online, like from social platforms and apps. It can also be tricky for companies to determine their data privacy responsibilities when the company is headquartered in one place, but potentially has users around the world. This can make an organization responsible to comply with multiple different privacy regulations. Many such laws are extraterritorial, in which case it only matters where users are located with regards to rights and protections, not the companies.

A lot of consumers don’t focus too much on just how much data they create on a daily basis, who might have access to it, and how it could be used. Children may not pay attention or fully understand user data generation or processing at all, even though most data privacy laws require extra protections and consent for access to their data. That consent must typically be obtained from a parent or legal guardian if the child is under a certain age threshold determined by the specific law.

A number of data privacy laws do not cover personal data that people make publicly available, which could include that generated on social platforms. Perhaps posts, comments, and photos are not a big privacy concern to some. But what about private messages or chats? Those could contain far more sensitive material.

Once data has been collected, ideally with user consent, people should know what happens to it. It’s a condition of most privacy laws that the controller—the entity responsible for collecting and using the data—notify users about what data will be collected and for what purposes. If those purposes change, under many privacy laws the controller must notify users and get new consent. With AI training, this could require a lot of granular detail, and could change often.

Because AI systems are often still experimental and the results unpredictable, it can make some data privacy requirements tricky. Organizations can notify users about what they want to use data for, but it’s possible what the data actually gets used for, or how it may be changed, or the results from using it may be different.

While users are supposed to be notified before any new purpose is put in place, those doing the work may not know of the change until it’s happened. If data is being analyzed in vast quantities in real time, traditional mechanisms for obtaining user consent, like cookie banners, may not be fast or granular enough, or otherwise sufficient.

User-facing AI systems can be potentially manipulative, resulting in users providing information they didn’t anticipate. Systems may also surface more sophisticated and nebulous connections between data points, enabling identification and profiling at a level we have not seen before. This could potentially turn just about any data into personally identifiable or sensitive data. Current consent requirements may not adequately address this.

While manipulative user interface and user experience functions commonly known as dark patterns are increasingly frowned upon and, in some cases, have been regulated against, those tend to focus on tactics that are already familiar. Responsive design could enable the development of new and more sophisticated ways of manipulating users.

Popular video conferencing platform Zoom updated its terms of service (TOS) in March 2023, which is a common enough thing for a company to do. However, two sections appeared to have broad implications for Zoom’s permissions regarding user data, referred to as “Service Generated Data”, which includes telemetry, product usage, diagnostics, and similar data or content generated by using Zoom and that the company collects in the course of using the platform.

The updated TOS provided all rights to Service Generated Data to Zoom, including rights to modify, distribute, process, share, maintain, and store the data “for any purpose, to the extent and in the manner permitted under applicable law.” Zoom’s right to use user data for machine learning and artificial intelligence, including training and tuning models and algorithms, was explicitly mentioned.

So Zoom could collect a variety of user data from the use of its platform and apply it to many uses, including AI training, without the need to get explicit consent from users or to enable them to opt out.

This may be legal under current privacy laws in the United States, where Zoom is headquartered—the country does not have a single federal law, but only a number of state-level laws—but it is not legal under the EU’s GDPR, which requires consent to be “informed”, among other requirements (Recital 32 GDPR).

Under the GDPR, for consent to be valid it must also be obtained before data collection begins, and requires notification to users that is clear and understandable. Zoom’s TOS are somewhat cryptic, as are those of plenty of other companies.

The backlash to the discovery and public coverage of this TOS change was substantial. Companies were concerned that proprietary information from confidential meetings could be used without consent. Or that Zoom would own their creative content, like interviews for videos or podcasts.

Some US companies using Zoom for healthcare-related purposes panicked because of concerns about privacy violations of the Health Insurance Portability and Accountability Act (HIPAA). There were fears of the company owning and being able to use the content of people’s therapy sessions, for example. These uses weren’t necessarily ever the company’s intent for the data use, but public perception is powerful.

Zoom responded to the backlash with another TOS update to clarify data use, stating that it would not train its AI models with customers’ audio, video, or chat content without first obtaining consent.

It also added the line in section 10.2 that states: “Zoom does not use any of your audio, video, chat, screen sharing, attachments or other communications-like Customer Content (such as poll results, whiteboard and reactions) to train Zoom or third-party artificial intelligence models.”

However, some users still expressed concern at the apparently wide-ranging permissions granted to Zoom if consent was provided, and many still are not clear about exactly what Service Generated Content includes.

It’s important to note that Zoom is not an outlier. Other companies use AI for functions on their platforms. Google uses it to create transcripts of Google Meet calls (with results of varying quality). Facebook parent company Meta was also found to “hide” consent for use of user data in personalized advertising in its TOS in 2022. In January 2023, the company was prohibited from using personal data for advertising with this type of “consent”, which most users were completely unaware of. Meta has since said they would change their model and request consent for advertising in the EU.

Other companies have been caught attempting similarly non-transparent tactics. Some have been caught burying “consent” or questionable permissions into their terms of service, since they know that few users read them in detail. This is poor practice at best, and illegal at worst, since many regulations require consent to be informed.

The need for greater clarity around AI training, user-generated content on platforms, and consent is clearly here, and will only become a more pressing issue over time.

Companies that acquire data for AI training or other uses can and should ensure that consent was obtained from the sources or users. In some cases it may be a requirement for doing business with partners or vendors.

Consent is also becoming important to monetization strategy. For example, increasingly, premium advertisers are insisting on proof of consent for collection of user data before partnering with app developers.

Companies that collect user data from their own platforms and users for AI training or other uses have direct responsibility for obtaining valid consent and complying with data protection laws. There are a number of ways companies can achieve compliance and valid consent.

Transparency – Privacy laws require clear, accessible notifications, and companies should provide understandable information to users about how user data will be used and processed, including for AI training. As the uses for personal data change, companies need to update their privacy notices, inform users, and, under many privacy laws, get new consent for the new uses of personal data.

Granular consent – Users must be able to accept or decline the collection and processing of their personal data, but they should be able to do it at a detailed level, e.g. approving some kinds of processing, like targeted advertising or AI training, but not others, like sale of the data. This also helps ensure people are informed, which is a requirement for consent to be valid under most privacy laws.

User-friendly mechanisms – Just as notifications must be clear and accessible, the way users accept or decline consent must be easy to understand and access. Information to inform users about data processing must be available there as well as the ability to consent or decline at a granular level. It must also be as easy to decline consent as it is to accept, and under many privacy laws users must also be able to easily change their consent preferences.

Regulatory familiarity – Different jurisdictions have different privacy laws with different requirements and consent models. It’s important for companies to know which laws they need to comply with, and how to do so. It can be important to consult with or appoint qualified legal counsel or a privacy expert, e.g. a data protection officer (DPO), which is also required by some privacy laws. Such a role helps to establish guidelines and processes, update operations, and manage security for data and processing.

Consumers’ rights regarding their personal data depend on a number of factors, including where the user lives and what privacy laws are in place, what the platform is for and what data the user is providing to or generating on it, and what the platform’s terms of service are.

In the European Union, companies collecting and processing personal data must obtain user consent before doing so. This applies equally to social media platforms, a blog, a government website, or an ecommerce store. Users’ data may be collected to learn how people use a site and improve how it works. Or to enable fulfillment when they buy something online, or to show them ads, or to train AI models.

Platforms around the world that are used for financial activities or healthcare have stronger requirements for privacy and security under multiple regulations because of the kinds of information they handle.

In some jurisdictions, it is still allowed to display a cookie banner that says you consent to collection and use of your personal data by continuing to use the site or service. But in the EU and other jurisdictions, this is not acceptable and granular consent is required.

Use of cookies online has been declining as there are newer and better technologies to accomplish what cookies are used for. The question today and going forward is less how AI uses cookies, or may do so, and more how AI could accelerate the replacement of cookies.

Apple and Mozilla have blocked third-party cookies, and Google plans to deprecate them entirely. New tools and methods also enable better data privacy and consent, and can result in higher quality user data.

Current cookie consent models may not be sufficient to cover AI use, since AI systems may analyze large amounts of data in real-time, rather than tools analyzing data from active cookies over time. For consent to be obtained before data collection or use begins, with current pop-ups the user would have to be bombarded with consent banners faster and more often than a human could process them.

AI models can enable more effective ads or personalized user experiences without relying on collection of personally identifiable information, as they can analyze large amounts of data very quickly to group people into audiences based on behaviors. If the system doesn’t need to collect user data, then consent may not be needed, at least for the data collection.

Laws and best practices would likely still require users to be notified of how their behaviors could be tracked and analyzed, and what that analysis could be used for, e.g. personalized ads or shopping experiences. But people’s personal data couldn’t be sold if it was never collected.

The EU AI Act is a law on artificial intelligence (AI) proposed by the European Commission. It is the world’s first comprehensive law to regulate AI. The aim is to balance positive uses of the technology while mitigating negative ones and codifying rights. There is also a goal to clarify many current and future questions about AI development and make the Act a global standard, as the GDPR has become.

The law would assign applications of AI technology to one of several categories:

Unacceptable risk – AI with unacceptable risks would be banned entirely, e.g. the Chinese government’s social scoring tool

High risk – AI with potential risks, permitted subject to compliance with AI requirements and forecasted conformity assessment, e.g. a tool that ranks job applicants by scanning resumes

Medium risk – AI with specific transparency obligations, permitted but subject to information requirements, e.g. bots that can be used for impersonation

Minimal or no risk – AI with no notable risks, permitted without restrictions

The AI Act is currently in a draft state, and may change prior to becoming law. At present user consent and data privacy and protection are addressed in its statutes on a number of fronts:

High Risk – explicit consent is required for use of high-risk AI systems, e.g. critical infrastructure, employment, healthcare, and law enforcement.

Transparency – AI providers must provide clear information about systems’ intended purpose, capabilities, and limitations to ensure users are informed to make decisions and understand potential impacts on their rights.

Right to Explanation – users have the right to obtain meaningful explanations of AI systems’ decisions.

Right to User Control – users should have the ability to opt out, disable, or uninstall AI systems, particularly when fundamental rights or interests are at stake (under some privacy laws users have the right to opt out of “automated decision-making”).

Data Protection and Privacy – the AI Act emphasizes the need for data minimization, purpose limitation, and safeguards to protect personal data when using AI systems, and aligns with existing data privacy regulations like the GDPR.

AI technology is here to stay. Its capabilities and potential use cases will continue to evolve rapidly. This is a challenge for regulation, as creating and updating laws tends to happen much more slowly than the speed at which technologies develop.

However, users should not be faced with “caveat emptor”, especially online, with regards to new uses for their personal data and challenges to their privacy. Regulators must craft and update laws that are clear and comprehensive, but flexible enough to be interpretable and enforceable today and in the future.

Organizations need to be clear on what privacy regulations they must comply with, what they stipulate, and what that means for their operations. This needs to be regularly reviewed as operations change, and clearly communicated. Trying to sneak in changes to terms of use or use collected data for new purposes without getting new user consent is an easy way to damage brand reputation. It is also illegal in many jurisdictions. As consumers get even more savvy about their data and privacy, companies will have to be more clear, not less, about data collection and use.

Companies should also implement best practices like privacy by design to ensure they are respecting people—the source of their data—and complying with the law. This will also help ensure that consent is obtained and data collection and use limited to legal allowances for all operations, whether fulfilling ecommerce orders or training new AI models.

AI is just the latest technology to bring new challenges to consumers, companies, and regulators alike. It’s not the first and won’t be the last. But best practices for achieving compliance, building trust with users, and successfully growing businesses (or doing science) continue to be the same and serve both organizations and consumers well.

To learn more, talk to our experts today.

Usercentrics does not provide legal advice, and information is provided for educational purposes only. We always recommend engaging qualified legal counsel or privacy specialists regarding data privacy and protection issues and operations.