Artificial intelligence (AI), personal data and consent

Artificial intelligence (AI) seems to be everywhere, and has been getting almost as much investment funding as media attention. Is it the latest tech buzzword or is it changing—and will continue to change—nearly everything about how we create and work? Who owns the input data and the results?

AI development has been said to rest on the pillars of algorithms, hardware, and data. Data is the pillar that is the least “solved”, and user consent is an important part of the question.

The rapid advancement of AI training and uses of the technology has raised concerns about user consent and ethical implications with uses of personal data. If users’ data is used to train AI, do they have rights to the outputs? Should organizations needing AI training data have to obtain consent for data already published online? For how many granular purposes should AI tools or services providers have to obtain explicit consent from users?

AI refers to the development of machines that can perform tasks that typically require human intelligence. This includes areas such as text or speech recognition, problem solving, and decision-making. Developing AI often requires input of large amounts of data to help the systems “learn”.

Machine learning is a subset of AI that focuses on developing algorithms and models that enable computers to learn from data and make predictions or decisions without being explicitly programmed. It’s a way for computers to “learn” from examples and improve their performance over time.

Large Language Models are a recent breakthrough in AI research, designed to understand and generate human-like language. ChatGPT from OpenAI and Bard from Google are examples of publicly accessible LLMs. Some tools developed using them can be used for SEO, marketing content, and other business purposes.

The purpose of training an LLM is to enable it to understand the structure, meaning, and context of human language, which, for one use, enables more accurate responses when queried by people.

LLMs are trained on vast amounts of text from books, articles, websites, and other sources. To date there have been data privacy issues with content being scraped and analyzed without the creators’ or owners’ consent. There is the possibility that data accessed could be sensitive, in addition to having been used without consent.

AI training, also known as machine learning training, refers to the process of teaching an AI system to learn patterns and make predictions or decisions based on data provided to it. Training is crucial to developing AI systems that can perform specific tasks, recognize patterns, provide accurate information, or make informed judgments.

The training process goes through a number of steps. In short, they begin with procuring relevant data and making it ready for use, to selecting what the model will be meant to do with AI training data sets, to data input and analysis. Then there’s working to make output or predictions match actual outcomes or improve in accuracy, and ensuring the AI model works well on any data set, including real world data, and not just on AI training data. AI models have to pass all of the steps before they’re ready for broader use.

Organizations could raise questions about what defines “use” of personal data. How much change renders it no longer personal data? For example, to get the data into a format the training model could use, it may have to be transformed from the format in which when collected. Additionally, should an organization need to get consent to use data to train AI models even if it’s for research only and not commercial purposes? Maybe no one but the researchers will ever have access to it.

AI can be trained on many kinds of data. What the trainers need may depend on what the system is meant to be able to do, e.g. answer questions, make decisions, generate graphics or text, etc.

Some common types of training data for AI include:

• text – e.g. from books, articles, websites, or social media; used for translation, sentiment analysis, chatbot development, etc.
• images – from large numbers of labeled images; used for image recognition, object detection, and image generation
• audio – e.g. from spoken words, sounds, or acoustic patterns; used for speech recognition, voice assistants, and audio analysis models
• video data – from video sequences; used in video analysis, surveillance, video generation, and to learn temporal patterns
• gaming data – from gameplay data and interactions; used to develop game play and strategy
• structured data – e.g. from databases or spreadsheets; used for predictive analytics, recommendation systems, or fraud detection
• sensor data – from cameras, lidar, radar, etc.; used for autonomous vehicle systems, industrial automation, etc.
• healthcare data – from medical imaging like x-rays and MRIs, patient records, and clinic data; used for assistance in diagnoses, treatment, and research
• financial data – from existing financial data from markets and transaction records; used for stock price prediction, credit scoring, and fraud detection
• genomic data – from DNA sequences, genetic markers, and other related biological data; used for personalized medicine and improving understanding of genetics
• simulation data – from data generated by simulations; used for learning how systems behave under different conditions

Many of these kinds of AI training data are explicitly referenced in data privacy laws. Many are types of personal data, and some are PII data, also called personally identifiable information. Some of these types of data are also categorized under privacy laws as sensitive, meaning they could do greater harm if accessed or used without authorization.

Healthcare, genomic, and financial information are particularly significant examples of sensitive personal data. Sensitive data usually requires user consent to collect or use under data privacy law, while data that is personal, but not sensitive, sometimes only requires consent before being sold or used for targeted advertising, profiling, etc.

It’s also important to note that not all batches of training data are equal. Quality, quantity, diversity, and permission for use can vary widely. That can have a significant impact on the “learning” and performance of the systems. It could also mean consent is required to use some types of data in the training batch, but not for others. Poorly balanced or non-diverse data can also produce skewed results, sometimes with offensive or legally precarious output, like systems that produce discriminatory recommendations or inaccurate identification.

Under many privacy laws, data subjects have the right to have their data corrected by the entity that has collected it, if it’s incomplete or inaccurate. What about if their data is correct, but used to produce inaccurate results? What are their rights then? Uses of these technologies pose many complex questions for regulators that include the ethics of automation.

Research firm Gartner has predicted that by the end of 2023, 65% of the world’s population will have its personal data protected by data privacy regulations. By 2024 they predict the number will be 75%. The only things changing faster than privacy regulation coverage are technology itself and demands for data. It helps drive everything from scientific breakthroughs to marketing campaigns.

But data isn’t like air. It doesn’t just freely exist for anyone to use. A lot of the data that exists, and that organizations want access to, is generated by people, and thus they have rights regarding protection of it and access to it. Consumers are increasingly savvy these days about data privacy and their rights where their personal data is concerned. Even if they may not understand how AI systems and other functions work in detail.

With the passing of more privacy legislation around the world, organizations need to be increasingly careful about meeting their data privacy responsibilities. Potentially huge fines, like some of those levied under the European Union’s General Data Protection Regulation (GDPR), also highlight the importance of taking privacy regulations and consumers’ rights seriously.

There are ever more potential sources of user data, especially online, like from social platforms and apps. It can also be tricky for companies to determine their data privacy responsibilities when the company is headquartered in one place, but potentially has users around the world. This can make an organization responsible to comply with multiple different privacy regulations. Many such laws are extraterritorial, in which case it only matters where users are located with regards to rights and protections, not the companies.

A lot of consumers don’t focus too much on just how much data they create on a daily basis, who might have access to it, and how it could be used. Children may not pay attention or fully understand user data generation or processing at all, even though most data privacy laws require extra protections and consent for access to their data. That consent must typically be obtained from a parent or legal guardian if the child is under a certain age threshold determined by the specific law.

A number of data privacy laws do not cover personal data that people make publicly available, which could include that generated on social platforms. Perhaps posts, comments, and photos are not a big privacy concern to some. But what about private messages or chats? Those could contain far more sensitive material.

Once data has been collected, ideally with user consent, people should know what happens to it. It’s a condition of most privacy laws that the controller—the entity responsible for collecting and using the data—notify users about what data will be collected and for what purposes. If those purposes change, under many privacy laws the controller must notify users and get new consent. With AI training, this could require a lot of granular detail, and could change often.

Because AI systems are often still experimental and the results unpredictable, it can make some data privacy requirements tricky. Organizations can notify users about what they want to use data for, but it’s possible what the data actually gets used for, or how it may be changed, or the results from using it may be different.

While users are supposed to be notified before any new purpose is put in place, those doing the work may not know of the change until it’s happened. If data is being analyzed in vast quantities in real time, traditional mechanisms for obtaining user consent, like cookie banners, may not be fast or granular enough, or otherwise sufficient.

User-facing AI systems can be potentially manipulative, resulting in users providing information they didn’t anticipate. Systems may also surface more sophisticated and nebulous connections between data points, enabling identification and profiling at a level we have not seen before. This could potentially turn just about any data into personally identifiable or sensitive data. Current consent requirements may not adequately address this.

While manipulative user interface and user experience functions commonly known as dark patterns are increasingly frowned upon and, in some cases, have been regulated against, those tend to focus on tactics that are already familiar. Responsive design could enable the development of new and more sophisticated ways of manipulating users.

In December 2023, the European Commission, Council of the European Union and European Parliament reached a political agreement on the AI Act, a proposal that was initially released in April 2021. The Act was adopted in March 2024. The EU has 23 official languages, so the Act’s final text has to be extensively translated.

It’s expected that the AI Act will officially enter into force, with the first stages beginning in May or June 2024. EU countries will be expected to ban prohibited AI systems within six months of the regulation’s effective date. Rules will start to apply for general purpose AI systems after one year. After two years, the entirety of the AI Act will be enforceable.

European Commission President Ursula von der Leyen noted the Act’s historic and global potential, “Our AI Act will make a substantial contribution to the development of global rules and principles for human-centric AI.”

The primary goals of the AI Act are two-fold, to respect and protect the fundamental rights of EU citizens, while also boosting innovation. Parliamentarians agreed that how the Act is implemented will be of key importance in achieving these goals.

The EU AI Act is a law on artificial intelligence (AI) adopted by the European Commission in March 2024. It is the world’s first comprehensive law to regulate AI. The aim is to balance positive uses of the technology while mitigating negative ones and codifying rights. There is also a goal to clarify many current and future questions about AI development and make the Act a global standard, as the GDPR has become.

The law assigns applications of AI technology to one of several categories:

Unacceptable risk – AI with unacceptable risks is banned entirely, e.g. the Chinese government’s social scoring tool

High risk – AI with potential risks is permitted subject to compliance with AI requirements and forecasted conformity assessment, e.g. a tool that ranks job applicants by scanning resumes

Medium risk – AI with specific transparency obligations is permitted but subject to information requirements, e.g. bots that can be used for impersonation

Minimal or no risk – AI with no notable risks is permitted without restrictions

All parties have agreed on several main rule categories:

• safeguards regarding general purpose artificial intelligence
• limitations on law enforcement’s use of biometric identification systems
• social scoring using AI is banned
• manipulation or exploitation of users’ vulnerabilities using AI is banned
• consumers have the right to launch complaints and receive meaningful responses

Banned AI applications

Certain applications of AI by corporations, governments, law enforcement, etc., have been banned under the Act, with some exceptions, based on recognized potential threats to the rights, health, and safety of citizens and democracy more generally.

• biometric categorization systems that use sensitive characteristics, aka sensitive data (e.g. political, religious or philosophical beliefs, sexual orientation, race, etc.)
• untargeted scraping of facial images from the internet or closed-circuit television (CCTV) footage to create facial recognition databases (remote biometric identification)
• emotion recognition in the workplace and educational institutions
• social scoring based on social behavior or personal characteristics
• AI systems that manipulate human behavior to circumvent their free will
• AI used to exploit the vulnerabilities of people (due to age, disability, social or economic situation, etc.)

AI with reasonably high potential risks to health, safety, human rights, the environment, etc. is allowed, but is subject to certain requirements like maintaining use logs, ensuring transparency and accuracy, and ensuring human oversight, as well as assessments (before and after going on the market) to reduce risks. This category includes:

• critical infrastructure that could risk citizens lives and health (e.g. transportation)
• essential private and public services (e.g. healthcare, banking, e.g. credit scoring affecting loan qualification)
• education and vocational training (as it could influence access to education and professional opportunities, e.g. exam scoring)
• employment (including management of workers and access to self-employment, e.g. software for resume sorting)
• justice and democratic processes (e.g. court rulings, elections processing)
• law enforcement (certain systems, e.g. evidence evaluation)
• migration, asylum, and border management (e.g. visa application examinations)

General purpose AI (GPAI), risks and obligations

General purpose AI includes tools and applications that tend to be widely available to academia, business, and consumers, e.g. ChatGPT and similar tools. There are further safeguards for more powerful AI models that pose greater systemic risks, including:

• additional risk management obligations
• monitoring of serious incidents
• evaluation of models/modeling
• red teaming (adopting an adversarial approach to rigorously challenge plans, policies, systems, etc.)

Codes of practice around these new requirements will be jointly developed by industry, the scientific community, the public, and others.
It is understood that GPAI systems can do a wide variety of tasks and analysis, and such systems’ capabilities are rapidly expanding. As a result, certain “guardrails” have been agreed upon as control mechanisms:

• transparency requirements for what the systems are designed to do, how, with what data, and for what purposes are clear
• detailed summaries about content used to train AI systems will need to be disseminated
• adherence to EU copyright law
• comprehensive technical documentation

GPAI models with potential high impact and systemic risks will have additional and more stringent requirements:

• conducting model/modeling evaluations
• assessing and mitigating systemic risks
• conducting adversarial testing
• reporting to the European Commission on serious incidents
• ensuring strong cybersecurity
• reporting on energy efficiency
• reliance on codes of practice for regulatory compliance (until harmonized EU standards are published)

Transparency requirements for AI systems and use

The agreed-upon rules for general purpose AI include requirements for transparency about data sources, purposes, etc. But transparency will be a requirement for many systems and uses of AI. Summaries of copyrighted data used for training need to be published. Users will have to be informed if they are interacting with a chatbot, for example. AI generated or edited content must be labeled. If biometrics categorization or emotion recognition systems are in use, users who may be affected must be informed.

Exemptions for AI use by law enforcement

AI-powered tools and systems can be extremely useful to law enforcement, but risks to personal privacy and human rights are also recognized. So a series of safeguards and well-defined exemptions have been agreed upon with regards to the use of real-time biometric identification systems, such as facial recognition, by law enforcement in public spaces.

Such access will require prior judicial authorization, and will be limited to strictly defined lists of crimes. Use of biometric identification systems after the fact, e.g. reviewing footage and analysis, would be done only in the case of a targeted search for a person who has been convicted of a serious crime or is suspected of having committed one.

Use of real-time biometric identification systems would be limited by time and location, and for the following purposes:

• targeted searches of victims (e.g. abduction, trafficking, sexual exploitation)
• prevention of a specific and present terrorist threat
• localization or identification of a person suspected of having committed one of the specific crimes mentioned in the regulation (e.g. terrorism, trafficking, sexual
• exploitation, murder, kidnapping, rape, armed robbery, participation in a criminal organization, environmental crime)

High-risk AI systems, obligations and restrictions

A wide variety of industries, systems and tools can and will be identified as high risk under the Act, including healthcare, financial systems, public infrastructure, and the legal system.

Where AI is used in these areas, assessments before and after launch will have to be done and risk mitigation will have to be implemented or bolstered. Datasets used will have to be of confirmed high quality and copyrighted data summarized and published. Documentation and logging will have to be detailed, there will have to be human oversight, information for users will need to be clear, and strong cybersecurity measures will need to be taken and maintained. Regulatory sandboxes will be used where authorities can facilitate testing of organizations’ systems.

Individuals will be able to launch complaints about AI systems and have the right to receive explanations about decisions based on high-risk AI system activities that may impact their rights.

Support for innovation and SMEs with AI solutions

The legislators understand that AI tools and systems can be strong drivers of innovation in business, and do not want companies, especially SMEs, to be hamstrung by excessive regulation, or be pressured by industry giants with outsized industry influence.
To help mitigate these possibilities, the agreement under the Act promotes the use of regulatory “sandboxes” for development, as well as real-world testing for innovations. National authorities will establish these environments and initiatives to develop and train AI before it is launched to the market.

An AI Office will be established at the EU level, within the European Commission. It will work to coordinate national governance among member countries and supervise enforcement of general purpose AI rules. National authorities within the EU will govern the Act more directly, using qualified market surveillance.

Under the Act there will be multiple levels of fines based on risk and severity of the violation. There are caps on potential fines for startups and SMEs.

• 1.5 percent of global annual turnover or up to € 7.5 million, whichever is higher, for disclosing inaccurate information
• 3 percent of global annual turnover or up to € 15 million, whichever is higher, for violations relating to high-risk systems
• 7 percent of global annual turnover or up to € 35 million, whichever is higher, for violations of unacceptable risk

User consent and data privacy and protection are addressed in the Act’s statutes on a number of fronts.

High Risk – explicit consent is required for use of high-risk AI systems, e.g. critical infrastructure, employment, healthcare, and law enforcement.

Transparency – AI providers must provide clear information about systems’ intended purpose, capabilities, and limitations to ensure users are informed to make decisions and understand potential impacts on their rights.

Right to Explanation – users have the right to obtain meaningful explanations of AI systems’ decisions.

Right to User Control – users should have the ability to opt out, disable, or uninstall AI systems, particularly when fundamental rights or interests are at stake (under some privacy laws users have the right to opt out of “automated decision-making”).

Data Protection and Privacy – the AI Act emphasizes the need for data minimization, purpose limitation, and safeguards to protect personal data when using AI systems, and aligns with existing data privacy regulations like the GDPR.

Companies that acquire data for AI training or other uses can and should ensure that consent was obtained from the sources or users. In some cases it may be a requirement for doing business with partners or vendors.

Consent is also becoming important to monetization strategy. For example, increasingly, premium advertisers are insisting on proof of consent for collection of user data before partnering with app developers.

Companies that collect user data from their own platforms and users for AI training or other uses have direct responsibility for obtaining valid consent and complying with data protection laws. There are a number of ways companies can achieve compliance and valid consent.

Transparency – Privacy laws require clear, accessible notifications, and companies should provide understandable information to users about how user data will be used and processed, including for AI training. As the uses for personal data change, companies need to update their privacy notices, inform users, and, under many privacy laws, get new consent for the new uses of personal data.

Granular consent – Users must be able to accept or decline the collection and processing of their personal data, but they should be able to do it at a detailed level, e.g. approving some kinds of processing, like targeted advertising or AI training, but not others, like sale of the data. This also helps ensure people are informed, which is a requirement for consent to be valid under most privacy laws.

User-friendly mechanisms – Just as notifications must be clear and accessible, the way users accept or decline consent must be easy to understand and access. Information to inform users about data processing must be available there as well as the ability to consent or decline at a granular level. It must also be as easy to decline consent as it is to accept, and under many privacy laws users must also be able to easily change their consent preferences.

Regulatory familiarity – Different jurisdictions have different privacy laws with different requirements and consent models. It’s important for companies to know which laws they need to comply with, and how to do so. It can be important to consult with or appoint qualified legal counsel or a privacy expert, e.g. a data protection officer (DPO), which is also required by some privacy laws. Such a role helps to establish guidelines and processes, update operations, and manage security for data and processing.

Consumers’ rights regarding their personal data depend on a number of factors, including where the user lives and what privacy laws are in place, what the platform is for and what data the user is providing to or generating on it, and what the platform’s terms of service are.

In the European Union, companies collecting and processing personal data must obtain user consent before doing so. This applies equally to social media platforms, a blog, a government website, or an ecommerce store. Users’ data may be collected to learn how people use a site and improve how it works. Or to enable fulfillment when they buy something online, or to show them ads, or to train AI models.

Platforms around the world that are used for financial activities or healthcare have stronger requirements for privacy and security under multiple regulations because of the kinds of information they handle.

In some jurisdictions, it is still allowed to display a cookie banner that says you consent to collection and use of your personal data by continuing to use the site or service. But in the EU and other jurisdictions, this is not acceptable and granular consent is required.

Use of cookies online has been declining as there are newer and better technologies to accomplish what cookies are used for. The question today and going forward is less how AI uses cookies, or may do so, and more how AI could accelerate the replacement of cookies.

Apple and Mozilla have blocked third-party cookies, and Google plans to deprecate them entirely. New tools and methods also enable better data privacy and consent, and can result in higher quality user data.

Current cookie consent models may not be sufficient to cover AI use, since AI systems may analyze large amounts of data in real-time, rather than tools analyzing data from active cookies over time. For consent to be obtained before data collection or use begins, with current pop-ups the user would have to be bombarded with consent banners faster and more often than a human could process them.

AI models can enable more effective ads or personalized user experiences without relying on collection of personally identifiable information, as they can analyze large amounts of data very quickly to group people into audiences based on behaviors. If the system doesn’t need to collect user data, then consent may not be needed, at least for the data collection.

Laws and best practices would likely still require users to be notified of how their behaviors could be tracked and analyzed, and what that analysis could be used for, e.g. personalized ads or shopping experiences. But people’s personal data couldn’t be sold if it was never collected.

AI technology is here to stay. Its capabilities and potential use cases will continue to evolve rapidly. This is a challenge for regulation, as creating and updating laws tends to happen much more slowly than the speed at which technologies develop.

However, users should not be faced with “caveat emptor”, especially online, with regards to new uses for their personal data and challenges to their privacy. Regulators must craft and update laws that are clear and comprehensive, but flexible enough to be interpretable and enforceable today and in the future.

Organizations need to be clear on what privacy regulations they must comply with, what they stipulate, and what that means for their operations. This needs to be regularly reviewed as operations change, and clearly communicated. Trying to sneak in changes to terms of use or use collected data for new purposes without getting new user consent is an easy way to damage brand reputation. It is also illegal in many jurisdictions. As consumers get even more savvy about their data and privacy, companies will have to be more clear, not less, about data collection and use.

Companies should also implement best practices like privacy by design to ensure they are respecting people—the source of their data—and complying with the law. This will also help ensure that consent is obtained and data collection and use limited to legal allowances for all operations, whether fulfilling ecommerce orders or training new AI models.

AI is just the latest technology to bring new challenges to consumers, companies, and regulators alike. It’s not the first and won’t be the last. But best practices for achieving compliance, building trust with users, and successfully growing businesses (or doing science) continue to be the same and serve both organizations and consumers well.

To learn more, talk to our experts today.

Usercentrics does not provide legal advice, and information is provided for educational purposes only. We always recommend engaging qualified legal counsel or privacy specialists regarding data privacy and protection issues and operations.