LLM Applications In Production

1. Key Challenges & Risks

When you deploy LLMs in production, the main risks around data retention & privacy are:

2. Core Principles & Best Practices

To manage those risks, companies typically follow these practices:

1. Data Minimization
   Only collect/store what is strictly necessary. Avoid storing full logs or raw PII unless essential. (Tonic)

2. Anonymization / Pseudonymization
   Remove identifying information (names, addresses, IDs) or replace with tokens. Pseudonymization allows re-identification under controlled conditions; anonymization makes data not attributable. (Data Mastery)

3. Ephemeral Data / Session‐based Memory
   Keep only recent conversation or context, don’t retain for long. Automatically expire or delete data after certain period. (Protecto)

4. Strong Access Controls
   Role-based access, least privilege, authentication, user permissions. Only allow users/systems access to data needed for their function. (Cobalt)

5. Encryption
   Both at rest and in transit. If using storage for logs, vector databases, embeddings, etc., encrypt them. (Cobalt)

6. Sanitization & Filtering of Inputs/Outputs
   Scrub PII or confidential data from inputs (what user sends) and outputs (what model returns). Use input sanitization, output filtering. (Tonic)

7. Transparency / Consent / Privacy Policies
   Inform users which data is collected, for how long, purposes; give option to opt-out, request deletion. (Wooshii)

8. Audit Logging & Monitoring
   Keep records of who accessed what, what data was stored, any anomalous or unauthorized accesses. Also monitor what data is being output by the model (check for leaks). (Cobalt)

9. Privacy-Enhancing Technologies

   • Differential Privacy (add noise, so model can’t reveal individual data). (Lamatic.ai Labs)

   • Federated Learning (keep data locally, only share model updates). (Lamatic.ai Labs)

   • Trusted Execution Environments / Confidential Computing (so computations occur in secure hardware). (Wikipedia)

10. Retention Policies & Data Deletion
    Define how long different types of data (user prompts, conversational history, logs) should be kept, and when/how they must be securely deleted. Also support user “right to be forgotten”. (Superlinear)

3. Typical Procedure / Lifecycle: How to Design & Operate LLM Deployment with Data Retention & Privacy in Mind

Below is a procedural workflow (step-by-step) that a company can follow to ensure compliance. You can adapt to your size, industry, and legal environment.

Step A: Initial Design & Legal Assessment

1. Map Data Types

   • Identify what data the system will collect/use: namespace (customer data, internal documents, PII, IP, etc.).

   • Identify which of that is sensitive, regulated, or specially protected (e.g. personally identifiable, financial, health).

2. Legal & Regulatory Check

   • Check what laws/regulations apply in Vietnam (e.g. Law on Cybersecurity, Personal Data Protection law when enacted), and in any country where your customers / data reside.

   • Understand cross-border data transfer laws (if data stored overseas or processed in cloud services located outside Vietnam).

3. Define Retention Requirements

   • For each data type, define: how long to keep, who can access, under what conditions to delete.

   • Define criteria for data deletion (e.g. after X days, after account termination, when no longer needed for business or compliance).

4. Privacy by Design

   • Architect system so privacy is built in (not added later).

   • Minimize capturing PII; build filtering/sanitization into data pipelines.

5. Vendor / Third-Party Assessment

   • If using external LLM API providers, check their policies: retention, training on user data, logging, data residency.

   • If possible, choose providers that offer “no training on user input”, “zero / minimal retention”, or allow hosting models on your infrastructure.

Step B: Implementation

6. Access Control & Authentication

   • Configure RBAC (role-based access control).

   • Secure APIs, endpoints. Ensure only authorized roles can see sensitive data, logs, etc.

7. Input & Output Sanitization

   • Before storing data (or sending to model), strip / mask sensitive fields (names, IDs, addresses) where possible.

   • Check outputs (especially if content includes retrieved documents or previous user data) to ensure they don’t inadvertently leak PII from other users.

8. Memory / Context Handling

   • If the model has memory (e.g. chat history, user profile, vector embeddings), limit what is stored. Only necessary context, and possibly only recent context.

   • Also context should be scoped / isolated per user or per domain, so users can’t retrieve other user’s data via prompts.

9. Encryption Everywhere

   • Data in transit (APIs, internode communication) → use TLS.

   • Data at rest (databases, vector stores, logs) → encrypt.

   • Consider encryption for backups, archives.

10. Audit & Logging

    • Log who accessed data / actions performed. Logs should themselves be protected (privacy, encryption, retention).

    • Track usage (how many prompts, what context, which user roles).

11. Retention & Deletion Mechanisms

    • Implement automated deletion for old data. For example: after 30/60/90 days, or after user’s account closed, etc., depending on your policy.

    • Ensure deletion is secure (i.e. data not just marked “deleted”, but removed / overwritten / purged).

12. Privacy Enhancing Measures (if needed)

    • Differential privacy during training/fine-tuning.

    • Federated learning if you want to avoid sending raw data elsewhere.

    • Use secure / confidential computing platforms for sensitive operations.

Step C: Monitoring, Review, & Incident Response

13. Monitoring and Alerts

    • Watch for unusual behavior: e.g. model outputs leaking private data, API endpoints receiving suspicious inputs, unauthorized access attempts.

    • Also monitor retention statistics (how much data stored, approaching retention limits).

14. Audits & Privacy Impact Assessments

    • Periodic audits to ensure that practices (data deletion, sanitization, access control) follow your policies.

    • Privacy Impact Assessments (PIA) especially when adding new data sources or new functionality.

15. User Rights & Transparency

    • Provide means for users (customers, employees, etc.) to request their data (view), correct data, delete data.

    • Publish clear privacy policies showing how long data is stored, what is collected, what is shared.

16. Incident Response Plan

    • Have plan in case of data breach.

    • Define who will notify (regulators, customers), what logs will be traced, how remediation will occur.

4. Example: How Your Factories Might Apply This

Given your operations (multiple factories, manufacturing, customer orders, internal HR etc.), here’s how you might concretely apply the above: