Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

cryptography, security, privacy, cybersecurity

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem VINCENZO LONGO, Politecnico di Torino, Italy ALBERTO VERNA, Politecnico di Torino, Italy NIKHIL JHA, Politecnico di Torino, Italy MARCO MELLIA, Politecnico di Torino, Italy The Microsoft 365 (M365) ecosystem hosts thousands of third-party applications that integrate with enterprise tenants via fine-grained OAuth permissions, potentially granting access to sensitive organisational resources such as emails, files, calendars, chats, and user directories. Despite the security implications of these permission grants, the M365 ecosystem has not been systematically studied. We present the first privacy- and security-oriented measurement of M365 third-party applications. By combining public marketplace APIs with automated tenant-side deployment, we crawl over 8,000 applications. We find that only 1,069 of them expose both descriptions and permission sets, with significant inconsistencies in transparency across official distribution channels. Next, we leverage a topic-aware anomaly detection framework to assess whether requested permissions align with declared

arXiv:2608.02336v1 [cs.CR] 3 Aug 2026

functionality. We cluster applications via Neural Topic Modelling and apply unsupervised anomaly detection within each topic to identify deviations from peer permission profiles. LLM-assisted analysis of the most anomalous cases and blind manual inspection reveal a correlation between anomalous permission profiles and the risk associated with the requested permissions. We find that many applications request overly broad tenant-wide scopes (e.g., directory-wide read/write access), violating least-privilege principles and increasing the organisational attack surface. Our pipeline provides tenant administrators with actionable insights by identifying anomalous applications and the permissions that most contribute to their anomalousness. Overall, our findings expose systemic opacity and structural immaturity in the M365 app ecosystem, where permission disclosure is inconsistent and over-privileged access is common. Additional Key Words and Phrases: Microsoft 365 characterisation. Neural Topic Modelling, Anomaly Detection, Permission Misuse. ACM Reference Format: Vincenzo Longo, Alberto Verna, Nikhil Jha, and Marco Mellia. YYYY. Lost in Permissions: Exploring the Microsoft 365 App Ecosystem. In Proceedings of . ACM, New York, NY, USA, 26 pages. https://doi.org/XXXXXXX.XXXXXXX

Introduction

Microsoft 365 (M365) is widely deployed in enterprise environments and enables third-party applications to integrate with tenants through a fine-grained OAuth permission model, which comprises more than 300 distinct permissions. These may grant access to highly sensitive organisational resources, including emails, files, chats, calendars, and tenant-wide user directories. While applications extend platform functionality, the permissions they request may also introduce significant privacy and security risks when they request excessive or misaligned access. Authors’ Contact Information: Vincenzo Longo, Politecnico di Torino, Torino, Italy, [email protected]; Alberto Verna, Politecnico di Torino, Torino, Italy, [email protected]; Nikhil Jha, Politecnico di Torino, Torino, Italy, [email protected]; Marco Mellia, Politecnico di Torino, Torino, Italy, [email protected]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © YYYY Copyright held by the owner/author(s). Publication rights licensed to ACM. Manuscript submitted to ACM Manuscript submitted to ACM

Longo et al. The M365 ecosystem offers multiple channels to install apps: the official Teams and Office add-ins stores, the global

M365 Marketplace, and, lastly, any generic third-party app can access M365 functionalities via API. Curiously, permission disclosure is inconsistent across distribution channels: some stores expose detailed permission lists; others omit them; some do not even require explicit consent during installation. Security best practice requires applications to follow the principle of least privilege, requesting only permissions strictly necessary for their functionality. This may not always be the case, with developers who can ask for permissions that are unrelated or beyond the application’s true needs, causing potential privacy issues or malicious access to the user’s and organisation’s resources. Unlike mobile ecosystems [1, 12, 16, 19, 21], M365 apps can obtain tenant-wide scopes, exposing organisation-wide data and increasing the attack surface of enterprise environments. Despite the scale and sensitivity of the M365 ecosystem, no prior work has systematically analysed the permission landscape of third-party applications or assessed whether requested scopes align with declared functionality. In this work, we compile the first dataset of M365 applications by aggregating data from multiple sources: Marketplace APIs, store Web scraping, and applications installed by our University’s M365 service users. Out of more than 8,000 apps we crawled, only 1,069 expose both a textual description and the list of requested permissions – the remaining ones simply do not make this information publicly accessible. Our dataset provides the first consolidated view of permission usage in the M365 app ecosystem. Next, we propose a topic-aware anomaly detection framework that models permission profiles within semantically coherent clusters of applications, enabling detection of deviations relative to functional peers rather than across unrelated apps. We apply Neural Topic Modelling (NTM) to cluster applications according to their declared functionality, obtaining 24 semantic topics. To identify applications whose permission sets deviate from those of their peers, we apply unsupervised anomaly detection (AD) within each topic. We highlight both over- and under-privileged applications, which we validate through LLM-assisted evaluation and blind manual inspection. We identify applications requesting tenant-wide read/write access to directories, calendars, chats, and files without clear alignment to their declared purpose, as well as applications lacking permissions required to accomplish their stated goals. Results indicate that topic-aware anomaly detection captures meaningful deviations in permission usage that are often associated with potentially risky permissions. As such, anomaly scores provide tenant administrators with an additional criterion for assessing whether an application deserves further scrutiny before approval. In summary, this paper makes four contributions: • we construct the first consolidated dataset of M365 third-party applications, combining marketplace APIs, automated deployment, and operational tenant data; • we introduce a topic-aware anomaly detection framework to assess permission-functionality alignment; • we empirically characterise systemic inconsistencies and over-privileged access patterns in the M365 ecosystem; • we present our pipeline as a decision-support tool for evaluating the risk of granting an application access to a tenant. To allow reproducibility and further investigation, we share both the collected dataset and the analysis pipeline.1 The rest of the paper is structured as follows. In Section 2, we offer a brief overview of the Microsoft 365 ecosystem. In Section 3 and Section 4, we describe the process of retrieving the dataset and running a semantically-coherent clustering of the applications, respectively. In Section 5 we show the outcome of the anomaly detection algorithms, and 1 Code and dataset will be shared upon the paper’s acceptance, to preserve authors’ anonymity.

Manuscript submitted to ACM

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

(a) Installing the Wikipedia app on Microsoft Word.

(b) Installing the Wikipedia app on Microsoft Teams.

Fig. 1. Example installation interfaces for the same application across different M365 distribution channels, illustrating inconsistencies in permission disclosure.

in Section 6 we observe the correlation between an app’s anomaly and its riskiness. We summarise related works in Section 7 before drawing conclusions and outlining perspectives for future research on the topic in Section 8. 2

The Microsoft 365 ecosystem

In this section, we provide an overview of the quite complex Microsoft 365 ecosystem, what it offers and how it is structured. We introduce the concept of tenant, the applications within it and the permission model that governs the access to resources. 2.1

Microsoft Entra ID and Tenants

M365 services are organised into isolated environments called tenants. A tenant represents a dedicated instance of Microsoft 365 associated with a specific organisation, containing its users, applications, and data. Each tenant is backed by a Microsoft Entra ID directory, which manages identities and authorisation. When a developer registers an application globally, an application object is created. Upon installation in a tenant, a corresponding service principal is instantiated locally. Service principals therefore represent applications within a tenant. They act as the application’s identity within that tenant and are the entities to which permissions are granted. Permissions assigned to a service principal determine which of the tenant resources the application can access, including user-specific data and organisation-wide resources. As a result, analysing service principals and their associated permissions provides a direct view of the access surface introduced by third-party applications within a tenant. 2.2

OAuth Permission Model in M365

Authorisation in M365 relies on the OAuth 2.0 framework, where applications obtain access tokens containing explicitly granted permissions (also referred to as scopes). These permissions define the set of resources and operations an application is allowed to perform within a tenant. M365 distinguishes three main types of permissions: Delegated permissions allow an application to act on behalf of a signed-in user. In this case, the application’s access is limited to the resources that the user is authorised to access. Application permissions grant an application direct access to resources at the tenant level, independently of any signed-in user. These permissions typically require administrator consent and may enable organisation-wide access Manuscript submitted to ACM

Longo et al.

to sensitive resources such as directories, mailboxes, files, or chats. Resource-Specific Consent (RSC) permissions provide scoped access to specific Teams resources (e.g., a particular team or chat), enabling finer-grained control compared to tenant-wide permissions. For instance, a person owning both team A and team B can consent to an application accessing only the resources of team A without affecting team B. From a security perspective, application permissions are particularly sensitive, as they may enable persistent and tenant-wide access to organisational data. Analysing the type and scope of permissions requested by service principals, therefore, provides insight into the potential attack surface introduced by third-party applications. 2.3

Microsoft Graph and permission pattern

Microsoft Graph is the primary API through which third-party applications access M365 resources. These resources are locked behind authentication and role-based authorisation, which is managed through permissions. While OAuth standards do not define a strict pattern for naming permissions, the best practice is to use the pattern <resource>.<operation>.<constraint>, where • resource refers to a Microsoft Graph resource to which the permission allows access (e.g., User, Chat, Mail, Files, ...); • operation refers to the Microsoft Graph operations that are allowed on the data exposed by the resource (e.g., Read, ReadWrite, ReadBasic, Create, Send, ...); • constraint determines the potential extent of access an app has within the directory (e.g., All, Directory, OwnedBy, Shared, ...). This field is optional, and its default value is OwnedBy. For example, User.Read allows an application to read the profile of the signed-in user, whereas User.Read.All enables reading the profile information of all users in the tenant. The All constraint therefore expands access from user-scoped to tenant-wide. 2.4

Application Types and Distribution Channels

Applications integrating with M365 can be broadly divided into two categories: • Add-ins: extend built-in services such as Teams, SharePoint, Outlook, or Word by adding UI elements, commands, tabs, or workspace-level functionality. • Third-Party Applications: externally-provided applications, often SaaS services, that interact with M365 resources through Microsoft Graph APIs (e.g., project management platforms or calendar integrations). These categories are not mutually exclusive: an add-in may also request Microsoft Graph permissions and be represented in the tenant as a service principal. Applications can be installed through multiple distribution channels, including the official Teams and Office stores, and the global Microsoft Marketplace.2 When installed, each application requests OAuth permissions according to its functionality. One example is the “Wikipedia” app for Word and Excel,3 that allows users to directly browse Wikipedia within Word or Excel. In addition, a tenant or a user can distribute custom-developed app and offer them to users in other tenants. These custom app are not listed in any store. At last, third-party apps can access M365 resources via API. For instance, Zoom or Notability can request to access Outlook Calendar or OneDrive files in the M365 cloud. No official store or repository exists, as any third-party app 2 Microsoft calls Office add-ins and Teams Apps those app which extends Office or Teams products, respectively; the term Apps for Microsoft 365 is the

generic term which includes all options. 3 https://marketplace.microsoft.com/en-us/product/office/wa104099688, accessed on August 4, 2026. Manuscript submitted to ACM

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

can use the Entra ID API, a subset of the Graph API. All apps are instantiated as service principals within an Entra ID tenant, as previously described. Importantly, permission transparency and consent mechanisms vary across these channels. Some stores expose detailed permission lists before installation, while others provide limited or no visibility into the scopes that will be granted. This fragmentation complicates systematic auditing and motivates the need for an ecosystem-wide analysis of requested permissions. It is important to note that the decision whether to allow or not the installation of an app in a tenant is totally delegated to the tenant administrator. Given the opaqueness and lack of information in the M365 app ecosystem, admins need tools to make informed decisions about authorising the use of an app. 3

Dataset collection

Given our goal of exploring the permissions apps ask for, we need at least three main features: • App ID: the app identifier and name. • Description: a textual description of the app’s purposes and functionalities, later used for topic modelling. • Set of Permissions: the set of requested permissions, required for the AD stage. 3.1

Challenges in dataset retrieval

The M365 ecosystem does not provide a unified catalogue exposing all these attributes. Applications are distributed across multiple channels (e.g., Teams store, SharePoint store, Microsoft Marketplace, and direct third-party integrations), and each exposes different subsets of metadata. In particular, permission information is often incomplete, inconsistently disclosed, or accessible only after installation within a tenant. Figure 1 shows three examples of installing the Wikipedia application in different contexts: Microsoft Word, Microsoft Teams, and the Microsoft Marketplace, which offer a version of the application compatible with Word and Excel. Each of the stores follows a different path for installing the app, and, most importantly, the permission-related information is not expressed (or present) in a consistent way, complicating a user’s understanding of the privacy-related consequences of one’s choice. For instance, among the three options, only Microsoft Teams (Figure 1b) provides the users with some information about the permissions that will be requested from the application. This fragmentation prevents direct large-scale measurement and necessitates a multi-source data collection strategy. We therefore combine public APIs, automated tenant-side deployment and scraping, and operational tenant data to construct the most comprehensive dataset currently attainable. Figure 2 summarises the different collection paths: (1) We leverage public APIs that multiple app marketplaces (Marketplace, Teams Store, SharePoint Store, etc.) expose to retrieve the list of apps. (2) We design a Web scraping tool that automatically accesses a test tenant, deploys apps, accepts installation prompts (if any) and extracts their properties through the Entra ID portal; (3) We analyse apps that were installed in our University’s operational tenant. The set includes some third-party apps that are not present in the other sources. Merging and consolidating the information from these different sources requires ingenuity, given that no unique app ID is available. Manuscript submitted to ACM

Longo et al. Public API path

Teams App Catalog

Teams dataset Sharepoint Store

Web crawler path

Marketplace crawler

Entra ID portal permissions

Sharepoint dataset

AppSource dataset Total dataset

Marketplace API

Microsoft Defender for Cloud Apps API

Marketplace descriptions

Third-party apps descriptions

University tenant path

University dataset

University permissions

Fig. 2. Overview of the data collection pipeline via public API extraction, tenant-side deployment, and operational tenant data integration.

3.2

Public APIs

We first leverage publicly available APIs exposed by official Microsoft application stores to retrieve application metadata at scale. In particular, we collect data from the Microsoft Teams app catalogue and the SharePoint store, which together host a substantial portion of M365 add-ins. Teams store: The Microsoft Graph provides an officially documented API4 to access the Microsoft Teams app catalogue. Crawled in September 2025, it contains about 2,800 apps with descriptions, out of which only 347 contain a non-empty RCS permission field. SharePoint store: When logged into SharePoint, a user can access the SharePoint Store5 to install apps. Given the app ID (obtained via the Marketplace API), it is possible to retrieve the list of requested permissions by calling an undocumented and authenticated API.6 The API requires setting the right cookies within the request headers. Following the crawl, 221 apps out of 1,222 have a non-empty permission field. Across both sources, permission disclosure is partial and inconsistent: many applications provide descriptions but omit their requested OAuth scopes. This limitation prevents comprehensive ecosystem analysis using public APIs alone and motivates the need for additional collection strategies. 3.3

Scraping Marketplace for Apps

The Microsoft Marketplace exposes application descriptions but does not publicly disclose the full set of requested OAuth permissions. Since permission information becomes visible only after installation within a tenant, we implement an automated tenant-side deployment procedure to retrieve this data. 3.3.1 Retrieving list of apps and descriptions. First, we use undocumented but publicly accessible Marketplace endpoints to enumerate applications and retrieve their metadata (e.g., name and description). Crawling in September 2025 identifies 4 https://graph.microsoft.com/v1.0/appCatalogs/teamsApps?\protect\T1\textdollarexpand=appDefinitions, accessed on August 4, 2026

5 https://<TENANT_NAME>.sharepoint.com/sites/appcatalog/_layouts/15/appStore.aspx/sharePointStore, accessed on August 4, 2026

6 https://<TENANT_NAME>.sharepoint.com/sites/appcatalog/_layouts/15/storefront.aspx?task=GetAppAdditionalDetails&bm=US&cm=en-

US&appid=<APP_ID>&catalog=0&uirequest=1, accessed on August 4, 2026. Manuscript submitted to ACM

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

8,232 add-ins across M365 products (Table 1). The API returns the apps’ description and additional information (e.g., rating, popularity scores, etc.). Yet, these endpoints do not expose permission information. Table 1. Marketplace applications per M365 product category. The total is lower than the sum of individual counts because some applications are listed under multiple products.

Product

Total Apps

Excel

1,338

Office

412

Outlook

2,069

PowerPoint

480

SharePoint

1,222

Teams

2,595

Word

1,306

Total (unique)

8,232

3.3.2 Collecting permissions from a tenant. To obtain permissions, we create a dedicated test tenant and implement an automated deployment tool using Selenium WebDriver. For each application, the tool initiates installation via the Marketplace interface, completes authentication with administrative credentials, accepts consent prompts when present, and extracts the granted permissions from the Entra ID admin portal. This process allows us to retrieve the effective OAuth scopes associated with each service principal.7 The tool handles most of the exceptions that could happen during the deployment process, such as the absence of the OAuth prompt, timeouts, generic errors, etc. Despite this, the crawler successfully deployed 432 apps listed in Marketplace (about 4% of the total). This low success rate is primarily due to (i) the OAuth consent unexpectedly failing during deployment (further attempts do not help)8 ; (ii) apps not asking for permissions when deployed (useless for our purpose); (iii) the deployment process of the app requiring additional steps (e.g., registration through the app portal). We acknowledge that, for some applications, automatically installing them may be against their terms of service. However, we only install the applications without ever using them, restricting the impact of the automated data extraction only to Microsoft servers, for which the few hundred of downloaded applications represent a negligible load. 3.3.3 Linking descriptions and permissions. Applications extracted from the Marketplace APIs and those observed in the Entra ID portal may differ in naming conventions. For example, an application listed in the Marketplace under a generic name may appear within a tenant with product-specific suffixes (e.g., “for M365” or “– Teams”), resulting in distinct service principal names. To associate application descriptions with their corresponding permission sets, we adopt a two-stage matching strategy. First, we attempt an exact match between the app names and service principal display names. When no exact match is found, we apply a controlled partial matching procedure based on manually curated regular expressions that normalise common suffixes and prefixes. 7 The crawling tool’s source code will be available upon acceptance.

8We frequently encountered this error also during manual deployment.

Manuscript submitted to ACM

Longo et al. Table 2. Number of applications collected from each source prior to deduplication.

Source

# of Apps

Crawler University SharePoint Teams Total

432 285 221 348 1,286

To minimise incorrect associations, we inspect all ambiguous matches and discard cases where multiple candidate associations cannot be resolved confidently. Applications with uncertain mappings are excluded from the dataset. After merging crawler-derived permissions with metadata from other sources, the crawler contributes 330 unique applications. While deployment success is limited to a subset of Marketplace applications, successfully deployed apps span multiple product categories and exhibit heterogeneous permission counts, with coverage across diverse app types. This conservative approach prioritises precision over recall in linking descriptions to permissions, reducing the risk of misattributing permission sets to incorrect applications. 3.4

University tenant apps

To complement the Marketplace-based collection, we analyse applications installed in our University’s operational M365 tenant, comprising more than 35,000 users who installed applications in the tenant. The file represents the snapshot in May 2025.9 This source captures both add-ins and third-party applications actively deployed in a real-world enterprise environment, including integrations not listed in official stores. University IT administrators provided an anonymised export of service principals present in the tenant. The dataset includes 285 applications along with their associated permission sets. Because the Entra ID interface does not expose application descriptions, we retrieve corresponding descriptions by querying Marketplace metadata and, when necessary, the Microsoft Defender for Cloud Apps API. As before, naming discrepancies between service principals and Marketplace listings require controlled matching. We apply the same two-stage matching strategy and manually validate ambiguous cases to ensure correct associations between descriptions and permission sets. After filtering unmatched or ambiguous entries, this source contributes 285 applications with both descriptions and permissions.10 3.5

The final dataset

We merge applications collected from public APIs, Marketplace deployment, and the operational tenant into a unified dataset. Because no global identifier exists across distribution channels, consolidation requires careful deduplication and validation. Handling duplicates. Given that no global identifier exists, we check for possible duplicates using the app’s official name and that of the service principal: for apps appearing in the Teams and the crawler data (49 apps), we merge permission information as the RCS and tenant-wide permissions are complementary. For duplicates in the Teams 9 In mid-2025, Microsoft updated its M365 security baseline, switching the third-party application consent policy from user-level opt-out to admin-required

opt-in to prevent unauthorized data access. Since then, users cannot install any new application.

10 Permissions observed in an operational tenant may reflect administrator-imposed restrictions and therefore represent an effective permission

configuration rather than the full set declared by the developer. In our case, all apps were allowed with their default permissions. Manuscript submitted to ACM

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

Table 3. Summary of dataset consolidation and preprocessing steps, showing the resulting application count at each stage.

Preprocessing Step

# of Apps

Initial number of apps with descriptions Handling duplicates Handling discrepancies Filtered out (not in English) Filtered out (less than ten words)

1,286 −96 −61 −40 −20

Final number of apps

1,069

CCDF

100 Sharepoint Teams University Crawler

10−1 10−2 10−3

10 20 30 Number of permissions per application

Fig. 3. CCDF of the number of requested permissions per application, stratified by data source.

and the University data (19 apps) and in the crawler and University data (28 apps), we keep the Teams and crawler permissions that are more complete (University data may reflect tenant-level filtering). Handling incorrect matches. We next look for discrepancies that could have been introduced during the “direct-thenpartial” match approach used to complete the description field. Specifically, we look for those apps with the same title (i.e., the same manifest) but different appName (i.e., different Service Principals). These cases may happen because the same apps might trigger the creation of multiple Service Principals (e.g., Zoom → “Zoom for M365” and “Zoom for Teams”). After manually checking all these candidates, we drop 61 apps with ambiguous associations. Description cleaning. Finally, we perform a basic pre-processing step consisting of filtering out apps with very short descriptions (i.e., less than 10 words) or with non-English descriptions. We also clean the description text by removing HTML tags and non-alphanumeric characters. Table 3 summarises the pre-processing steps and the final number of apps we use for our analysis. Overall, the coexistence of multiple app stores, incomplete metadata, the absence of a unified management interface, and non-standardised Service Principal names permits us to retrieve information for only 1,069 out of more than 8,000 applications, highlighting the current fragmentation and limited maturity of the M365 app ecosystem—see Table 3. Manuscript submitted to ACM

Longo et al.

Read and ReadBasic ReadWrite

Permissions

400

Write Other

300 200 100 0

200 400 600 800 1000 1200 App rank by number of requested permissions

Fig. 4. Permission–application matrix. Applications (x-axis) are ordered by decreasing number of requested permissions; permissions (y-axis) are grouped by operation type.

3.6

Dataset characterisation

We analyse the distribution of requested permissions across the 1,069 applications in the final dataset. Figure 3 reports the complementary cumulative distribution function (CCDF) of the number of permissions requested per application, stratified by data source. The distribution exhibits substantial variability both across and within sources. Applications obtained via Marketplace deployment display a heavier tail, with several apps requesting more than 20 permissions. In contrast, applications observed in the operational tenant tend to request fewer permissions, likely reflecting administrator-level filtering or differences in app type. Across all sources, permission counts range from 1 to 32 per application. In total, we observe more than 300 distinct OAuth permissions. Some permissions are highly prevalent: for example, 48% of applications request User.Read, 37% request openid, and 36% request profile. Conversely, a large number of permissions appear only in a small fraction of applications. Figure 4 visualises the full permission–application matrix. Applications are ordered by decreasing number of requested permissions, while permissions are grouped by operation type (e.g., Read, ReadWrite, ...). The figure highlights both highly permission-intensive applications and others requesting rare or uncommon scopes. Rarer permissions are requested by the more permission-demanding apps (dots in the leftmost part of the plot). Interestingly, some apps request very few but rarely requested permissions (dots in the rightmost part of the plot). This heterogeneity suggests that applications differ significantly in their effective privilege profiles. In particular, some applications request extensive combinations of permissions, while others request only a minimal subset. We hypothesise that part of this variability reflects differences in declared functionality, motivating a topic-aware analysis of permission patterns. 4

Topic retrieval

Given the heterogeneity of apps and permissions, we proceed by clustering apps based on their purpose. For this task, we use the applications’ description and rely on state-of-the-art topic modelling to group apps into clusters. We use BERTopic [8], one of the most recent, scalable and effective models for topic extraction, leveraging transformer-based embeddings. Manuscript submitted to ACM

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem 4.1

Topic retrieval methodology

BERTopic is organised in a six-block pipeline: (1) Embedding Model: the first step consists of the extraction of document embeddings. The default embedding model is Sentence-BERT [17], which is highly capable of capturing the semantic similarity among documents. This step maps each document in the embedding space where semantically similar (different) documents are projected nearby (far apart). (2) Dimensionality Reduction: As the embedding space is usually high-dimensional, BERTopic reduces the representation space dimensionality to avoid the curse of dimensionality problem during the clustering step. BERTopic uses UMAP [18], one of the most effective algorithms in preserving both the local and global highdimensional space in lower dimensions. (3) Clustering Algorithm: To find clusters of documents with similar topics, BERTopic uses HDBSCAN [11], a parameter-free and effective algorithm to capture clusters of varying densities. The results are clusters of documents that are semantically similar. (4) Word Vectorisation: To characterise each topic, BERTopic adopts a bag-of-words representation. All documents within a cluster are combined, and each word is represented as a feature whose value corresponds to its frequency within that cluster. We employ the CountVectorizer tool to compute word occurrence counts, which are later used to identify the most representative terms for each topic. (5) Topic Representation: The final step is to extract the most characterising words for a topic, i.e., words that are frequent in such a topic, and rare in others. BERTopic uses a class-based Term Frequency-Inverse Document Frequency (c-TF-IDF) for this. The result is the extraction of keywords that well represent each topic. (6) Topic Fine-Tuning: Optionally, one can further refine the topic representation by using other Natural Language Processing techniques. We do not proceed with this step. All the blocks composing the BERTopic pipeline require tuning of their own hyperparameters. The objective is to extract the most coherent topics possible from the descriptions of the applications in the dataset. To achieve this, we follow a structured cascade approach for hyperparameter tuning: for each model in the pipeline (i.e., SBERT, UMAP, HDBSCAN, CountVectorizer, c-TF-IDF), we select the best hyperparameters and the possible values to test. For each block, we select the best-performing configuration based on objective metrics such as the clusters’ silhouette scores and semantic coherence. Having fixed the best parameters, we move on to the next block. We repeat each experiment using three different seeds to ensure stability of the results and robustness against randomness. We detail the BERTopic final parameters and the search space values in Appendix B. 4.2

Retrieved topics

We run BERTopic using the description of each app as input documents. As output, BERTopic returns clusters of apps whose descriptions are semantically similar, along with the top-5 keywords that better characterise their topic extracted using the c-TF-IDF scores. We identify 24 clusters, which we manually name in a more descriptive form based on the returned keywords. As the keywords only serve to provide an intuition for the cluster app goals, we consider 5 keywords as a plausible trade-off between synthesis and semantic richness. The topics reflect the nature of M365 apps and services, with the most popular ones related to productivity and collaboration. Clusters contain a variable number of apps, the largest one containing 120 of them ("File management" topic), 40% of them containing more than 50 applications, 30% containing less than 30 apps. Manual inspection indicates Manuscript submitted to ACM

Longo et al.

coherent separation of applications into semantically meaningful topics. Here, we provide a qualitative interpretation of the top four and the bottom four clusters by number of applications. For completeness, we report the complete list, their top-5 keywords, description, and number of applications in Appendix C. The four most popular topics revolve around the following functions: • File Management, 122 apps: applications related to file storage, sharing, and handling; • Security, 75 apps: applications focused on asset management and protection; • Notifications, 65 apps: applications that provide notifications and alerts to users; • E-learning, 65 apps: applications for online learning and educational purposes. On the other hand, the least popular topics are more specific and represent niche or innovative areas of the M365 application world in our dataset. These are: • Human Resources, 20 apps: applications focused on human resources management, recruitment and life cycle handling; • AI Translation, 19 apps: applications focused on translation services using artificial intelligence; • PDF Management, 15 apps: applications focused on PDF document management and manipulation; • AI Legal Assistant, 12 apps: applications designed for legal assistance and document review using artificial intelligence. 4.3

Topics and permissions

To analyse how permissions vary across functional categories, we construct a permission profile for each topic. Let P denote the set of all distinct permissions observed in the dataset. For each topic 𝑖, we define a vector 𝑃𝑖 ∈ R | P | , where each component corresponds to a permission 𝑝 ∈ P. The term frequency (TF) of permission 𝑝 in topic 𝑖 is defined as the proportion of applications in topic 𝑖 requesting 𝑝. To emphasise permissions that are distinctive to specific topics, we apply an inverse document frequency (IDF) weighting computed across topics. The resulting TF-IDF representation mitigates the effect of topic size and down-weights globally common permissions. We compute pairwise cosine similarity between topic vectors: 𝑠 (𝑖, 𝑗) =

𝑃𝑖 · 𝑃 𝑗 , ∥𝑃𝑖 ∥∥𝑃 𝑗 ∥

where higher values indicate more similar permission profiles. Figure 5 reports the resulting similarity matrix. While some topics exhibit overlapping permission patterns—reflecting shared resource requirements—others display markedly distinct profiles (e.g., “Intranet & WebParts” and “AI Translation”), suggesting functional differentiation in access needs. Overall, apps with different purposes request distinct permission sets, confirming that permission profiles mirror functional goals. To further examine intra-topic variability, Figure 6 reports the complementary cumulative distribution functions (CCDFs) of the number of permissions requested by applications within selected topics. Even among applications serving similar functional purposes, we observe considerable dispersion: some apps request only a small number of permissions, while others request more than 20. This pattern holds across both general productivity topics and AI-related clusters. Figure 6 shows the distribution of permissions within the same topic, and across different—but related—topics. In Figure 6a, we compare the permission distribution for the four most popular topics. Security-related apps request a Manuscript submitted to ACM

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

1.0

AI Translation Notifications Employee Recognition Well-being & Productivity Collaborative Brainstorming Surveys & Feedback Communication Tools AI Knowledge Base Security Legal AI Assistant File Management Digital Signatures E-Learning Project Management Confluence Integration Scheduling & Calendar Workspace Booking Expense Management AI Meeting Assistants PDF Management SharePoint Integration Intranet & WebParts Human Resources Forms & Workflows Design

0.6

Cosine similarity

0.8

0.4

0.2

Fig. 5. Pairwise cosine similarity between topic-level permission profiles computed from TF-IDF weighted permission frequency vectors.

1.00

1.00 File management Security Notifications E-Learning

0.50 0.25 0.00

AI Knowledge Base AI Meeting Assistants AI Translation

0.75 CCDF

CCDF

0.75

0.50 0.25

15 20 Permissions

(a) Five larger topics.

0.00

20 Permissions

(b) Three topics related to Artificial Intelligence.

Fig. 6. Complementary CDFs for the number of permissions requested by applications within selected topics.

larger share of permissions. All the topics present at least one application that requests more than 20 permissions. The same pattern emerges by comparing applications offering AI-driven services—see Figure 6b. “AI Meeting Assistants” tend to request more permissions, with ≈20% of them requesting more than 10 permissions. Curiously, almost 40% of “AI Translation” apps require just one permission, mostly OnlineMeetingParticipant.ToggleIncomingAudio.Chat. Yet other apps that offer similar functionalities request 10-20 permissions. Manuscript submitted to ACM

Longo et al.

Ratio of apps

File management Scheduling & Calendar Notifications Security

0.6 0.4 0.2 0.0

Re er. Us

eni

ess file acc pro ne i offl

ll ll ll te at ail .A .A c.A dWri Ch em ead Basi ead te. a i R R e r . . W er ry .R ad Us ge. Re ars cto er. nd ire Sta e g l D Us n Ca eti Me Permissions

Fig. 7. Fraction of applications within selected topics requesting each of the ten most globally frequent permissions.

These results show that applications requesting a high number of permissions appear in every topic, while other applications, on the other hand, require very few permissions. This high disparity suggests the possible presence of outliers inside each topic. To further corroborate the assumption that different topics require different permission sets, we plot in Figure 7 the percentage of applications inside each topic that request a given permission. We select the four topics with the largest number of applications. We consider the top-10 permissions according to their global popularity inside our dataset. For each topic, we report the ratio of apps requesting such a permission. Different patterns exist for different topics. For instance, File management applications tend to request more permissions about User.Read, openid, and profile with respect to the other topics. On the other hand, Notification applications require a smaller share of the top permissions. At last, we observe that only a small percentage of apps request less popular topics (e.g., Calendars.ReadWrite and MeetingStage.Write.Chat). Taken together, these observations suggest that permission usage exhibits both inter-topic structure and intra-topic variability. While functional similarity explains part of the permission patterns, the presence of significant deviations within topics motivates a topic-aware anomaly detection approach to identify applications whose permission profiles diverge from their functional baseline. 5

Anomaly detection

Our final objective is to identify applications whose requested permissions deviate from those of other applications serving similar functional purposes. 5.1

Anomaly detection methodology

We employ three complementary unsupervised anomaly detection algorithms: One-Class SVM (OC-SVM), Local Outlier Factor (LOF), and Isolation Forest (iForest) [3]. OC-SVM models a global decision boundary enclosing the majority of samples; LOF detects local density deviations relative to neighbouring samples; iForest isolates anomalous points through recursive random partitioning. Using multiple paradigms reduces reliance on a single notion of abnormality and improves robustness. Each algorithm assigns an anomaly score to each sample, allowing us to rank them from the most to the least anomalous. Manuscript submitted to ACM

Portion Detected Oultiers (TPR)

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

1.0 0.8 0.6 0.4

Chance Hard (Synthetic)

0.2

Medium (Synthetic) Easy (Synthetic)

0.0 0.0

0.2

0.4 0.6 0.8 Portion Native Outliers (FPR)

1.0

Fig. 8. Average ROC curves for OC-SVM with synthetic outliers injection across all topics.

Each application is represented as a binary vector over the permission vocabulary P, where each dimension indicates the presence or absence of a given OAuth scope. This representation captures the effective access profile granted to the application’s service principal. We perform anomaly detection separately within each semantic topic, under the assumption that applications with similar declared functionality should exhibit comparable permission patterns. Detecting anomalies across the entire dataset without topic separation would confound functional heterogeneity with abnormal behaviour. Because no ground truth labels exist for anomalous applications, we adopt a controlled synthetic outlier injection strategy to tune hyperparameters. For each topic, we randomly select 20% of applications and perturb their permission vectors by (i) removing 𝑁 common permissions among the top 30 most frequent in the topic and (ii) adding 𝑁 rare permissions among the 30 least frequent, with 𝑁 = 1 (hard), 2 (medium), 4 (easy). This procedure simulates deviations from typical permission usage while preserving realistic permission structures. We evaluate model performance using ROC curves and Area Under the Curve (AUC), treating perturbed applications as positives and unmodified ones as negatives. Experiments are repeated across 𝑘 = 5 folds to mitigate randomness. For each model, we select hyperparameters that maximise average AUC across topics under the medium-difficulty setting. Figure 8 shows the results for the OC-SVM algorithm, which is particularly good in identifying those apps for which we altered their permissions (being those global outliers in fact). For the sake of space, we detail all the results of the tuning phase in Appendix D. This tuning procedure provides a principled way to calibrate detection models in the absence of labelled anomalies while maintaining comparability across topics. 5.2

Anomaly detection results

We apply each algorithm to identify the outliers in each topic. Each AD algorithm produces a set of applications identified as anomalous. To increase robustness and reduce model-specific biases, we define the final set of anomalous applications as the intersection of the three sets. This consensus-based approach ensures that only applications consistently flagged as anomalous across all detection methods are retained for further analysis. Table 4 shows the pairwise agreement rates, computed as the proportion of applications classified identically by both models over the total number of anomalies flagged (intersection over union). Overall, there is substantial overlap among the models’ predictions, ranging from a 74% agreement between OC-SVM and iForest to 84% between LOF and Manuscript submitted to ACM

Longo et al. Table 4. Pairwise agreement between anomaly detection models, measured as intersection-over-union of flagged apps.

Agreement OC-SVM LOF OC-SVM iForest LOF iForest Overall

0.77 0.74 0.84 0.67

iForest. At last, we verify the global agreement score as the arithmetic mean of the pairwise Cohen’s kappa coefficients, following the approach proposed in [9] for scenarios involving three raters evaluating the same set of items. The overall Cohen’s kappa coefficient across the three models is 0.48 on a scale from –1 to 1. Values around 0.5 are generally regarded as indicative of a good level of agreement among raters. The relatively high pairwise agreement rates and the strong global agreement score suggest that the models capture consistent notions of deviation. Across all topics, 139 applications (13% of the dataset) are flagged as anomalous under the consensus criterion. When anomaly detection is performed without topic separation, substantially fewer applications are identified, primarily those exhibiting extreme permission counts. This contrast highlights the importance of modelling permission profiles relative to functional peers rather than across heterogeneous application categories. We emphasise that anomalous classification does not imply malicious intent. Rather, flagged applications exhibit permission profiles that significantly deviate from those of other applications within the same semantic topic. Such deviations may correspond to over-privileged configurations, under-privileged configurations, or legitimately distinct design choices. To better understand the nature of these deviations, we perform qualitative validation through LLMassisted analysis and blind manual inspection. 5.3

Finding anomalous permissions

To better understand why an application is identified as anomalous, we perform a topic-aware analysis of permission anomalies. Since the set of 139 anomalous applications was obtained by combining the outputs of multiple anomaly detection algorithms, merging their individual explanation mechanisms would have produced results that are difficult to interpret. Instead, we introduce a simple and interpretable surprise score for each permission, quantifying how unusual the application’s permission request is with respect to the typical behaviour of applications belonging to the same topic. For each permission 𝑗, we compute the score: 𝑥 𝑗 log 𝑝 𝑗 + (1 − 𝑥 𝑗 ) log(1 − 𝑝 𝑗 ) , − log(𝑁 ) where 𝑥 𝑗 indicates whether the target application requests permission 𝑗, 𝑝 𝑗 is the probability that permission 𝑗 is 𝑆=

requested by applications belonging to the same topic as the target application (i.e., the fraction of applications in that topic requesting the permission), and 𝑁 is the number of applications in the topic. The most anomalous case occurs when all applications in a topic request a permission while the target application does not; or vice versa. In this situation, the numerator equals − log 𝑁 . Dividing by this quantity normalizes 𝑆 𝑗 to the range [0, 1], independently of the number of applications in the topic, where a value of 1 denotes the highest degree of anomaly. Manuscript submitted to ACM

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem 6

Qualitative verification

Our anomaly detection methodology leaves one important question open: are anomalous applications associated with higher security and privacy risks? To address this question, we use an LLM to provide a scalable, independent assessment of the alignment between application descriptions and requested permissions. Specifically, we analyse the most and least anomalous applications in each topic-specific anomaly ranking. We then manually inspect a subset of the most anomalous applications to evaluate the potential risks associated with their requested permissions. Our assessment is purely static and is based on the inherent risk of the permissions requested by each application. Dynamic analysis of how permissions are actually used at runtime is beyond the scope of this work and is left for future research. 6.1

LLM validation

To obtain a scalable semantic assessment of permission–description alignment, we employ GPT-5.2 via API as an auxiliary evaluation tool. For each application, we provide the LLM with its Marketplace description and full list of requested OAuth permissions, and prompt the LLM to classify the alignment between declared functionality and requested scopes. Choose ’Coherent, Safe’ if the permissions are coherent with the description and do not raise any security concerns. Choose ’Coherent, Risky’ if the permissions are somewhat coherent with the description but raise some security concerns. Choose ’Incoherent, Risky’ if the permissions are not coherent with the description and raise significant security concerns. Choose ’Insufficient’ if the permissions are not sufficient to accomplish the intents declared in the description. We evaluate applications at the tails of the anomaly distribution. For each anomaly detection model and each topic, we select the top 5% most anomalous and top 5% least anomalous applications based on per-topic anomaly scores. We also compute an aggregated ranking by summing the ranks across models and select the corresponding extremes. This strategy ensures coverage across topics and across different notions of deviation. For each anomaly detection algorithm rank, we select the 65 most-anomalous and 65 least-anomalous applications (after rounding per-topic quotas). Given description and permissions, the LLM returns both a categorical label and a short textual justification. Figure 9 summarises the distribution of LLM classifications. The last row of the heatmaps details the number of unique applications with the given classification across all three algorithms and the rank sum. Applications ranked among the least anomalous are predominantly labelled Coherent, Safe, whereas those ranked among the most anomalous are primarily labelled Coherent, Risky or Incoherent, Risky. This pattern is consistent across detection models and the aggregated ranking. Notably, many anomalous applications are classified as Coherent, Risky. Inspection of the LLM explanations indicates that these cases typically involve permissions that are functionally plausible but excessively broad (e.g., tenant-wide *.Read.All or *.ReadWrite.All scopes). For instance, Mindomo 9.0, a collaborative mind mapping and brainstorming application, contains the Files.ReadWrite.All permissions. While this scope is required for storing mind maps as declared in the description, it also extends access to all files across the entire organisation, significantly expanding the potential attack surface. Manuscript submitted to ACM

Longo et al. IForest

IForest

LOF

OCSVM

Overall (Sum)

Total (Unique)

t e y ien isk Saf ffic nt, t, R u e n s r e In he her Co Co

nt ere

nt cie

oh Inc

R nt, ere

Algorithm

isk

Classification

(a) Breakdown of the most anomalous apps.

Inc

isk

ent her

, Sa

uffi Ins

Classification

(b) Breakdown of the least anomalous apps.

Fig. 9. Breakdown of LLM classification outcomes, for each anomaly detection model and for the aggregated ranking.

Conversely, the Incoherent, Risky label is typically assigned only in the presence of obvious mismatches involving highly sensitive administrative scopes. For example, monday.com, a project management platform, requests the UserAuthenticationMethod.ReadWrite.All permission, effectively allowing the app to change the authentication methods of all users in the organisation. Interestingly, 13–15 applications ranked among the most anomalous are classified as Coherent, Safe. The motivations given by the LLM clarify that these cases rely mostly on resource-specific permissions. Although the choice of such permissions would be ideal from a security standpoint, the AD models flag these applications as anomalous because RSC permissions are rarely used. Moving onto the least anomalous apps in Figure 9b, most of them are classified as Coherent, Safe, as expected. Only two applications, HubSpot and Applauz, are labelled as Incoherent, Risky. In both cases, the LLM highlights a clear semantic mismatch between the declared purpose and the requested permissions. These applications are described as notification-oriented tools, yet they request read access to Teams conversations. Notably, similar conversational read scopes appear with high frequency within the “Notifications” topic, thus not triggering the AD algorithms. The high presence of Insufficient classifications among the least anomalous cases highlights the anomaly detection models’ struggle in detecting applications that are missing permissions required to accomplish declared functionalities. We note that LLM judgements do not constitute ground truth. Rather, they provide an independent semantic signal indicating whether detected deviations correspond to meaningful differences in apparent access scope. The observed separation between anomalous and least-anomalous groups suggests that topic-aware anomaly detection is correlated with possible security and privacy risks for the users installing the applications. While topic analysis and anomaly detection do not provide a definitive or standalone method for assessing an application’s risk profile, they can serve as valuable complementary tools for tenant administrators. By running the anomaly detection tool against an unknown application, the admin can obtain an additional metric to estimate the probability of the app showing potential misuse or malicious intent. 6.2

Manual inspection and verification

To complement the LLM-assisted evaluation and assess its reliability, we conduct a blind manual inspection of a subset of applications. The goal is not to estimate detection precision statistically, but to qualitatively assess whether anomalous permission profiles correspond to semantically meaningful deviations in declared access scope. We select Manuscript submitted to ACM

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

Table 5. A sample of anomalous applications, the LLM verdict, and the most anomalous permissions they request, with the scores calculated as described in 5.3. Resource-Specific Consent (RSC) permissions are marked in bold. Permissions considered anomalous for not being requested are marked by a "!".

App Name

Vacation Tracker

Seismic for Outlook

TeamOrgChart

monday.com

Officely

Brochesia

Vizerto

Podio Prod

Topic

Workspace Booking

File Management

LLM Verdict

Top 5 Anomalous Permissions

Score

Coherent, Risky

MailboxSettings.ReadWrite Group.Read.All MailboxSettings.Read InAppPurchase.Allow.User Team.ReadBasic.All

1.00 0.83 0.66 0.60 0.60

Coherent, Risky

Group.Read.All Contacts.Read.Shared GroupMember.Read.All Directory.Read.All Contacts.Read

0.86 0.86 0.77 0.71 0.66

email offline_access openid profile User.Read

1.00 1.00 1.00 1.00 0.33

User.Invite.All User.Export.All TeamsAppInstallation. ReadWriteSelfForUser.All User.ReadWrite UserAuthenticationMethod.Read.All

1.00 1.00 1.00

Incoherent, Risky

ChatMember.ReadWrite.All ChannelSettings.Read.Group Channel.ReadBasic.All Chat.ManageDeletion.All TeamsAppInstallation. ReadWriteAndConsentSelfForTeam.All

1.00 1.00 1.00 1.00 1.00

Coherent, Risky

TeamsTab.Delete.Group TeamsTab.Create.Group Channel.Delete.Group Channel.Create.Group TeamsTab.Read.Group

1.00 1.00 1.00 1.00 0.83

Coherent, Safe

ChatMessage.Send.Chat ChatMember.Read.Chat TeamMember.Read.Group Member.Read.Group ! User.Read

1.00 0.82 0.72 0.72 0.12

Coherent, Risky

Files.ReadWrite Contacts.Read Contacts.ReadWrite Mail.Read Files.Read

1.00 0.82 0.82 0.82 0.72

Intranet & WebParts Insufficient

Project Management Incoherent, Risky

Workspace Booking

E-Learning

AI Knowledge Base

1.00 1.00

Manuscript submitted to ACM

Longo et al.

eight applications among those flagged as anomalous. Table 5 summarises the apps’ topic, verdict from the LLM, and the five most anomalous permissions, with their relative surprise score. Overall, the manual inspection supports the effectiveness of our methodology. In six of the eight inspected cases, the anomalous permissions identified by our score correspond to privileges that appear excessive or difficult to justify given the application’s declared functionality. The remaining cases highlight meaningful limitations of the approach, primarily due to the rarity of Resource-Specific Consent (RSC) permissions or to applications requesting fewer permissions than expected. Rather than serving as a definitive security verdict, topic-aware anomaly detection provides a practical mechanism to prioritise applications and permissions for further inspection, complementing existing security assessment processes. In the following, we offer our findings, highlighting possible risks we found. Vacation Tracker. This app allows users to manage their paid time off within Microsoft Teams, Outlook, and Office. It was correctly assigned to the “Workspace Booking” topic, given the planning nature suggested by its description. It requests 13 total permissions, some of which we judge as excessive and not justified by the features declared in the app’s description. For instance, the Calendars.ReadWrite (𝑆 = 0.30) was granted with Application scope, which, as introduced in Section 2.2, allows the app to perform any operation to calendar events for every user across the tenant. This permission is rarely requested by other apps in the same topic, thus raising the anomaly score. The app also requests the MailboxSettings.ReadWrite (𝑆 = 1.00) permission, which also does not appear appropriate, as one would not expect a vacation tracker to modify a user’s global mail settings. Seismic for Outlook. Seismic is a sales management tool that facilitates content sharing and meeting workflows. This app adds Outlook integration to the Seismic software. It is correctly assigned to the “File Management” topic, given its focus on file sharing. Of the 10 total permissions requested by the app, we notice Directory.Read.All (𝑆 = 0.71), an Application-scoped permission which grants the ability to read the entire organisation’s directory structure. This level of access does not seem justified based on its stated functionality. In fact, this permission is not requested by other apps (at large). TeamOrgChart. This app is designed to generate and manage organisational charts and is correctly assigned to “Intranet & WebParts”, which focuses on SharePoint Web parts and related tools. This app requests only 5 minimal, delegated sign-in permissions. Of them, the only major permission is User.Read (𝑆 = 0.33), allowing the app to read the information of users who provide consent. We consider this as an example of under-privileged application that our models flagged as anomalous because the number and type of requested permissions differs from those of other apps in the same topic. The LLM correctly flag this app permissions as insufficient because it requests fewer permissions than the minimal necessary to perform the required functionality. This may also raise concerns about the app actual purpose. Monday.com. The application is designed to simplify the management of a team projects, providing a collaboration platform for organising and tracking tasks, team members, and custom workflows. Its domain fits well with the assigned topic “Project Management”. Among the 14 requested permissions, we find UserAuthenticationMethod.ReadWrite.All (𝑆 = 1.00), effectively requesting to read and modify the authentication methods of all users in the organisation. Authentication methods include things like a user’s phone numbers and Authenticator app settings, opening the door for possible abuse and data leaks. From the app’s description, the need for such permissions is questionable (and no other applications in this category require such high privileges). Officely. The application is a desk and workspace booking tool integrated within Microsoft Teams. It was correctly assigned to the topic “Workspace Booking”. Despite some of the 14 requested permissions being justified (e.g., User.Read.All (𝑆 = 0.32) to identify which users are booking desks), the overall permission set is excessive considering the application’s purpose stated in its description. Permissions like ChatMember.ReadWrite.All (𝑆 = 1.00) and Manuscript submitted to ACM

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

Chat.ManageDeletion.All (𝑆 = 1.00) are questionable and pose a security risk, as they allow the application to add or remove members from any chat and delete any chat message across the entire organisation. Brochesia. This application is designed to help on-site workers solve technical problems via Teams using Augmented Reality. We consider its classified topic (E-Learning) appropriate, even if not exactly a classic e-learning platform. All the 8 permissions requested by it are resource-specific (RSC), therefore restricting management to the specific Team it is installed into. Although the choice of using RSC permission would be ideal from a security standpoint, the AD flags this application as anomalous, as RSC permissions are rare within our dataset. This is a limitation of our methodology, which focuses more on the frequency of permissions rather than their adherence to security best practices. Vizerto. This app declares itself as a GenAI assistant to provide context-specific answers to business- and productrelated questions within an organisation. It was correctly assigned to the topic “AI Knowledge Base”, in line with its description. The app requests 4 RSC permissions only. These permissions are limited to reading specific chat, group and team member information (e.g., TeamMember.Read, 𝑆 = 0.72). It also includes the ChatMessage.Send.Chat (𝑆 = 1.00) permission, which allows the assistant to send messages in user chats. We consider these permissions appropriate based on the app’s description. This app likely shares the same reason as Brochure for flagging, given its heavy reliance on RSC permissions. Podio. The application provides no-code tools for building software solutions and setting up custom workflows. It was assigned to the topic “AI Knowledge Base”, although it could also fit the “Project Management” topic. Among the 9 permissions it requests, we notice some of them are quite high-level, such as Files.ReadWrite (𝑆 = 1.00) and Contacts.ReadWrite (𝑆 = 0.82). The LLM mark thus the app as risky. Nonetheless, these permissions appear well aligned with the app’s intended functionality: its workflow automation goals inherently require access to a wide range of features to support organisational processes, which could be seen as justified. The permissions are thus coherent for the LLM.

Related works

The security and privacy of application ecosystems have been widely investigated across different domains. Numerous works have analysed permission misuse, over-privileged access and data collection practices in Google’s Play Store for Android [1, 16, 21], Apple’s App Store for iOS [12, 19], and the general Web ecosystem [6]. More recently, similar concerns have emerged for large language model (LLM) ecosystems where large-scale analyses have uncovered misleading descriptions, abusive functionality, and excessive data collection among GPT-based chat systems [10, 20]. Few works exist involving the detection of suspicious consent grants in the Microsoft ecosystem, i.e., risks of users granting access to their data as a result of a phishing or unauthorised code execution attack. Previous works in the literature focus on attacking the ecosystem via machine-learning-based approaches [5] or exploiting security vulnerabilities [4]. Compared with these works, our study considers permissions that users voluntarily grant to applications that do not try to evade system defences. Another work [15] looks at network traces to detect potential information leaks from applications installed from the Microsoft Store. However, transmission of such data is unaware of the application’s behavioural context, which makes it hard to detect possible violations. Researchers have started exploring the possibility of using easily collectable metadata features to build stronger context and improve detection. This is enforced by the fact that anomalous behaviour always depends on the context, and this is particularly true in the field of application security. For instance, a feature that is anomalous for one application can be perfectly normal for another [7]. Late improvements in Natural Language Models and the introduction of Neural Manuscript submitted to ACM

Longo et al.

Topic Modelling (NTM) Techniques, such as Latent Dirichlet Allocation (LDA) [2] and BERTopic [8], have inspired researchers to extract semantic features to intersect context and behaviour. A first attempt to use NTM techniques in malware detection was made by Gorla et al. [7] who presented CHABADA, a system that was able to flag novel malware by leveraging NTM and clustering (K-Means in their case) to extract topics from the application descriptions. Further improvements were made by Ranaweera et al. who proposed BERTDetect [16], a system that leverages BERTopic to extract topics, achieving improvements in False Negative Rate (FNR) and True Positive Rate (TPR). In our work, we build on similar techniques to identify homogeneous apps classes and then run AD algorithms. Considering NTM, BERTopic [8] is one of the most recent and effective models for topic extraction, as it leverages the power of transformer-based embeddings, unlike probabilistic bag-of-words models such as LDA [13] or TF-IDF. BERTopic is proven to be effective in capturing semantic similarity among documents, remaining competitive across a wide range of benchmarks involving traditional models in topic modelling. For anomaly detection, we rely on well-established algorithms. Recent approaches have explored the use of deep neural network–based models, such as autoencoders, for anomaly detection [14]. However, these methods typically require large amounts of training data and introduce additional computational and modelling complexity, which are not well-suited to our limited dataset and purely unsupervised setting. 8

Conclusions

We presented the first systematic measurement of the Microsoft 365 (M365) third-party application ecosystem from a permission-centric security perspective. By combining public APIs, automated tenant-side deployment, and operational tenant data, we crawled more than 8,000 applications, of which only 1,069 expose both textual descriptions and OAuth permission sets. This limited coverage itself highlights the lack of transparency and the fragmented nature of the current M365 application ecosystem. To assess whether applications request permissions consistent with their declared functionality, we proposed a topic-aware analysis pipeline combining Neural Topic Modelling with unsupervised anomaly detection. Rather than comparing applications across the entire ecosystem, our approach models permission profiles relative to semantically similar peers, enabling the identification of both over-privileged and under-privileged applications. Using a conservative consensus criterion, we identified 139 anomalous applications (13% of the analysed dataset). LLM-assisted evaluation and blind manual inspection support the effectiveness of the proposed methodology, showing that many detected anomalies correspond to permission requests that appear difficult to justify given the application’s declared functionality. Although metadata alone cannot prove misuse or malicious intent, topic-aware anomaly detection provides a useful signal for prioritising applications for further security review. Overall, our findings expose structural opacity and non-uniform privilege allocation in the M365 ecosystem, underscoring the need for improved transparency and automated permission auditing in enterprise environments. We believe the proposed pipeline can provide tenant administrators with a practical decision-support tool, helping them identify applications whose requested permissions deviate from those of functionally similar peers before granting consent, thereby supporting more informed decisions and promoting the principle of least privilege. Acknowledgments This work was supported by the AI4CTI FISA under Project #FISA-2023-00168, funded by the Italian Ministry of University and Research (MUR). Manuscript submitted to ACM

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

The authors used generative AI-based tools to revise the text, improve flow and correct any typos, grammatical errors, and awkward phrasing. The authors are fully responsible for the content of the paper, including the study design, data collection, analysis, interpretation of results, and all conclusions.

References [1] Marco Alecci, Jordan Samhi, Tegawende F. Bissyande, and Jacques Klein. 2024. Revisiting Android App Categorization. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE ’24). [2] David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res. (2003). [3] Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: A survey. ACM Comput. Surv. (2009). [4] Yunang Chen, Yue Gao, Nick Ceccio, Rahul Chatterjee, Kassem Fawaz, and Earlence Fernandes. 2022. Experimental Security Analysis of the App Model in Business Collaboration Platforms. In 31st USENIX Security Symposium (USENIX Security 22). [5] Stefan Cracea. 2023. Security Analysis of Azure Active Directory Logs using Machine Learning. Master’s thesis. Eindhoven University of Technology. [6] Alberto Fernandez-de Retana, Jannis Rautenstrauch, Igor Santos-Grueiro, and Ben Stock. 2025. A Permissions Odyssey: A Systematic Study of Browser Permissions on Modern Websites. In Proceedings of the 2025 ACM Internet Measurement Conference. [7] Alessandra Gorla, Ilaria Tavecchia, Florian Gross, and Andreas Zeller. 2014. Checking app behavior against app descriptions. In Proceedings of the 36th International Conference on Software Engineering. [8] Maarten Grootendorst. 2022. BERTopic: Neural topic modeling with a class-based TF-IDF procedure. arXiv:2203.05794 [9] Kevin A Hallgren. 2012. Computing inter-rater reliability for observational data: An overview and tutorial. Tutor. Quant. Methods Psychol. (2012). [10] Xinyi Hou, Yanjie Zhao, and Haoyu Wang. 2025. On the (in) security of llm app stores. In 2025 IEEE Symposium on Security and Privacy (SP). [11] Leland McInnes, John Healy, Steve Astels, et al. 2017. HDBSCAN: Hierarchical density based clustering. J. Open Source Softw. (2017). [12] Madihah Mohd Saudi, Muhammad Afif Husainiamer, Azuan Ahmad, and Mohd Yamani Idna Idris. 2024. iOS mobile malware analysis: a state-of-theart. Journal of Computer Virology and Hacking Techniques (2024). [13] Martin Nguyen and Tirath Ramdas. 2025. Topic Modeling: A Comparative Overview of BERTopic, LDA, and Beyond. chamomile.ai/topic-modelingoverview/ [14] Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel. 2021. Deep learning for anomaly detection: A review. ACM computing surveys (CSUR) (2021). [15] Florin-Razvan Potcoveanu. 2025. Analyzing Privacy Risks in Microsoft Store Applications’ Network Traffic. Master’s thesis. Radboud University. [16] Nishavi Ranaweera, Jiarui Xu, Suranga Seneviratne, and Aruna Seneviratne. 2025. BERTDetect: A Neural Topic Modelling Approach for Android Malware Detection. In Companion Proceedings of the ACM on Web Conference 2025 (WWW ’25). [17] Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). [18] Tim Sainburg, Leland McInnes, and Timothy Q Gentner. 2021. Parametric UMAP embeddings for representation and semisupervised learning. Neural Computation (2021). [19] Gian Luca Scoccia, Marco Autili, Giovanni Stilo, and Paola Inverardi. 2022. An empirical study of privacy labels on the Apple iOS mobile app store. In Proceedings of the 9th IEEE/ACM International Conference on Mobile Software Engineering and Systems. [20] Yuhao Wu, Evin Jaff, Ke Yang, Ning Zhang, and Umar Iqbal. 2025. An in-depth investigation of data collection in llm app ecosystems. In Proceedings of the 2025 ACM Internet Measurement Conference. [21] Shishuai Yang, Qinsheng Hou, Shuang Li, Fenghao Xu, and Wenrui Diao. 2025. From guidelines to practice: assessing Android app developer compliance with google’s security recommendations. Empirical Software Engineering (2025).

Ethics

The ethical considerations about this work mostly reside in the data collection phase. First, we took all the precautions to prevent the crawling of applications’ permissions via the Microsoft Marketplace from negatively impacting the destination platform: we only performed a few thousand operations on the platform in a range of more than 48 hours, a negligible volume for a company with resources such as Microsoft’s. Second, while collecting app permissions from the University tenant, we received from the system administrators anonymised data about installed applications, which did not contain any identifier that could be linked to any University student or faculty. Manuscript submitted to ACM

Longo et al. Table 6. Final selected BERTopic configurations.

Parameter

Base Value

SBERT_model n_components n_neighbors min_cluster_size min_samples min_df max_df ngram_range top_n_words

all-MiniLM-L6-v2 35 5 10 10 5 0.95 (1, 2) 10

BERTopic parameter selection

As described in Section 4.1, we perform a “bottom-up” hyper-parameter tuning process selecting the set of parameters which can have the highest impact on the model performance. We proceed with the tuning process, starting from one model and running each configuration of parameters with three different random states. At each step, we select the best configurations to include in the next search for the upper model and its parameters, and so on. For the sake of clarity, we leave the details to a separate document we will link to upon acceptance, to preserve the authors’ anonymity. We simply report the BERTopic final parameters in Table 6. C

Extracted topics

The list of extracted topics, their meaning, and the number of associated applications is detailed in Table 7. D

Anomaly detection model tuning

D.1

Injecting Outliers

As mentioned in Section 5.1, we used three distinct anomaly detection algorithms to identify outliers in each topic: One-Class SVM (OC-SVM), Isolation Forest (iForest), and Local Outlier Factor (LOF). This means that for each algorithm we train and evaluate a separate model for each topic. Before applying the models directly on the topics, we first look to assess their performance on identifying anomalies and select the best hyper-parameters for each algorithm. The complete list of parameters for each level of difficulty are reported in Table 8. We evaluated the performance of each model on each injection strategy (e.g., Hard Synthetic) plotting the average ROC curve across all topics. On the x-axis of the ROC curve we report the False Positive Rate (FPR), which represents the percentage of native outliers detected by the model. On the y-axis we report the True Positive Rate (TPR), representing the percentage of injected outliers correctly flagged by the model. The objective is to maximize the TPR while keeping an acceptable FPR (i.e., not flagging too many native applications as outliers). As shown in the average ROC curves in Figure 10, the OC-SVM reflects the difficulty levels almost perfectly for the easy level. As expected, the medium and hard levels result more challenging for the model. Both LOF and iForest models show more difficulty in solving the Synthetic strategy compared to OC-SVM, although they still display decent performance. The difficulty levels chosen for each algorithm are reported in Table 9 along with the best global hyperparameters identified after the grid search with k-fold cross-validation. Manuscript submitted to ACM

Lost in Permissions: Exploring the Microsoft 365 App Ecosystem

Table 7. Representative words, interpretation, and application count for each topic.

Top-5 Terms

Interpretation

Count

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

(crm, file, template, attachment, send) (360, student, learning, course, training) (ticket, notification, onboarde, core, incident) (office, room, book, desk, space) (asset, security, permission, software, expense) (calendar, event, meeting, appointment, scheduling) (idea, board, meeting, visual, brainstorm) (sharepoint, page, card, viva, connection) (birthday, web, sharepoint, webpart, navigation) (signature, sign, document, signing, digital) (task, project, checklist, board, gantt) (sms, call, communication, message, agent) (recognition, culture, employee, recognize, peer) (ai, knowledge, answer, enterprise, unleash) (meeting, ai, summary, transcript, meet) (survey, video, feedback, quiz, meeting) (break, copilot, habit, journey, movement) (language, translation, meeting, multilingual, ai) (confluence, tab, view, analytic, meeting) (form, approval, travel, sharepoint, component) (pdf, pdfs, document, convert, file) (directory, employee, people, hr, profile) (expense, project, task, budget, claim) (legal, ai, firm, matter, capture)

File Management E-Learning Notifications Workspace Booking Security Scheduling & Calendar Collaborative Brainstorming SharePoint Integration Intranet & WebParts Digital Signatures Project Management Communication Tools Employee Recognition AI Knowledge Base AI Meeting Assistants Surveys & Feedback Well-being & Productivity AI Translation Confluence Integration Forms & Workflows Design PDF Management Human Resources Expense Management Legal AI Assistant

122 65 65 59 75 63 62 49 45 33 50 31 29 50 37 28 36 19 55 25 15 20 24 12

Table 8. Parameters used for synthetic outlier injection under different perturbation difficulty levels.

Difficulty

Parameter

Easy

num_common_to_remove num_rare_to_add

4 4

Medium

num_common_to_remove num_rare_to_add

2 2

Hard

num_common_to_remove num_rare_to_add

1 1

D.2

Value

Model Performance on Injected Outliers

Once selected the best hyper-parameters for each algorithm, we evaluate the performance of each model on the injected outliers. All anomaly detection algorithms used in this project are designed to work with thresholds and anomaly scores. Specifically, OC-SVM has a threshold which is implicitly set to zero, as the anomaly score is the distance from Manuscript submitted to ACM

1.0 0.8 0.6 0.4

Chance Hard (Synthetic)

0.2

Medium (Synthetic) Easy (Synthetic)

0.0 0.0

0.2

0.4 0.6 0.8 Portion Native Outliers (FPR)

1.0

Portion Detected Oultiers (TPR)

Longo et al. Portion Detected Oultiers (TPR)

26 1.0 0.8 0.6 0.4

Chance Hard (Synthetic)

0.2

Medium (Synthetic) Easy (Synthetic)

0.0 0.0

(a) Average ROC curves for LOF.

0.2

0.4 0.6 0.8 Portion Native Outliers (FPR)

1.0

(b) Average ROC curves for iForest.

Fig. 10. Average ROC curves for OC-SVM, LOF, and Isolation Forest with synthetic outlier injection across all topics. Table 9. Selected difficulty levels and hyperparameters for each anomaly detection algorithm.

Algorithm

Difficulty Level

Best Hyperparameters

OC-SVM LOF iForest

Medium Medium Medium

nu=0.1 k=5 n_estimators=100

Table 10. Summary of selected thresholds and corresponding performance for each anomaly detection algorithm on injected outliers.

Algorithm OC-SVM LOF Isolation Forest

Selected Threshold

TPR

FPR

0 (default) -1.29 (median) 0.39 (median)

0.91 0.61 0.79

0.32 0.31 0.39

the separating hyperplane, while LOF and iForest require the setting of a threshold on the anomaly scores to classify an application as an outlier. For these two algorithms, we also provide an analysis of the optimal thresholds for each topic, selecting the best global threshold across all topics based on the median of optimal thresholds. The optimal threshold for each topic is identified by maximizing the Youden’s Index, i.e., 𝑇 𝑃𝑅 − 𝐹 𝑃𝑅. The summary of the selected thresholds and the corresponding performance (TPR, FPR) is reported in Table 10.

Manuscript submitted to ACM

Record · ID 423835 · SHA-256 9a7db7864f92a0c5
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.