Stirna Consulting

Category: Artificial intelligence

  • The AI Scaling Problem: Why Millions Are Spent on Pilots That Nobody Uses (And How to Fix It)

    Applying a Realist Evaluation Framework to Public Sector AI and Climate Governance

    Artificial Intelligence is firmly on the agenda of governments worldwide. From the Republic of Korea to Spain, and India to Brazil, IT leaders and ministers are actively defining policies to apply AI to modernize public sector organizations, vastly improving the effectiveness and efficiency of administration.

    The ambition is staggering. As highlighted during the recent OECD global report launch for Governing with Artificial Intelligence, the UK estimates that AI could automate 84% of repetitive public service transactions, saving the equivalent of 1,200 person-years of work annually. In Brazil, the government has introduced a strategic plan to avoid climate disaster in the Amazon using AI-enabled environmental monitoring systems.

    Yet, despite these ambitious modernization programs, governments face a growing crisis: the rise of failed implementations. Millions are spent on AI solutions that do not scale or never progress past the pilot stage.

    Why are these transformative initiatives failing to deliver genuine public value? The answer lies not in the technology, but in inappropriate evaluation methods.

    The OECD Framework and the Evaluation Trap

    To guide leaders in their AI-enabled modernization programs, the OECD (2025) proposed the Framework for Trustworthy AI in Government. It is built upon three broad pillars:

    1. Enablers: The foundational building blocks required to facilitate AI adoption, including data, digital infrastructure, skills/talent, and governance mechanisms (such as AI accelerators and innovation hubs).
    2. Guardrails: The binding and non-binding instruments needed to guide trustworthy use, encompassing risk management, transparency, accountability, and robust oversight.
    3. Engagement: Approaches to shape user-centered adoption by actively involving citizens, civil servants, and stakeholders to ensure AI is responsive to societal needs.

    However, even with these pillars in place, the scaling problem persists. During the OECD launch event, speakers widely acknowledged the “challenges that we’re all facing in actually scaling AI projects… from a pilot to large-scale change.” The core issue is that standard evaluations are fundamentally flawed for complex digital transformation. Traditional evaluations treat success as a binary on/off switch: Was the software deployed? Yes or No? Did the users log in? Yes or No? But human adoption, trust, and institutional change in complex environments—whether in a sprawling public agency or the Amazon basin—do not operate in binary terms.

    The Solution: A Realist Approach to Digital Transformation

    As the Spanish Minister emphasized at the OECD event, we must look deeply into specific use cases to understand successes and failures, reminding us that “this is AI for humans… and that’s also talking about transparency because that is a key issue now.” Relying on use cases makes sense because it provides insights into the unique contextual conditions of an application domain.

    To solve the scaling problem, a novel digital transformation evaluation framework has been developed by researchers from Reykjavik University alongside industry practitioners (Hjaltalin et al., forthcoming).

    This framework moves beyond asking “Did it work?” by adopting and adapting a Context + MechanismMechanism refers to a specific intervention introduced in a particular social program. For example, the police install CCTV cameras on streets with a high rate of burglary. The functioning of mechanism in a program is best understood through metaphors. One is that of a ‘trigger’: a pistol fires a bullet triggered by the ignition of gunpowder within the pistol’s barrel. Another is the ‘dimmer-switch’ metaphor: a dimmer-switch operates on a continuum, where at maximum the light is very bright and at minimum it is pitch black. My understanding aligns closer to the latter metaphor and is used as such in my research. = Outcome (CMO) realist evaluation approach:

    • Context (C): It examines the particular contextual conditions (e.g., governance structures, common standards, digital rights) that enable or constrain the initiative.
    • Mechanism (M): It assesses the human drivers—such as trust and behavioral shifts among civil servants or citizens.
    • Outcome (O): It evaluates to what extent the AI initiative actually produced public value outcomes under those particular circumstances.

    Moving from Binary to Continuum

    The true power of this framework is that it treats a particular mechanism as producing outcomes on a continuum, rather than a binary on/off switch.

    Take Brazil’s AI climate governance plan as an example. Instead of a standard evaluation that simply asks, “Do the environmental enforcement agents use the AI risk predictions? (Yes/No),” the CMO framework evaluates the mechanism of trust on a continuum.

    Perhaps the agents in a specific region are currently at “skeptical but experimenting,” which produces the outcome of “partial verification of AI data.” By mapping this continuum and identifying how specific contextual conditions constrain these mechanisms, policymakers don’t just see a “failed” pilot. They see exactly where the human friction is. They can then intervene—perhaps through targeted training or greater stakeholder engagement—moving the mechanism further along the continuum toward active reliance and co-creation.

    Bridging the Gap Between Pilot and Public Value

    Deploying technology is only ten percent of the battle. An incredibly sophisticated AI model that environmental agents ignore on the ground produces zero real-world value.

    To overcome the scaling problem, governments and enterprises need to stop treating digital transformation as a simple IT deployment. Leaders must adopt analytical guardrails that measure context, mechanism, and outcomes to ensure that when an AI initiative scales, it genuinely empowers users and delivers measurable public value.

    How Stirna Consulting Can Help

    At Stirna Consulting, we understand that AI implementation is deeply sociotechnical. Led by our CEO and lead analyst, Hjaltalin—co-author of the forthcoming realist evaluation framework mentioned above—we don’t just help build systems. We provide the proprietary analytical methodologies and cross-functional change management necessary to ensure your digital transformation doesn’t stall at the pilot phase.
    Whether you are deploying enterprise CRM software or navigating complex public sector IT programs, we ensure your technology empowers your team rather than dictating to them. Reach out to Stirna Consulting today for practical advice on digital transformation and clear next steps.

    Sources

    Hjaltalin, I. T., Candi, M., Sigurðarson, H. Th. (2026). Agile Governance in Public Sector Digital Transformation: a Program Evaluation [Manuscript submitted for publication]. Department of Business and Economics, Reykjavik University.

    Hjaltalin, I. T., Bjartmarz, Th. (2026). Artificial Intelligence in Environmental Governance: Reconfiguring Public Value, Legitimacy, and Operational Capacity in EU Multi-Level Networks [Manuscript in preparation]. Department of Strategy & Innovation, Copenhagen Business School.

  • Speaking Icelandic when you are an American AI: Multilingual capabilities of LLMs

    Speaking Icelandic when you are an American AI: Multilingual capabilities of LLMs

    Introduction

    Being an author of a tech blog, one must constantly seek new or noteworthy ideas or material to talk about. Scarcity of ideas is not really a problem. Rather, the ideas are not always good. The idea behind this blog article, that is, to compare AI models in terms of their capacity to speak Icelandic (or any non-English language for that matter), would have been dismissed as ridiculous had I thought of it when I launched this website and blog in 2020. That year, I started a blog and an independent IT practice. I also started my PhD study on AI and e-government, being accepted into the PhD program at Reykjavik University. At the time, generative AI was still recent and apps such as Chat GPT did not exist. Most tech bloggers were not interested in AI’s linguistic potential. Instead, bloggers curious about AI emphasized machine learning and big data capabilities. In brief, AI applications such as machine translations (read Google Translate) have, historically, been terrible at translating small languages, particularly Icelandic.

    Large language models or LLMs, such as GPT series, are enabled by free open big data on the internet and powerful cloud-based infrastructure, have given rise to the development of virtual agents like ChatGPT. Such AI systems are called generative AI (GenAI) applications and can generate text in response to a particular prompt or input from users (Hjaltalin, forthcoming).

    Icelandic is a micro language, only spoken by approximately 300.000 people, most of whom live in Iceland. Icelanders ought not to take for granted that it will be in major LLMs. AI researchers in Iceland have worked hard to create an Icelandic

    language model (not to be confused with LLMs, however). The aim is to ensure that AI models, including popular LLMs (e.g., GPT-4), can speak and understand Icelandic (see Heimisdottir, 2024).

    The method

    Against this (albeit brief) background, let’s delve into the specifics of the analysis. I chose Gemini (Google’s LLM) and Co-Pilot (powered by OpenAI’s LLM, i.e., GPT-4) as my GenAI sample. While OpenAI’s GPT-4 represents the first LLM to learn Icelandic (“Preserving languages for the future”, 2023), Gemini has, more recently, learned it as well (see below). Both LLMs were presented with the following two questions:

    1. Talar þú íslensku? / Do you speak Icelandic?
    2. Hvernig myndir þú lýsa Íslandi? / How would you describe Iceland?

    The evaluation of the results is not based on the content’s size or quality but on grammatical soundness. For instance, I check if they use appropriate words and if they use the correct form (e.g., past tense, case, etc.). Particularly, fallbeyging or declination (cases shaping nouns and pronouns) is a difficult grammar rule that foreigners (and some Icelanders) struggle with when learning the language. If the model accomplishes declination of (pro)nouns, this would indicate superb Icelandic skills.

    Analysis and results

    In this section, I present my evaluation of the performance of the two models. For your information, I am not a linguist or Icelandic teacher, but I have a good understanding of my native tongue. I feel confident that my analysis is robust and valid for Icelandic.

    Results indicate that both models speak (or write) Icelandic. Excellent!

    This covers the first part of my analysis. In the second part, we will explore how well or poorly they perform in their spoken (or written) Icelandic, particularly focusing on grammar rules (see above).

    Gemini’s Icelandic skills are surprisingly good. The text is remarkably elo- quent and creative in terms of vocabulary and descriptions. On the other hand, as presented in Figure 1, there were minor errors in Gemini’s response to my question.

    Figure 1: Gemini’s (G) answer to question 2: How would you describe Iceland?
    Figure 1: Gemini’s (G) answer to question 2: How would you describe Iceland?

    Specifically, three errors were found in terms of pronouns and forming adjectives depending on the gender of the subject at hand.

    Co-pilot’s Icelandic skills were decent, although I hoped for a more creative response. While it is clearly not as eloquent and rich as Gemini’s response, this is not the determining factor in the evaluation. Indeed, Co-pilot performed quite well in terms of grammar and even outperformed Gemini in some areas, such as using pronouns (see Figure 2).

    Figure 2: Co-pilot’s (C) answer to question 2: How would you describe Iceland?
    Figure 2: Co-pilot’s (C) answer to question 2: How would you describe Iceland?

    Gemini is stronger in fallbeyging or declination; I did not detect any errors in its response in terms of declination of nouns. However, Co-pilot uses the incorrect case of Geysir, as illustrated in Figure 2. As such, Gemini’s Icelandic skills are superior to Co-pilot’s.

    Conclusion

    Most (if not all) Icelandic linguists can agree with the claim that declination of nouns is the “holy grail” of Icelandic proficiency. If you have mastered declination (or fallbeyging), you have reached an advanced proficiency in the Icelandic language.

    Overall, I was impressed with the proficiency of both AI models in Icelandic. I am curious to know how well they perform in other languages. Of course, the benchmark used here (i.e., declination) does not apply to other languages, so this should be adapted in future research. Please share your thoughts in the comments section below.

    References

    Heimisdottir, L. (2024, September). The Icelandic Approach: Preserving and Re- vitalizing Linguistic and Cultural Diversity in AI.

    Preserving languages for the future. (2023, March).

  • Generative AI applications in business applications: a comparison between Salesforce and Microsoft’s digital platforms

    Generative AI applications in business applications: a comparison between Salesforce and Microsoft’s digital platforms

    Generative AI and its applications

    Generative AI (GenAI) represents a significant shift from traditional AI models that involves moving from recognizing patterns to generating new content such as text, audio, video, and images. In the context of enterprise software, GenAI has various applications:

    • Automation: Automates repetitive tasks, enhancing productivity and efficiency.
    • Customer Service: Provides personalized responses and support through chatbots and virtual assistants.
    • Data Analysis: Generates insights from large datasets, aiding decision-making and strategic planning.

    Recent Trends and Advancements

    We can note recent advancements in GenAI within the enterprise sector that include the development of so-called large language models (LLMs), such as OpenAI’s GPT series. LLMs can generate human-like text and have an exceptional understanding of natural language. Major tech companies like Salesforce and Microsoft are embedding GenAI into their platforms to innovate. This involves a significant increase in investments in AI technologies and efforts to forge new partnerships with GenAI startups to drive innovation and stay competitive.

    Importance of Efficiency in AI Models

    Efficiency in new AI models like LLMs is crucial to reduce innovation barriers. In fact, very few companies can afford to buy the computational resources needed to effectively run LLMs. In particular, efficiency of LLMs should be improved because of:

    • Computational resources: Efficient models require fewer computational resources, making them more accessible and sustainable.
    • Speed: Faster models can process and generate outputs quickly, improving user experience.
    • Cost-effectiveness: Reducing the computational load lowers operational costs, making AI solutions more affordable (or feasible).

    Evaluating the Efficiency of GenAI Models

    To evaluate the efficiency of GenAI models, we must consider issues such as (Hugging Face, n.d.):

    • Resource utilization: Measuring the computational power and memory required.
    • Processing speed: Assessing the time taken to generate outputs.
    • Cost: Analyzing the operational costs associated with running the models.
    • Performance metrics: Evaluating the generated content’s accuracy, relevance, and quality.

    Salesforce’s Platform and GenAI Integration

    Salesforce, a leading CRM platform, has integrated GenAI into its various Cloud services, including Sales Cloud, Service Cloud, Marketing Cloud, and Commerce Cloud (see Haki et al., 2025). This integration was found to enhance personalized communication, improve process automation, and data-driven insights.

    Bring Your Own Large Language Model (BYOLLM)

    BYOLLM allows customers to use their LLMs within the Salesforce platform. This approach offers flexibility and ensures customers can leverage models that best fit their specific needs, potentially improving efficiency and performance.

    Examples of Salesforce’s Adoption of Appropriately Sized LLMs

    In addition to the BYOLLM approach, Salesforce’s AI strategy about ‘appropriately’ sized (or not so large) LLMs concerns efficiency and performance enhancement through (1) their own AI platform and (2) existing models:

    1. Proprietary models: Salesforce has developed its models, like xLAM and xGen-Sales, which are optimized for specific tasks and industries.
    2. Fine-tuned models: Salesforce uses fine-tuned versions of existing models like OpenAI’s GPT. These models are customized with Salesforce data to align with customer needs.

    Such strategies (i.e., 1 and 2) ensure that Salesforce can deliver high-performance GenAI capabilities while effectively managing computational resources and costs (Haki et al., 2025).

    Microsoft’s AI Platform and Integration of Generative AI Models

    Microsoft’s AI platform, primarily powered by Azure, integrates generative AI models to enhance enterprise software applications, such as Microsoft Dynamics (CRM). Azure OpenAI Service allows businesses and institutions to leverage LLMs, including GPT-4, for various tasks, e.g., data analysis, customer support, and workflow automation. A key feature of this digital platform is that it includes a framework for integrating AI models with external tools and data sources (“Azure OpenAI Service”, n.d.), linking AI models and external services. The integration framework (or MCP) standardizes communication, which could enhance service integration, accessibility, and AI capabilities (Kaur, 2025).

    Microsoft’s BYOLLM Approach

    Microsoft’s Bring Your Own LLM (BYOLLM) approach allows businesses to use their LLMs on Azure. This approach can enhance efficiency by enabling organizations to customize their AI solutions based on their needs, which can lead to better performance and relevant results. For example, this allows businesses to perform fine-tuning, i.e., a particular machine learning technique (Roberts et al., 2020), which significantly reduces the time and computational resources required.

    Examples of Appropriately Sized LLMs for Efficiency and Performance

    Microsoft’s AI strategy to optimize LLMs involves (Jiang et al., 2023):

    • LLMLingua: A technique that compresses prompts for LLMs, i.e., reduces the length of prompts while maintaining their effectiveness. This technique led to circa 20x reduction in prompt size that improved inference speed and reduced latency.

    • On-Prem LLMs: Developing tools that allow businesses to deploy LLMs on their on-premise IT infrastructure, supporting increased data security and privacy. These tools helped optimize inference performance on existing infrastructure that maintained consistent latency and throughput.

    • Scalable Fine-Tuning: Azure OpenAI Service supports scalable fine-tuning of LLMs, allowing businesses to adapt models to their specific tasks efficiently, as noted above.

    These examples highlight Microsoft’s commitment to providing a resilient and robust digital platform for AI development (i.e., Azure OpenAI Service) tailored to the needs of individual businesses (Jiang et al., 2023).

    Conclusion

    Both Salesforce and Microsoft are actively embedding GenAI into their platforms, focusing on efficiency through strategies like BYOLLM and using appropriately sized or optimized models tailored to specific business needs. Both companies offer flexible and high-performance AI solutions for enterprises.

    Their AI strategies differ slightly, however. Whereas Salesforce emphasizes proprietary and fine-tuned models integrated deeply within its CRM platform, Microsoft leverages its broad Azure infrastructure, offering advanced optimization techniques and flexible deployment options like on-premise solutions.

    Feel free to leave your thoughts in the comments section below.

    References

    Azure OpenAI Service. (n.d.). Retrieved from azure.microsoft.com on April 14th 2025.

    Haki, K., Safaei, D., Magan, A., & Griffiths, M. (2025). Integrating Generative AI Into Enterprise Platforms: Insights From Salesforce. Information Systems Journal. doi: 10.1111/isj.12593

    Hugging Face. (n.d.). LLMs. Retrieved from huggingface.co/docs/ on April 15th 2025.

    Jiang, H., Wu, Q., Lin, C.-Y., Yang, Y., & Qiu, L. (2023). LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023).

    Kaur, S. (2025, March). Unleashing the Power of Model Context Protocol (MCP): A Game-Changer in AI Integration — Microsoft Community Hub.

    Roberts, A., Raffel, C., & Shazeer Google, N. (2020). How Much Knowledge Can You Pack Into the Parameters of a Language Model? Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, 5418– 5426.

  • ChatGPT and Microsoft

    ChatGPT and Microsoft

    Earlier this year (Jan 2023), Microsoft announced its partnership with OpenAI concerning artificial intelligence (AI) development in the coming years. This announcement was particularly interesting because, just a few weeks earlier, OpenAI released ChatGPT3, which caught the world by surprise, to say the least. So what does this mean for Microsoft users, in particular those using its business applications?

    To answer this question, I researched one new product, Sales Copilot, and tested it (see below). It is an add-on to Outlook that integrates CRM data with Outlook emails using AI. This seems appropriate as Outlook is the app many people use to talk with colleagues and partners. ChatGPT (the underlying technology) is precisely that, i.e., an interface used to chat with AI to solve problems or find solutions to questions. Imagine an AI agent that is always available in case we have questions or need answers to work-related issues. Wouldn’t that be something?!

    Before we get too excited, GhatGPT, and other systems based on similar technology, are still in their infancy. These systems are not yet mature enough to have a serious transformational impact on our daily work. Nevertheless, the impact has been large in terms of people’s increased interest in AI, which, generally, I think is a good thing. This interest will drive further investments into AI research and development.

    Anyway, turning to the topic at hand. In the announcement mentioned above, Microsoft said: “we will deploy OpenAI’s models across our consumer and enterprise products and introduce new categories of digital experiences built on OpenAI’s technology.” Moreover, it talks about how Microsoft has supplied the company with the IT infrastructure needed to develop its products, including ChatGPT and other solutions. However, Microsoft didn’t reveal much about its intentions to use these solutions until months later when it announced Microsoft Sales Copilot (previously Viva Sales).

    The Sales Copilot is an AI application that enables sales teams to better access relevant client information to help them close more deals.

    Sales Copilot in Outlook

    Sales Copilot can show you highlights from email conversations with clients or prospects, which you can use to perform various tasks, e.g., create a reply, save information to the CRM (Salesforce or Dynamics 365).

    If you would like to test the AI, it’s easy to add it to Teams and/or Outlook. Let me know in the comments what you think.